All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v4 0/6] media: v4l2: Add extended fmt and buffer ioctls
@ 2020-07-17 11:54 Helen Koike
  2020-07-17 11:54 ` [PATCH v4 1/6] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more) Helen Koike
                   ` (8 more replies)
  0 siblings, 9 replies; 29+ messages in thread
From: Helen Koike @ 2020-07-17 11:54 UTC (permalink / raw)
  To: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus, linux-media
  Cc: Helen Koike, tfiga, hiroh, nicolas, Brian.Starkey, kernel,
	boris.brezillon, narmstrong, linux-kernel, frkoenig, mjourdan,
	stanimir.varbanov

Hi,

I'm sorry for taking too long to submit v4.

It is not perfect, not all v4l2-compliance tests passes, but I'd like a review,
specially on the API and potential problems, so I can focus on improving implementation
and maybe drop the RFC tag for next version.

Follow below what changed in v4 and some items I'd like to discuss:


* Ioctl to replace v4l2_pix_format
---------------------------------------------------------------------------------
During last media summit, we agreed to create ioctls that replace the v4l2_pix_format
struct and leave the other structs in the v4l2_format union alone.
Thus I refactored the code to receive struct v4l2_ext_pix_format, and I renamed the
ioctls, so now we have:

int ioctl(int fd, VIDIOC_G_EXT_FMT, struct v4l2_ext_pix_format *argp);
int ioctl(int fd, VIDIOC_S_EXT_FMT, struct v4l2_ext_pix_format *argp);
int ioctl(int fd, VIDIOC_TRY_EXT_FMT, struct v4l2_ext_pix_format *argp);

The only valid types are V4L2_BUF_TYPE_VIDEO_CAPTURE and V4L2_BUF_TYPE_VIDEO_OUTPUT,
all the other types are invalid with this API.


* Modifiers
---------------------------------------------------------------------------------
I understand that unifying DRM and V4L2 pixel formats is not possible, but I'd like
to unify the modifiers [1].

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/drm/drm_fourcc.h#n290

Should we use the DRM modifiers directly in the V4L2 API?
Or should we move this header to a common place and change the prefix? (which requires
us to sync with DRM community).
Or should we create a v4l2 header, defining V4L2_ prefixed macros mapping to DRM_
macros?

For now, patch 1/6 includes drm/drm_fourcc.h and it is using DRM_FORMAT_MOD_*

As discussed before, It would be nice to have documentation describing DRM fourcc
equivalents (I'm not sure if someone started this already), listing the number of
planes per format.

We should also document which pixelformats are valid for the EXT_API, since multiplanar
and tile versions like V4L2_PIX_FMT_NV12MT_16X16 (which seems equivalent to
DRM_FORMAT_MOD_SAMSUNG_16_16_TILE, and could have a more generic name) should be
replaced by a modifier.

Regarding flags [2] field in struct v4l2_pix_format_mplane [3]:
The only defined flag is V4L2_PIX_FMT_FLAG_PREMUL_ALPHA, and it is only used by vsp1 driver.
Which I believe could be replaced by a modifier, to avoid another field that changes
pixel formats, so I removed it from the EXT API (we can always add it back later with
the reserved fields).

[2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/pixfmt-reserved.html#format-flags
[3] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/pixfmt-v4l2-mplane.html?highlight=v4l2_pix_format_mplane#c.v4l2_pix_format_mplane

We also discussed to add a new ENUM_FMT_EXT ioctl to return all pixelformats + modifiers
combinations. I still didn't add it in this version, but I don't think it affects
what is in this RFC and it can be added later.


* Buffers/Plane offset
---------------------------------------------------------------------------------

My understanding is that inside a memory buffer we can have multiple planes in random
offsets.
I was comparing with the DRM API [4], where it can have the same dmabuf for multiple
planes in different offsets, and I started to think we could simplify our API, so
I took the liberty to do some more changes, please review struct v4l2_ext_plane in
this RFC.

I removed the data_offset, since it is unused (See Laurent's RFC repurposing this
field [5]). And comparing to the DRM API, it seems to me we only need a single offset
field.

We could also check about overlapping planes in a memory buffer, but this is complicated
if we use the same memory buffer with different v4l2_ext_buffer objects. We can also leave
to the driver to check situations that may cause HW errors.

[4] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/drm/drm_mode.h#n489
[5] https://patchwork.linuxtv.org/patch/29177/


* Multistream Channels
---------------------------------------------------------------------------------
During last media summit, we discussed about adding a channel number to the API to
support multistreams. i.e, to have multiple queues through a single video node.

Use cases:

    - Blitters: can take multiple streams as input, which would require multiple OUTPUT queues.

    As Nicolas was explaining me:
    "The blitters comes with a lot of variation between hardware. Most blitters at
    least support 3 frames buffer. 2 inputs and one output. The second input is usually
    optional, as the output buffer data is not always overwritten (e.g. SRC_OVER
    blend or 1 input). Some of them have additional solid color or pattern that can
    be used too. Advanced blitters will have composition feature, and may support more
    input buffers to reduce the added latency that would be normally done through cascading
    the operations. Note that each input can have different size and different cropping
    region. Many blitters can scale and render to a sub-region of the CAPTURE buffer."

    - Multis-calers: can produce multiple streams, which would require multiple CAPTURE queues.

    As Nicolas was explaining me:
    "This type of HW (or soft IP) is commonly found on HW used to produce internet
    streams for fragmented and scalable protocols (HLS, DASH).  Basically they are
    used to transform one stream into multiple sized streams prior from being encoded."

Modeling as channels allows the API to have synchronized Start/Stop between queues,
and also avoid the complexity of using the Media API in a topology with multiple video
nodes, which complicates userspace.

This requires adding a new channel id in ioctls for formats (G_FMT/S_FMT/TRY_FMT), and
also for buffers (QBUF/DBUF).
We also need a mechanism to enumerate channels and their properties.
Since we don't have a clear view how this would work, for now I'm leaving reserved bits
in the structs, so we can add them later.


* Timecode
---------------------------------------------------------------------------------
During last media summit, we discussed to return the v4l2_timecode field to the API,
since Nicolas mentioned that, even if it is not used by any upstreamed driver, it
is used by out-of-tree drivers.

I've been discussing with Nicolas about this, and we can avoid adding too many metadata
to the buffer struct by using the Read-Only Request API [6] for retrieving more information
when required, similar to HDR.

The RO Request API has the ability to read a control using a request that has already
completed, the control value lives as long as the request object. If it's not read
(or if there was no request), the data is simply ignored/discard.

Since no upstream driver uses the timecode field, there are no conversions that need
to be done.

[6] https://patchwork.kernel.org/cover/11635927/


* Other changes (and some questions) in this version:
---------------------------------------------------------------------------------
- Added reserved fields to struct

- The only difference between previously proposed VIDIOC_EXT_EXPBUF and VIDIOC_EXPBUF,
was that with VIDIOC_EXT_EXPBUF we can export multiple planes at once. I think we
can add this later, so I removed it from this RFC to simplify it.

- v4l2_buffer [7] has a memory field (enum v4l2_memory [8]). We kept this field in
struct v4l2_ext_buffer, buf I was wondering if this shouldn't be in struct v4l2_ext_plane
instead.

[7] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/buffer.html?highlight=v4l2_buffer#c.v4l2_buffer
[8] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/buffer.html?highlight=v4l2_memory#c.v4l2_memory

- In struct v4l2_ext_pix_format, we have:

        struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];

The number of planes can be deducted from plane_fmt[i].sizeimage != 0, so I removed
the num_planes field. Please let me know if we can't use sizeimage for this.
In DRM, we know the number of planes from drm_mode_fb_cmd2 by the number of handle
args passed which are not 0.
This also avoids num_planes to be bigger then VIDEO_MAX_PLANES.

- Added flags field to struct v4l2_ext_create_buffers


* Fixed bugs here and there
---------------------------------------------------------------------------------
I fixed some bugs found with v4l2-compliance (not all of them yet),
through script v4l-utils/contrib/test/test-media.

I adapted what Boris did for v4l-utils in previous version to this version:
https://gitlab.collabora.com/koike/v4l-utils/-/tree/ext-api/wip

Boris' questions regarding DMABUF in last version still holds [9].

[9] https://patchwork.linuxtv.org/project/linux-media/cover/20191008091119.7294-1-boris.brezillon@collabora.com/


Please, let me know your feedback,
Helen


Boris Brezillon (5):
  media: v4l2: Extend pixel formats to unify single/multi-planar
    handling (and more)
  media: videobuf2: Expose helpers to implement the _ext_fmt and
    _ext_buf hooks
  media: mediabus: Add helpers to convert a ext_pix format to/from a
    mbus_fmt
  media: vivid: Convert the capture and output drivers to
    EXT_FMT/EXT_BUF
  media: vimc: Implement the ext_fmt and ext_buf hooks

Hans Verkuil (1):
  media: v4l2: Add extended buffer operations

 .../media/common/videobuf2/videobuf2-core.c   |   2 +
 .../media/common/videobuf2/videobuf2-v4l2.c   | 549 +++++-----
 .../media/test-drivers/vimc/vimc-capture.c    |  61 +-
 drivers/media/test-drivers/vimc/vimc-common.c |   6 +-
 drivers/media/test-drivers/vimc/vimc-common.h |   2 +-
 drivers/media/test-drivers/vivid/vivid-core.c |  70 +-
 .../test-drivers/vivid/vivid-touch-cap.c      |  26 +-
 .../test-drivers/vivid/vivid-touch-cap.h      |   3 +-
 .../media/test-drivers/vivid/vivid-vid-cap.c  | 169 +---
 .../media/test-drivers/vivid/vivid-vid-cap.h  |  15 +-
 .../media/test-drivers/vivid/vivid-vid-out.c  | 193 ++--
 .../media/test-drivers/vivid/vivid-vid-out.h  |  15 +-
 drivers/media/v4l2-core/v4l2-dev.c            |  50 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          | 934 ++++++++++++++++--
 include/media/v4l2-ioctl.h                    |  60 ++
 include/media/v4l2-mediabus.h                 |  42 +
 include/media/videobuf2-core.h                |   6 +-
 include/media/videobuf2-v4l2.h                |  21 +-
 include/uapi/linux/videodev2.h                | 144 +++
 19 files changed, 1650 insertions(+), 718 deletions(-)

-- 
2.26.0


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v4 1/6] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)
  2020-07-17 11:54 [PATCH v4 0/6] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
@ 2020-07-17 11:54 ` Helen Koike
  2020-07-21 10:37   ` Hans Verkuil
  2020-07-17 11:54 ` [PATCH v4 2/6] media: v4l2: Add extended buffer operations Helen Koike
                   ` (7 subsequent siblings)
  8 siblings, 1 reply; 29+ messages in thread
From: Helen Koike @ 2020-07-17 11:54 UTC (permalink / raw)
  To: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus, linux-media
  Cc: Boris Brezillon, Helen Koike, tfiga, hiroh, nicolas,
	Brian.Starkey, kernel, narmstrong, linux-kernel, frkoenig,
	mjourdan, stanimir.varbanov

From: Boris Brezillon <boris.brezillon@collabora.com>

This is part of the multiplanar and singleplanar unification process.
v4l2_ext_pix_format is supposed to work for both cases.

We also add the concept of modifiers already employed in DRM to expose
HW-specific formats (like tiled or compressed formats) and allow
exchanging this information with the DRM subsystem in a consistent way.

Note that only V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] are accepted in
v4l2_ext_format, other types will be rejected if you use the {G,S,TRY}EXT_FMT
ioctls.

New hooks have been added to v4l2_ioctl_ops to support those new ioctls
in drivers, but, in the meantime, the core takes care of converting
{S,G,TRY}_EXT_FMT requests into {S,G,TRY}_FMT so that old drivers can
still work if the userspace app/lib uses the new ioctls.
The conversion is also done the other around to allow userspace
apps/libs using {S,G,TRY}_FMT to work with drivers implementing the
_ext_ hooks.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Signed-off-by: Helen Koike <helen.koike@collabora.com>
---
Changes in v4:
- Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
- Add reserved fields
- Removed num_planes from struct v4l2_ext_pix_format
- Removed flag field from struct v4l2_ext_pix_format, since the only
  defined value is V4L2_PIX_FMT_FLAG_PREMUL_ALPHA only used by vsp1,
  where we can use modifiers, or add it back later through the reserved
  bits.
- In v4l2_ext_format_to_format(), check if modifier is != MOD_LINEAR &&
  != MOD_INVALID
- Fix type assignment in v4l_g_fmt_ext_pix()
- Rebased on top of media/master (post 5.8-rc1)

Changes in v3:
- Rebased on top of media/master (post 5.4-rc1)

Changes in v2:
- Move the modifier in v4l2_ext_format (was formerly placed in
  v4l2_ext_plane)
- Fix a few bugs in the converters and add a strict parameter to
  allow conversion of uninitialized/mis-initialized objects
---
 drivers/media/v4l2-core/v4l2-dev.c   |  21 +-
 drivers/media/v4l2-core/v4l2-ioctl.c | 585 +++++++++++++++++++++++----
 include/media/v4l2-ioctl.h           |  34 ++
 include/uapi/linux/videodev2.h       |  55 +++
 4 files changed, 614 insertions(+), 81 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
index a593ea0598b55..e1829906bc086 100644
--- a/drivers/media/v4l2-core/v4l2-dev.c
+++ b/drivers/media/v4l2-core/v4l2-dev.c
@@ -607,25 +607,37 @@ static void determine_valid_ioctls(struct video_device *vdev)
 			set_bit(_IOC_NR(VIDIOC_ENUM_FMT), valid_ioctls);
 		if ((is_rx && (ops->vidioc_g_fmt_vid_cap ||
 			       ops->vidioc_g_fmt_vid_cap_mplane ||
+			       ops->vidioc_g_ext_pix_fmt_vid_cap ||
 			       ops->vidioc_g_fmt_vid_overlay)) ||
 		    (is_tx && (ops->vidioc_g_fmt_vid_out ||
 			       ops->vidioc_g_fmt_vid_out_mplane ||
-			       ops->vidioc_g_fmt_vid_out_overlay)))
+			       ops->vidioc_g_ext_pix_fmt_vid_out ||
+			       ops->vidioc_g_fmt_vid_out_overlay))) {
 			 set_bit(_IOC_NR(VIDIOC_G_FMT), valid_ioctls);
+			 set_bit(_IOC_NR(VIDIOC_G_EXT_PIX_FMT), valid_ioctls);
+		}
 		if ((is_rx && (ops->vidioc_s_fmt_vid_cap ||
 			       ops->vidioc_s_fmt_vid_cap_mplane ||
+			       ops->vidioc_s_ext_pix_fmt_vid_cap ||
 			       ops->vidioc_s_fmt_vid_overlay)) ||
 		    (is_tx && (ops->vidioc_s_fmt_vid_out ||
 			       ops->vidioc_s_fmt_vid_out_mplane ||
-			       ops->vidioc_s_fmt_vid_out_overlay)))
+			       ops->vidioc_s_ext_pix_fmt_vid_out ||
+			       ops->vidioc_s_fmt_vid_out_overlay))) {
 			 set_bit(_IOC_NR(VIDIOC_S_FMT), valid_ioctls);
+			 set_bit(_IOC_NR(VIDIOC_S_EXT_PIX_FMT), valid_ioctls);
+		}
 		if ((is_rx && (ops->vidioc_try_fmt_vid_cap ||
 			       ops->vidioc_try_fmt_vid_cap_mplane ||
+			       ops->vidioc_try_ext_pix_fmt_vid_cap ||
 			       ops->vidioc_try_fmt_vid_overlay)) ||
 		    (is_tx && (ops->vidioc_try_fmt_vid_out ||
 			       ops->vidioc_try_fmt_vid_out_mplane ||
-			       ops->vidioc_try_fmt_vid_out_overlay)))
+			       ops->vidioc_try_ext_pix_fmt_vid_out ||
+			       ops->vidioc_try_fmt_vid_out_overlay))) {
 			 set_bit(_IOC_NR(VIDIOC_TRY_FMT), valid_ioctls);
+			 set_bit(_IOC_NR(VIDIOC_TRY_EXT_PIX_FMT), valid_ioctls);
+		}
 		SET_VALID_IOCTL(ops, VIDIOC_OVERLAY, vidioc_overlay);
 		SET_VALID_IOCTL(ops, VIDIOC_G_FBUF, vidioc_g_fbuf);
 		SET_VALID_IOCTL(ops, VIDIOC_S_FBUF, vidioc_s_fbuf);
@@ -682,8 +694,11 @@ static void determine_valid_ioctls(struct video_device *vdev)
 		/* touch specific ioctls */
 		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FMT, vidioc_enum_fmt_vid_cap);
 		SET_VALID_IOCTL(ops, VIDIOC_G_FMT, vidioc_g_fmt_vid_cap);
+		SET_VALID_IOCTL(ops, VIDIOC_G_EXT_PIX_FMT, vidioc_g_fmt_vid_cap);
 		SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_vid_cap);
+		SET_VALID_IOCTL(ops, VIDIOC_S_EXT_PIX_FMT, vidioc_s_fmt_vid_cap);
 		SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_vid_cap);
+		SET_VALID_IOCTL(ops, VIDIOC_TRY_EXT_PIX_FMT, vidioc_try_fmt_vid_cap);
 		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMESIZES, vidioc_enum_framesizes);
 		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMEINTERVALS, vidioc_enum_frameintervals);
 		SET_VALID_IOCTL(ops, VIDIOC_ENUMINPUT, vidioc_enum_input);
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 02bfef0da76da..3b77433f6c32b 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -17,6 +17,8 @@
 
 #include <linux/videodev2.h>
 
+#include <drm/drm_fourcc.h>
+
 #include <media/v4l2-common.h>
 #include <media/v4l2-ioctl.h>
 #include <media/v4l2-ctrls.h>
@@ -378,6 +380,27 @@ static void v4l_print_format(const void *arg, bool write_only)
 	}
 }
 
+static void v4l_print_ext_pix_format(const void *arg, bool write_only)
+{
+	const struct v4l2_ext_pix_format *pix = arg;
+	unsigned int i;
+
+	pr_cont("type=%s, width=%u, height=%u, format=%c%c%c%c, modifier %llx, field=%s, colorspace=%d, ycbcr_enc=%u, quantization=%u, xfer_func=%u\n",
+		prt_names(pix->type, v4l2_type_names),
+		pix->width, pix->height,
+		(pix->pixelformat & 0xff),
+		(pix->pixelformat >>  8) & 0xff,
+		(pix->pixelformat >> 16) & 0xff,
+		(pix->pixelformat >> 24) & 0xff,
+		pix->modifier, prt_names(pix->field, v4l2_field_names),
+		pix->colorspace, pix->ycbcr_enc,
+		pix->quantization, pix->xfer_func);
+	for (i = 0; i < VIDEO_MAX_PLANES && pix->plane_fmt[i].sizeimage; i++)
+		pr_debug("plane %u: bytesperline=%u sizeimage=%u\n",
+			 i, pix->plane_fmt[i].bytesperline,
+			 pix->plane_fmt[i].sizeimage);
+}
+
 static void v4l_print_framebuffer(const void *arg, bool write_only)
 {
 	const struct v4l2_framebuffer *p = arg;
@@ -958,11 +981,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
 	switch (type) {
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
 		if ((is_vid || is_tch) && is_rx &&
-		    (ops->vidioc_g_fmt_vid_cap || ops->vidioc_g_fmt_vid_cap_mplane))
+		    (ops->vidioc_g_fmt_vid_cap ||
+		     ops->vidioc_g_ext_pix_fmt_vid_cap ||
+		     ops->vidioc_g_fmt_vid_cap_mplane))
 			return 0;
 		break;
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
-		if ((is_vid || is_tch) && is_rx && ops->vidioc_g_fmt_vid_cap_mplane)
+		if ((is_vid || is_tch) && is_rx &&
+		    (ops->vidioc_g_fmt_vid_cap_mplane ||
+		     ops->vidioc_g_ext_pix_fmt_vid_cap))
 			return 0;
 		break;
 	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
@@ -971,11 +998,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
 		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
 		if (is_vid && is_tx &&
-		    (ops->vidioc_g_fmt_vid_out || ops->vidioc_g_fmt_vid_out_mplane))
+		    (ops->vidioc_g_fmt_vid_out ||
+		     ops->vidioc_g_ext_pix_fmt_vid_out ||
+		     ops->vidioc_g_fmt_vid_out_mplane))
 			return 0;
 		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
-		if (is_vid && is_tx && ops->vidioc_g_fmt_vid_out_mplane)
+		if (is_vid && is_tx &&
+		    (ops->vidioc_g_ext_pix_fmt_vid_out ||
+		     ops->vidioc_g_fmt_vid_out_mplane))
 			return 0;
 		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
@@ -1055,6 +1086,134 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
 	       sizeof(fmt->fmt.pix) - offset);
 }
 
+int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
+				  struct v4l2_format *f, bool mplane_cap,
+				  bool strict)
+{
+	const struct v4l2_plane_ext_pix_format *pe;
+	struct v4l2_plane_pix_format *p;
+	unsigned int i;
+
+	memset(f, 0, sizeof(*f));
+
+	/*
+	 * Make sure no modifier is required before doing the
+	 * conversion.
+	 */
+	if (e->modifier && strict &&
+	    e->modifier != DRM_FORMAT_MOD_LINEAR &&
+	    e->modifier != DRM_FORMAT_MOD_INVALID)
+		return -EINVAL;
+
+	if (!e->plane_fmt[0].sizeimage && strict)
+		return -EINVAL;
+
+	if (e->plane_fmt[1].sizeimage && !mplane_cap && strict)
+		return 0;
+
+	if (!mplane_cap) {
+		f->fmt.pix.width = e->width;
+		f->fmt.pix.height = e->height;
+		f->fmt.pix.pixelformat = e->pixelformat;
+		f->fmt.pix.field = e->field;
+		f->fmt.pix.colorspace = e->colorspace;
+		f->fmt.pix.ycbcr_enc = e->ycbcr_enc;
+		f->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+		f->fmt.pix.quantization = e->quantization;
+		pe = &e->plane_fmt[0];
+		f->fmt.pix.bytesperline = pe->bytesperline;
+		f->fmt.pix.sizeimage = pe->sizeimage;
+		f->type = e->type;
+		return 0;
+	}
+
+	f->fmt.pix_mp.width = e->width;
+	f->fmt.pix_mp.height = e->height;
+	f->fmt.pix_mp.pixelformat = e->pixelformat;
+	f->fmt.pix_mp.field = e->field;
+	f->fmt.pix_mp.colorspace = e->colorspace;
+	f->fmt.pix_mp.ycbcr_enc = e->ycbcr_enc;
+	f->fmt.pix_mp.quantization = e->quantization;
+	if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		f->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
+	else
+		f->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
+
+	for (i = 0; i < VIDEO_MAX_PLANES; i++) {
+		pe = &e->plane_fmt[i];
+		p = &f->fmt.pix_mp.plane_fmt[i];
+		p->bytesperline = pe->bytesperline;
+		p->sizeimage = pe->sizeimage;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_ext_pix_format_to_format);
+
+int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
+				  struct v4l2_ext_pix_format *e, bool strict)
+{
+	const struct v4l2_plane_pix_format *p;
+	struct v4l2_plane_ext_pix_format *pe;
+	unsigned int i;
+
+	memset(e, 0, sizeof(*e));
+
+	switch (f->type) {
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+		e->width = f->fmt.pix.width;
+		e->height = f->fmt.pix.height;
+		e->pixelformat = f->fmt.pix.pixelformat;
+		e->field = f->fmt.pix.field;
+		e->colorspace = f->fmt.pix.colorspace;
+		if (f->fmt.pix.flags)
+			pr_warn("Ignoring pixelformat flags 0x%x\n",
+				f->fmt.pix.flags);
+		e->ycbcr_enc = f->fmt.pix.ycbcr_enc;
+		e->quantization = f->fmt.pix.quantization;
+		e->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
+		e->plane_fmt[0].sizeimage = f->fmt.pix.sizeimage;
+		e->type = f->type;
+		break;
+
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+		if ((f->fmt.pix_mp.num_planes > VIDEO_MAX_PLANES ||
+		     !f->fmt.pix_mp.num_planes) && strict)
+			return -EINVAL;
+
+		e->width = f->fmt.pix_mp.width;
+		e->height = f->fmt.pix_mp.height;
+		e->pixelformat = f->fmt.pix_mp.pixelformat;
+		e->field = f->fmt.pix_mp.field;
+		e->colorspace = f->fmt.pix_mp.colorspace;
+		if (f->fmt.pix.flags)
+			pr_warn("Ignoring pixelformat flags 0x%x\n",
+				f->fmt.pix.flags);
+		e->ycbcr_enc = f->fmt.pix_mp.ycbcr_enc;
+		e->quantization = f->fmt.pix_mp.quantization;
+		if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
+			e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+		else
+			e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
+
+		for (i = 0; i < VIDEO_MAX_PLANES; i++) {
+			pe = &e->plane_fmt[i];
+			p = &f->fmt.pix_mp.plane_fmt[i];
+			pe->bytesperline = p->bytesperline;
+			pe->sizeimage = p->sizeimage;
+		}
+		break;
+
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_format_to_ext_pix_format);
+
 static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
 				struct file *file, void *fh, void *arg)
 {
@@ -1558,6 +1717,38 @@ static void v4l_pix_format_touch(struct v4l2_pix_format *p)
 	p->xfer_func = 0;
 }
 
+static int v4l_g_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
+			     struct file *file, void *fh,
+			     struct v4l2_format *f)
+{
+	struct v4l2_ext_pix_format ef;
+	int ret;
+
+	switch (f->type) {
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
+		ef.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+		ret = ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, &ef);
+		break;
+
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+		ef.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
+		ret = ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, &ef);
+		break;
+
+	default:
+		return -EINVAL;
+	}
+
+	if (ret)
+		return ret;
+
+	return v4l2_ext_pix_format_to_format(&ef, f,
+					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
+					     true);
+}
+
 static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
 				struct file *file, void *fh, void *arg)
 {
@@ -1594,17 +1785,27 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
 
 	switch (p->type) {
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
-		if (unlikely(!ops->vidioc_g_fmt_vid_cap))
-			break;
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
-		ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
-		/* just in case the driver zeroed it again */
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
-		if (vfd->vfl_type == VFL_TYPE_TOUCH)
-			v4l_pix_format_touch(&p->fmt.pix);
-		return ret;
+		if (ops->vidioc_g_fmt_vid_cap) {
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+			ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
+			/* just in case the driver zeroed it again */
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+			if (vfd->vfl_type == VFL_TYPE_TOUCH)
+				v4l_pix_format_touch(&p->fmt.pix);
+			return ret;
+		} else if (ops->vidioc_g_ext_pix_fmt_vid_cap) {
+			ret = v4l_g_fmt_ext_pix(ops, file, fh, p);
+			if (vfd->vfl_type == VFL_TYPE_TOUCH)
+				v4l_pix_format_touch(&p->fmt.pix);
+			return ret;
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
-		return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
+		if (ops->vidioc_g_fmt_vid_cap_mplane)
+			return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
+		else if (ops->vidioc_g_ext_pix_fmt_vid_cap)
+			return v4l_g_fmt_ext_pix(ops, file, fh, p);
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
 		return ops->vidioc_g_fmt_vid_overlay(file, fh, arg);
 	case V4L2_BUF_TYPE_VBI_CAPTURE:
@@ -1612,15 +1813,22 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
 	case V4L2_BUF_TYPE_SLICED_VBI_CAPTURE:
 		return ops->vidioc_g_fmt_sliced_vbi_cap(file, fh, arg);
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
-		if (unlikely(!ops->vidioc_g_fmt_vid_out))
-			break;
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
-		ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
-		/* just in case the driver zeroed it again */
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
-		return ret;
+		if (ops->vidioc_g_fmt_vid_out) {
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+			ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
+			/* just in case the driver zeroed it again */
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+			return ret;
+		} else if (ops->vidioc_g_ext_pix_fmt_vid_out) {
+			return v4l_g_fmt_ext_pix(ops, file, fh, p);
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
-		return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
+		if (ops->vidioc_g_fmt_vid_out_mplane)
+			return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
+		else if (ops->vidioc_g_ext_pix_fmt_vid_out)
+			return v4l_g_fmt_ext_pix(ops, file, fh, p);
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
 		return ops->vidioc_g_fmt_vid_out_overlay(file, fh, arg);
 	case V4L2_BUF_TYPE_VBI_OUTPUT:
@@ -1639,6 +1847,76 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
 	return -EINVAL;
 }
 
+static int v4l_g_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
+			     struct file *file, void *fh, void *arg)
+{
+	struct v4l2_ext_pix_format *ef = arg;
+	struct v4l2_format f = {
+		.type = ef->type,
+	};
+	int ret;
+
+	ret = check_fmt(file, ef->type);
+	if (ret)
+		return ret;
+
+	memset(ef, 0, sizeof(*ef));
+	ef->type = f.type;
+
+	switch (f.type) {
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+		if (ops->vidioc_g_ext_pix_fmt_vid_cap)
+			return ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, ef);
+		break;
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+		if (ops->vidioc_g_ext_pix_fmt_vid_out)
+			return ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, ef);
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	ret = v4l_g_fmt(ops, file, fh, &f);
+	if (ret)
+		return ret;
+
+	return v4l2_format_to_ext_pix_format(&f, ef, true);
+}
+
+static int v4l_s_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
+			     struct file *file, void *fh,
+			     struct v4l2_format *f)
+{
+	struct v4l2_ext_pix_format ef;
+	int ret;
+
+	ret = v4l2_format_to_ext_pix_format(f, &ef, false);
+	if (ret)
+		return ret;
+
+	switch (f->type) {
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
+		ret = ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, &ef);
+		break;
+
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+		ret = ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, &ef);
+		break;
+
+	default:
+		return -EINVAL;
+	}
+
+	if (ret)
+		return ret;
+
+	return v4l2_ext_pix_format_to_format(&ef, f,
+					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
+					     true);
+}
+
 static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
 				struct file *file, void *fh, void *arg)
 {
@@ -1657,23 +1935,31 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
 
 	switch (p->type) {
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
-		if (unlikely(!ops->vidioc_s_fmt_vid_cap))
+		if (ops->vidioc_s_fmt_vid_cap) {
+			CLEAR_AFTER_FIELD(p, fmt.pix);
+			ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
+			/* just in case the driver zeroed it again */
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+		} else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
+			ret = v4l_s_fmt_ext_pix(ops, file, fh, arg);
+		} else {
 			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix);
-		ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
-		/* just in case the driver zeroed it again */
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+		}
+
 		if (vfd->vfl_type == VFL_TYPE_TOUCH)
 			v4l_pix_format_touch(&p->fmt.pix);
 		return ret;
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
-		if (unlikely(!ops->vidioc_s_fmt_vid_cap_mplane))
-			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
-		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
-			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
-					  bytesperline);
-		return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
+		if (ops->vidioc_s_fmt_vid_cap_mplane) {
+			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
+			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
+				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
+						  bytesperline);
+			return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
+		} else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
+			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
 		if (unlikely(!ops->vidioc_s_fmt_vid_overlay))
 			break;
@@ -1690,21 +1976,27 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
 		CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
 		return ops->vidioc_s_fmt_sliced_vbi_cap(file, fh, arg);
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
-		if (unlikely(!ops->vidioc_s_fmt_vid_out))
-			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix);
-		ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
-		/* just in case the driver zeroed it again */
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
-		return ret;
+		if (ops->vidioc_s_fmt_vid_out) {
+			CLEAR_AFTER_FIELD(p, fmt.pix);
+			ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
+			/* just in case the driver zeroed it again */
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+			return ret;
+		} else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
+			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
-		if (unlikely(!ops->vidioc_s_fmt_vid_out_mplane))
-			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
-		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
-			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
-					  bytesperline);
-		return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
+		if (ops->vidioc_s_fmt_vid_out_mplane) {
+			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
+			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
+				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
+						  bytesperline);
+			return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
+		} else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
+			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
 		if (unlikely(!ops->vidioc_s_fmt_vid_out_overlay))
 			break;
@@ -1744,6 +2036,82 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
 	return -EINVAL;
 }
 
+static int v4l_s_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
+			     struct file *file, void *fh, void *arg)
+{
+	struct video_device *vfd = video_devdata(file);
+	struct v4l2_ext_pix_format *ef = arg;
+	struct v4l2_format f;
+	int ret;
+
+	ret = check_fmt(file, ef->type);
+	if (ret)
+		return ret;
+
+	switch (ef->type) {
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+		if (ops->vidioc_s_ext_pix_fmt_vid_cap)
+			return ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, ef);
+		break;
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+		if (ops->vidioc_s_ext_pix_fmt_vid_out)
+			return ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, ef);
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	ret = v4l2_ext_pix_format_to_format(ef, &f,
+					    vfd->device_caps &
+					    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
+					     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
+					     V4L2_CAP_VIDEO_M2M_MPLANE),
+					    false);
+	if (ret)
+		return ret;
+
+	ret = v4l_s_fmt(ops, file, fh, &f);
+	if (ret)
+		return ret;
+
+	return v4l2_format_to_ext_pix_format(&f, ef, true);
+}
+
+static int v4l_try_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
+			       struct file *file, void *fh,
+			       struct v4l2_format *f)
+{
+	struct v4l2_ext_pix_format ef;
+	int ret;
+
+	ret = v4l2_format_to_ext_pix_format(f, &ef, false);
+	if (ret)
+		return ret;
+
+	switch (f->type) {
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
+		ret = ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh, &ef);
+		break;
+
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+		ret = ops->vidioc_try_ext_pix_fmt_vid_out(file, fh, &ef);
+		break;
+
+	default:
+		return -EINVAL;
+	}
+
+	if (ret)
+		return ret;
+
+	return v4l2_ext_pix_format_to_format(&ef, f,
+					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
+					     true);
+}
+
+
 static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
 				struct file *file, void *fh, void *arg)
 {
@@ -1759,23 +2127,32 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
 
 	switch (p->type) {
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
-		if (unlikely(!ops->vidioc_try_fmt_vid_cap))
-			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix);
-		ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
-		/* just in case the driver zeroed it again */
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
-		if (vfd->vfl_type == VFL_TYPE_TOUCH)
-			v4l_pix_format_touch(&p->fmt.pix);
-		return ret;
+		if (ops->vidioc_try_fmt_vid_cap) {
+			CLEAR_AFTER_FIELD(p, fmt.pix);
+			ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
+			/* just in case the driver zeroed it again */
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+			if (vfd->vfl_type == VFL_TYPE_TOUCH)
+				v4l_pix_format_touch(&p->fmt.pix);
+			return ret;
+		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
+			ret = v4l_try_fmt_ext_pix(ops, file, fh, p);
+			if (vfd->vfl_type == VFL_TYPE_TOUCH)
+				v4l_pix_format_touch(&p->fmt.pix);
+			return ret;
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
-		if (unlikely(!ops->vidioc_try_fmt_vid_cap_mplane))
-			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
-		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
-			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
-					  bytesperline);
-		return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
+		if (ops->vidioc_try_fmt_vid_cap_mplane) {
+			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
+			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
+				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
+						  bytesperline);
+			return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
+		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
+			return v4l_try_fmt_ext_pix(ops, file, fh, p);
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
 		if (unlikely(!ops->vidioc_try_fmt_vid_overlay))
 			break;
@@ -1792,21 +2169,27 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
 		CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
 		return ops->vidioc_try_fmt_sliced_vbi_cap(file, fh, arg);
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
-		if (unlikely(!ops->vidioc_try_fmt_vid_out))
-			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix);
-		ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
-		/* just in case the driver zeroed it again */
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
-		return ret;
+		if (ops->vidioc_try_fmt_vid_out) {
+			CLEAR_AFTER_FIELD(p, fmt.pix);
+			ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
+			/* just in case the driver zeroed it again */
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+			return ret;
+		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
+			return v4l_try_fmt_ext_pix(ops, file, fh, p);
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
-		if (unlikely(!ops->vidioc_try_fmt_vid_out_mplane))
-			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
-		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
-			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
-					  bytesperline);
-		return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
+		if (ops->vidioc_try_fmt_vid_out_mplane) {
+			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
+			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
+				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
+						  bytesperline);
+			return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
+		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
+			return v4l_try_fmt_ext_pix(ops, file, fh, p);
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
 		if (unlikely(!ops->vidioc_try_fmt_vid_out_overlay))
 			break;
@@ -1846,6 +2229,49 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
 	return -EINVAL;
 }
 
+static int v4l_try_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
+			       struct file *file, void *fh, void *arg)
+{
+	struct video_device *vfd = video_devdata(file);
+	struct v4l2_ext_pix_format *ef = arg;
+	struct v4l2_format f;
+	int ret;
+
+	ret = check_fmt(file, ef->type);
+	if (ret)
+		return ret;
+
+	switch (ef->type) {
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+		if (ops->vidioc_try_ext_pix_fmt_vid_cap)
+			return ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh,
+								   ef);
+		break;
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+		if (ops->vidioc_try_ext_pix_fmt_vid_out)
+			return ops->vidioc_try_ext_pix_fmt_vid_out(file, fh,
+								   ef);
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	ret = v4l2_ext_pix_format_to_format(ef, &f,
+					    vfd->device_caps &
+					    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
+					     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
+					     V4L2_CAP_VIDEO_M2M_MPLANE),
+					    false);
+	if (ret)
+		return ret;
+
+	ret = v4l_try_fmt(ops, file, fh, &f);
+	if (ret)
+		return ret;
+
+	return v4l2_format_to_ext_pix_format(&f, ef, true);
+}
+
 static int v4l_streamon(const struct v4l2_ioctl_ops *ops,
 				struct file *file, void *fh, void *arg)
 {
@@ -2765,7 +3191,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
 	IOCTL_INFO(VIDIOC_QUERYCAP, v4l_querycap, v4l_print_querycap, 0),
 	IOCTL_INFO(VIDIOC_ENUM_FMT, v4l_enum_fmt, v4l_print_fmtdesc, 0),
 	IOCTL_INFO(VIDIOC_G_FMT, v4l_g_fmt, v4l_print_format, 0),
+	IOCTL_INFO(VIDIOC_G_EXT_PIX_FMT, v4l_g_ext_pix_fmt, v4l_print_ext_pix_format, 0),
 	IOCTL_INFO(VIDIOC_S_FMT, v4l_s_fmt, v4l_print_format, INFO_FL_PRIO),
+	IOCTL_INFO(VIDIOC_S_EXT_PIX_FMT, v4l_s_ext_pix_fmt, v4l_print_ext_pix_format, INFO_FL_PRIO),
 	IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
 	IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
@@ -2812,6 +3240,7 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
 	IOCTL_INFO(VIDIOC_S_JPEGCOMP, v4l_stub_s_jpegcomp, v4l_print_jpegcompression, INFO_FL_PRIO),
 	IOCTL_INFO(VIDIOC_QUERYSTD, v4l_querystd, v4l_print_std, 0),
 	IOCTL_INFO(VIDIOC_TRY_FMT, v4l_try_fmt, v4l_print_format, 0),
+	IOCTL_INFO(VIDIOC_TRY_EXT_PIX_FMT, v4l_try_ext_pix_fmt, v4l_print_ext_pix_format, 0),
 	IOCTL_INFO(VIDIOC_ENUMAUDIO, v4l_stub_enumaudio, v4l_print_audio, INFO_FL_CLEAR(v4l2_audio, index)),
 	IOCTL_INFO(VIDIOC_ENUMAUDOUT, v4l_stub_enumaudout, v4l_print_audioout, INFO_FL_CLEAR(v4l2_audioout, index)),
 	IOCTL_INFO(VIDIOC_G_PRIORITY, v4l_g_priority, v4l_print_u32, 0),
diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
index 86878fba332b0..525ce86725260 100644
--- a/include/media/v4l2-ioctl.h
+++ b/include/media/v4l2-ioctl.h
@@ -48,11 +48,17 @@ struct v4l2_fh;
  * @vidioc_g_fmt_vid_cap: pointer to the function that implements
  *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video capture
  *	in single plane mode
+ * @vidioc_g_ext_pix_fmt_vid_cap: pointer to the function that implements
+ *	:ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
+ *	capture
  * @vidioc_g_fmt_vid_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay
  * @vidioc_g_fmt_vid_out: pointer to the function that implements
  *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video out
  *	in single plane mode
+ * @vidioc_g_ext_pix_fmt_vid_out: pointer to the function that implements
+ *	:ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
+ *	out
  * @vidioc_g_fmt_vid_out_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
  * @vidioc_g_fmt_vbi_cap: pointer to the function that implements
@@ -82,11 +88,16 @@ struct v4l2_fh;
  * @vidioc_s_fmt_vid_cap: pointer to the function that implements
  *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video capture
  *	in single plane mode
+ * @vidioc_s_ext_pix_fmt_vid_cap: pointer to the function that implements
+ *	:ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_s_ext_pix_fmt>` ioctl logic for video
+ *	capture
  * @vidioc_s_fmt_vid_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay
  * @vidioc_s_fmt_vid_out: pointer to the function that implements
  *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video out
  *	in single plane mode
+ * @vidioc_s_ext_pix_fmt_vid_out: pointer to the function that implements
+ *	:ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
  * @vidioc_s_fmt_vid_out_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
  * @vidioc_s_fmt_vbi_cap: pointer to the function that implements
@@ -116,11 +127,16 @@ struct v4l2_fh;
  * @vidioc_try_fmt_vid_cap: pointer to the function that implements
  *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video capture
  *	in single plane mode
+ * @vidioc_try_ext_pix_fmt_vid_cap: pointer to the function that implements
+ *	:ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_try_ext_pix_fmt>` ioctl logic for
+	video capture
  * @vidioc_try_fmt_vid_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
  * @vidioc_try_fmt_vid_out: pointer to the function that implements
  *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video out
  *	in single plane mode
+ * @vidioc_try_ext_pix_fmt_vid_out: pointer to the function that implements
+ *	:ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
  * @vidioc_try_fmt_vid_out_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
  *	output
@@ -319,10 +335,14 @@ struct v4l2_ioctl_ops {
 	/* VIDIOC_G_FMT handlers */
 	int (*vidioc_g_fmt_vid_cap)(struct file *file, void *fh,
 				    struct v4l2_format *f);
+	int (*vidioc_g_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
+					    struct v4l2_ext_pix_format *f);
 	int (*vidioc_g_fmt_vid_overlay)(struct file *file, void *fh,
 					struct v4l2_format *f);
 	int (*vidioc_g_fmt_vid_out)(struct file *file, void *fh,
 				    struct v4l2_format *f);
+	int (*vidioc_g_ext_pix_fmt_vid_out)(struct file *file, void *fh,
+					    struct v4l2_ext_pix_format *f);
 	int (*vidioc_g_fmt_vid_out_overlay)(struct file *file, void *fh,
 					    struct v4l2_format *f);
 	int (*vidioc_g_fmt_vbi_cap)(struct file *file, void *fh,
@@ -349,10 +369,14 @@ struct v4l2_ioctl_ops {
 	/* VIDIOC_S_FMT handlers */
 	int (*vidioc_s_fmt_vid_cap)(struct file *file, void *fh,
 				    struct v4l2_format *f);
+	int (*vidioc_s_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
+					    struct v4l2_ext_pix_format *f);
 	int (*vidioc_s_fmt_vid_overlay)(struct file *file, void *fh,
 					struct v4l2_format *f);
 	int (*vidioc_s_fmt_vid_out)(struct file *file, void *fh,
 				    struct v4l2_format *f);
+	int (*vidioc_s_ext_pix_fmt_vid_out)(struct file *file, void *fh,
+					    struct v4l2_ext_pix_format *f);
 	int (*vidioc_s_fmt_vid_out_overlay)(struct file *file, void *fh,
 					    struct v4l2_format *f);
 	int (*vidioc_s_fmt_vbi_cap)(struct file *file, void *fh,
@@ -379,10 +403,14 @@ struct v4l2_ioctl_ops {
 	/* VIDIOC_TRY_FMT handlers */
 	int (*vidioc_try_fmt_vid_cap)(struct file *file, void *fh,
 				      struct v4l2_format *f);
+	int (*vidioc_try_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
+					      struct v4l2_ext_pix_format *f);
 	int (*vidioc_try_fmt_vid_overlay)(struct file *file, void *fh,
 					  struct v4l2_format *f);
 	int (*vidioc_try_fmt_vid_out)(struct file *file, void *fh,
 				      struct v4l2_format *f);
+	int (*vidioc_try_ext_pix_fmt_vid_out)(struct file *file, void *fh,
+					      struct v4l2_ext_pix_format *f);
 	int (*vidioc_try_fmt_vid_out_overlay)(struct file *file, void *fh,
 					     struct v4l2_format *f);
 	int (*vidioc_try_fmt_vbi_cap)(struct file *file, void *fh,
@@ -724,6 +752,12 @@ long int video_usercopy(struct file *file, unsigned int cmd,
 long int video_ioctl2(struct file *file,
 		      unsigned int cmd, unsigned long int arg);
 
+int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
+				  struct v4l2_ext_pix_format *e, bool strict);
+int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
+				  struct v4l2_format *f,
+				  bool mplane_cap, bool strict);
+
 /*
  * The user space interpretation of the 'v4l2_event' differs
  * based on the 'time_t' definition on 32-bit architectures, so
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 303805438814f..fc04c81ce7713 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -2252,6 +2252,57 @@ struct v4l2_pix_format_mplane {
 	__u8				reserved[7];
 } __attribute__ ((packed));
 
+/**
+ * struct v4l2_plane_ext_pix_format - additional, per-plane format definition
+ * @sizeimage:		maximum size in bytes required for data, for which
+ *			this plane will be used
+ * @bytesperline:	distance in bytes between the leftmost pixels in two
+ *			adjacent lines
+ * @reserved:		extra space reserved for future fields, must be set to 0
+ */
+struct v4l2_plane_ext_pix_format {
+	__u32 sizeimage;
+	__u32 bytesperline;
+	__u32 reserved;
+} __attribute__ ((packed));
+
+/**
+ * struct v4l2_ext_pix_format - extended single/multiplanar format definition
+ * @type:		type of the data stream; V4L2_BUF_TYPE_VIDEO_CAPTURE or
+ *			V4L2_BUF_TYPE_VIDEO_OUTPUT
+ * @width:		image width in pixels
+ * @height:		image height in pixels
+ * @field:		enum v4l2_field; field order (for interlaced video)
+ * @pixelformat:	little endian four character code (fourcc)
+ * @modifier:		modifier applied to the format (used for tiled formats
+ *			and other kind of HW-specific formats, like compressed
+ *			formats)
+ * @colorspace:		enum v4l2_colorspace; supplemental to pixelformat
+ * @plane_fmt:		per-plane information
+ * @ycbcr_enc:		enum v4l2_ycbcr_encoding, Y'CbCr encoding
+ * @hsv_enc:		enum v4l2_hsv_encoding, HSV encoding
+ * @quantization:	enum v4l2_quantization, colorspace quantization
+ * @xfer_func:		enum v4l2_xfer_func, colorspace transfer function
+ * @reserved:		extra space reserved for future fields, must be set to 0
+ */
+struct v4l2_ext_pix_format {
+	__u32 type;
+	__u32 width;
+	__u32 height;
+	__u32 field;
+	__u32 pixelformat;
+	__u64 modifier;
+	__u32 colorspace;
+	struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];
+	union {
+		__u8 ycbcr_enc;
+		__u8 hsv_enc;
+	};
+	__u8 quantization;
+	__u8 xfer_func;
+	__u32 reserved[4];
+} __attribute__ ((packed));
+
 /**
  * struct v4l2_sdr_format - SDR format definition
  * @pixelformat:	little endian four character code (fourcc)
@@ -2569,6 +2620,10 @@ struct v4l2_create_buffers {
 
 #define VIDIOC_QUERY_EXT_CTRL	_IOWR('V', 103, struct v4l2_query_ext_ctrl)
 
+#define VIDIOC_G_EXT_PIX_FMT	_IOWR('V', 104, struct v4l2_ext_pix_format)
+#define VIDIOC_S_EXT_PIX_FMT	_IOWR('V', 105, struct v4l2_ext_pix_format)
+#define VIDIOC_TRY_EXT_PIX_FMT	_IOWR('V', 106, struct v4l2_ext_pix_format)
+
 /* Reminder: when adding new ioctls please add support for them to
    drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
 
-- 
2.26.0


^ permalink raw reply related	[flat|nested] 29+ messages in thread

* [PATCH v4 2/6] media: v4l2: Add extended buffer operations
  2020-07-17 11:54 [PATCH v4 0/6] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
  2020-07-17 11:54 ` [PATCH v4 1/6] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more) Helen Koike
@ 2020-07-17 11:54 ` Helen Koike
  2020-07-21 10:48   ` Hans Verkuil
  2020-07-21 11:26   ` Stanimir Varbanov
  2020-07-17 11:54 ` [PATCH v4 3/6] media: videobuf2: Expose helpers to implement the _ext_fmt and _ext_buf hooks Helen Koike
                   ` (6 subsequent siblings)
  8 siblings, 2 replies; 29+ messages in thread
From: Helen Koike @ 2020-07-17 11:54 UTC (permalink / raw)
  To: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus, linux-media
  Cc: Boris Brezillon, Helen Koike, tfiga, hiroh, nicolas,
	Brian.Starkey, kernel, narmstrong, linux-kernel, frkoenig,
	mjourdan, stanimir.varbanov

From: Hans Verkuil <hans.verkuil@cisco.com>

Those extended buffer ops have several purpose:
1/ Fix y2038 issues by converting the timestamp into an u64 counting
   the number of ns elapsed since 1970
2/ Unify single/multiplanar handling
3/ Add a new start offset field to each v4l2 plane buffer info struct
   to support the case where a single buffer object is storing all
   planes data, each one being placed at a different offset

New hooks are created in v4l2_ioctl_ops so that drivers can start using
these new objects.

The core takes care of converting new ioctls requests to old ones
if the driver does not support the new hooks, and vice versa.

Note that the timecode field is gone, since there doesn't seem to be
in-kernel users. We can be added back in the reserved area if needed or
use the Request API to collect more metadata information from the
frame.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Signed-off-by: Helen Koike <helen.koike@collabora.com>
---
Changes in v4:
- Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
- Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
I think we can add this later, so I removed it from this RFC to simplify it.
- Remove num_planes field from struct v4l2_ext_buffer
- Add flags field to struct v4l2_ext_create_buffers
- Reformulate struct v4l2_ext_plane
- Fix some bugs caught by v4l2-compliance
- Rebased on top of media/master (post 5.8-rc1)

Changes in v3:
- Rebased on top of media/master (post 5.4-rc1)

Changes in v2:
- Add reserved space to v4l2_ext_buffer so that new fields can be added
  later on
---
 drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
 drivers/media/v4l2-core/v4l2-ioctl.c | 349 +++++++++++++++++++++++++--
 include/media/v4l2-ioctl.h           |  26 ++
 include/uapi/linux/videodev2.h       |  89 +++++++
 4 files changed, 471 insertions(+), 22 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
index e1829906bc086..cb21ee8eb075c 100644
--- a/drivers/media/v4l2-core/v4l2-dev.c
+++ b/drivers/media/v4l2-core/v4l2-dev.c
@@ -720,15 +720,34 @@ static void determine_valid_ioctls(struct video_device *vdev)
 		SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_sdr_out);
 	}
 
+	if (is_vid || is_tch) {
+		/* ioctls valid for video and touch */
+		if (ops->vidioc_querybuf || ops->vidioc_ext_querybuf)
+			set_bit(_IOC_NR(VIDIOC_EXT_QUERYBUF), valid_ioctls);
+		if (ops->vidioc_qbuf || ops->vidioc_ext_qbuf)
+			set_bit(_IOC_NR(VIDIOC_EXT_QBUF), valid_ioctls);
+		if (ops->vidioc_dqbuf || ops->vidioc_ext_dqbuf)
+			set_bit(_IOC_NR(VIDIOC_EXT_DQBUF), valid_ioctls);
+		if (ops->vidioc_create_bufs || ops->vidioc_ext_create_bufs)
+			set_bit(_IOC_NR(VIDIOC_EXT_CREATE_BUFS), valid_ioctls);
+		if (ops->vidioc_prepare_buf || ops->vidioc_ext_prepare_buf)
+			set_bit(_IOC_NR(VIDIOC_EXT_PREPARE_BUF), valid_ioctls);
+	}
+
 	if (is_vid || is_vbi || is_sdr || is_tch || is_meta) {
 		/* ioctls valid for video, vbi, sdr, touch and metadata */
 		SET_VALID_IOCTL(ops, VIDIOC_REQBUFS, vidioc_reqbufs);
-		SET_VALID_IOCTL(ops, VIDIOC_QUERYBUF, vidioc_querybuf);
-		SET_VALID_IOCTL(ops, VIDIOC_QBUF, vidioc_qbuf);
 		SET_VALID_IOCTL(ops, VIDIOC_EXPBUF, vidioc_expbuf);
-		SET_VALID_IOCTL(ops, VIDIOC_DQBUF, vidioc_dqbuf);
-		SET_VALID_IOCTL(ops, VIDIOC_CREATE_BUFS, vidioc_create_bufs);
-		SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, vidioc_prepare_buf);
+		if (ops->vidioc_querybuf || ops->vidioc_ext_querybuf)
+			set_bit(_IOC_NR(VIDIOC_QUERYBUF), valid_ioctls);
+		if (ops->vidioc_qbuf || ops->vidioc_ext_qbuf)
+			set_bit(_IOC_NR(VIDIOC_QBUF), valid_ioctls);
+		if (ops->vidioc_dqbuf || ops->vidioc_ext_dqbuf)
+			set_bit(_IOC_NR(VIDIOC_DQBUF), valid_ioctls);
+		if (ops->vidioc_create_bufs || ops->vidioc_ext_create_bufs)
+			set_bit(_IOC_NR(VIDIOC_CREATE_BUFS), valid_ioctls);
+		if (ops->vidioc_prepare_buf || ops->vidioc_ext_prepare_buf)
+			set_bit(_IOC_NR(VIDIOC_PREPARE_BUF), valid_ioctls);
 		SET_VALID_IOCTL(ops, VIDIOC_STREAMON, vidioc_streamon);
 		SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, vidioc_streamoff);
 	}
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 3b77433f6c32b..5ddd57939c49c 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -527,6 +527,26 @@ static void v4l_print_buffer(const void *arg, bool write_only)
 			tc->type, tc->flags, tc->frames, *(__u32 *)tc->userbits);
 }
 
+static void v4l_print_ext_buffer(const void *arg, bool write_only)
+{
+	const struct v4l2_ext_buffer *e = arg;
+	const struct v4l2_ext_plane *plane;
+	unsigned int i;
+
+	pr_cont("%lld index=%d, type=%s, flags=0x%08x, field=%s, sequence=%d, memory=%s\n",
+		e->timestamp, e->index, prt_names(e->type, v4l2_type_names),
+		e->flags, prt_names(e->field, v4l2_field_names),
+		e->sequence, prt_names(e->memory, v4l2_memory_names));
+
+	for (i = 0; i < VIDEO_MAX_PLANES &&
+		    e->planes[i].buffer_length; i++) {
+		plane = &e->planes[i];
+		pr_debug("plane %d: buffer_length=%d, plane_length=%d offset=0x%08x\n",
+			 i, plane->buffer_length, plane->plane_length,
+			 plane->offset);
+	}
+}
+
 static void v4l_print_exportbuffer(const void *arg, bool write_only)
 {
 	const struct v4l2_exportbuffer *p = arg;
@@ -546,6 +566,15 @@ static void v4l_print_create_buffers(const void *arg, bool write_only)
 	v4l_print_format(&p->format, write_only);
 }
 
+static void v4l_print_ext_create_buffers(const void *arg, bool write_only)
+{
+	const struct v4l2_ext_create_buffers *p = arg;
+
+	pr_cont("index=%d, count=%d, memory=%s, ", p->index, p->count,
+		prt_names(p->memory, v4l2_memory_names));
+	v4l_print_ext_pix_format(&p->format, write_only);
+}
+
 static void v4l_print_streamparm(const void *arg, bool write_only)
 {
 	const struct v4l2_streamparm *p = arg;
@@ -1214,6 +1243,139 @@ int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
 }
 EXPORT_SYMBOL_GPL(v4l2_format_to_ext_pix_format);
 
+/*
+ * If mplane_cap is true, b->m.planes should have a valid pointer of a
+ * struct v4l2_plane array, and b->length with its size
+ */
+int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
+			      struct v4l2_buffer *b, bool mplane_cap)
+{
+	unsigned int planes_array_size = b->length;
+	struct v4l2_plane *planes = b->m.planes;
+	u64 nsecs;
+
+	if (!mplane_cap && e->planes[1].buffer_length != 0)
+		return -EINVAL;
+
+	memset(b, 0, sizeof(*b));
+
+	b->index = e->index;
+	b->flags = e->flags;
+	b->field = e->field;
+	b->sequence = e->sequence;
+	b->memory = e->memory;
+	b->request_fd = e->request_fd;
+	b->timestamp.tv_sec = div64_u64_rem(e->timestamp, NSEC_PER_SEC, &nsecs);
+	b->timestamp.tv_usec = (u32)nsecs / NSEC_PER_USEC;
+
+	if (mplane_cap) {
+		unsigned int i;
+
+		if (!planes || !planes_array_size)
+			return -EINVAL;
+
+		b->m.planes = planes;
+
+		if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
+			b->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
+		else
+			b->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
+
+		for (i = 0; i < VIDEO_MAX_PLANES && i < planes_array_size &&
+			    e->planes[i].buffer_length; i++) {
+
+			if (b->memory != V4L2_MEMORY_MMAP && e->planes[i].offset)
+				return -EINVAL;
+
+			memset(&b->m.planes[i], 0, sizeof(b->m.planes[i]));
+
+			if (b->memory == V4L2_MEMORY_MMAP)
+				b->m.planes[i].m.mem_offset = e->planes[i].offset;
+			else if (b->memory == V4L2_MEMORY_DMABUF)
+				b->m.planes[i].m.fd = e->planes[i].m.dmabuf_fd;
+			else
+				b->m.planes[i].m.userptr = e->planes[i].m.userptr;
+
+			b->m.planes[i].bytesused = e->planes[i].plane_length;
+			b->m.planes[i].length = e->planes[i].buffer_length;
+		}
+		/* In multi-planar, length contain the number of planes */
+		b->length = i;
+	} else {
+		b->type = e->type;
+		b->bytesused = e->planes[0].plane_length;
+		b->length = e->planes[0].buffer_length;
+
+		if (b->memory != V4L2_MEMORY_MMAP && e->planes[0].offset)
+			return -EINVAL;
+
+		if (b->memory == V4L2_MEMORY_MMAP)
+			b->m.offset = e->planes[0].offset;
+		else if (b->memory == V4L2_MEMORY_DMABUF)
+			b->m.fd = e->planes[0].m.dmabuf_fd;
+		else
+			b->m.userptr = e->planes[0].m.userptr;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_ext_buffer_to_buffer);
+
+int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
+			      struct v4l2_ext_buffer *e)
+{
+	memset(e, 0, sizeof(*e));
+
+	e->index = b->index;
+	e->flags = b->flags;
+	e->field = b->field;
+	e->sequence = b->sequence;
+	e->memory = b->memory;
+	e->request_fd = b->request_fd;
+	e->timestamp = b->timestamp.tv_sec * NSEC_PER_SEC +
+		b->timestamp.tv_usec * NSEC_PER_USEC;
+	if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
+		unsigned int i;
+
+		if (!b->m.planes)
+			return -EINVAL;
+
+		if (b->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
+			e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+		else
+			e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
+
+		/* In multi-planar, length contain the number of planes */
+		for (i = 0; i < b->length; i++) {
+			if (b->memory == V4L2_MEMORY_MMAP)
+				e->planes[i].offset = b->m.planes[i].m.mem_offset;
+			else if (b->memory == V4L2_MEMORY_DMABUF)
+				e->planes[i].m.dmabuf_fd = b->m.planes[i].m.fd;
+			else
+				e->planes[i].m.userptr = b->m.planes[i].m.userptr;
+
+			e->planes[i].buffer_length = b->m.planes[i].length;
+			e->planes[i].plane_length = b->m.planes[i].bytesused;
+			if (b->m.planes[i].data_offset)
+				pr_warn("Ignoring data_offset value %d\n",
+					b->m.planes[i].data_offset);
+		}
+	} else {
+		e->type = b->type;
+		e->planes[0].plane_length = b->bytesused;
+		e->planes[0].buffer_length = b->length;
+		if (b->memory == V4L2_MEMORY_MMAP)
+			e->planes[0].offset = b->m.offset;
+		else if (b->memory == V4L2_MEMORY_DMABUF)
+			e->planes[0].m.dmabuf_fd = b->m.fd;
+		else
+			e->planes[0].m.userptr = b->m.userptr;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_buffer_to_ext_buffer);
+
 static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
 				struct file *file, void *fh, void *arg)
 {
@@ -2467,31 +2629,112 @@ static int v4l_reqbufs(const struct v4l2_ioctl_ops *ops,
 	return ops->vidioc_reqbufs(file, fh, p);
 }
 
+static int v4l_do_buf_op(int (*op)(struct file *, void *,
+				   struct v4l2_buffer *),
+			 int (*ext_op)(struct file *, void *,
+				       struct v4l2_ext_buffer *),
+			 struct file *file, void *fh, struct v4l2_buffer *b)
+{
+	struct v4l2_ext_buffer e;
+	int ret;
+
+	ret = check_fmt(file, b->type);
+	if (ret)
+		return ret;
+
+	if (op)
+		return op(file, fh, b);
+
+	ret = v4l2_buffer_to_ext_buffer(b, &e);
+	if (ret)
+		return ret;
+
+	ret = ext_op(file, fh, &e);
+	if (ret)
+		return ret;
+
+	v4l2_ext_buffer_to_buffer(&e, b, V4L2_TYPE_IS_MULTIPLANAR(b->type));
+	return 0;
+}
+
+static int v4l_do_ext_buf_op(int (*op)(struct file *, void *,
+				       struct v4l2_buffer *),
+			     int (*ext_op)(struct file *, void *,
+					   struct v4l2_ext_buffer *),
+			     struct file *file, void *fh,
+			     struct v4l2_ext_buffer *e)
+{
+	struct video_device *vdev = video_devdata(file);
+	struct v4l2_plane planes[VIDEO_MAX_PLANES];
+	struct v4l2_buffer b;
+	bool mplane_cap;
+	int ret;
+
+	ret = check_fmt(file, e->type);
+	if (ret)
+		return ret;
+
+	if (ext_op)
+		return ext_op(file, fh, e);
+
+	mplane_cap = !!(vdev->device_caps &
+			(V4L2_CAP_VIDEO_CAPTURE_MPLANE |
+			 V4L2_CAP_VIDEO_OUTPUT_MPLANE |
+			 V4L2_CAP_VIDEO_M2M_MPLANE));
+	b.m.planes = planes;
+	b.length = VIDEO_MAX_PLANES;
+	ret = v4l2_ext_buffer_to_buffer(e, &b, mplane_cap);
+	if (ret)
+		return ret;
+
+	ret = op(file, fh, &b);
+	if (ret)
+		return ret;
+
+	v4l2_buffer_to_ext_buffer(&b, e);
+	return 0;
+}
+
 static int v4l_querybuf(const struct v4l2_ioctl_ops *ops,
-				struct file *file, void *fh, void *arg)
+			struct file *file, void *fh, void *arg)
 {
-	struct v4l2_buffer *p = arg;
-	int ret = check_fmt(file, p->type);
+	return v4l_do_buf_op(ops->vidioc_querybuf, ops->vidioc_ext_querybuf,
+			     file, fh, arg);
+}
 
-	return ret ? ret : ops->vidioc_querybuf(file, fh, p);
+static int v4l_ext_querybuf(const struct v4l2_ioctl_ops *ops,
+			    struct file *file, void *fh, void *arg)
+{
+	return v4l_do_ext_buf_op(ops->vidioc_querybuf,
+				 ops->vidioc_ext_querybuf, file, fh, arg);
 }
 
 static int v4l_qbuf(const struct v4l2_ioctl_ops *ops,
-				struct file *file, void *fh, void *arg)
+		    struct file *file, void *fh, void *arg)
 {
-	struct v4l2_buffer *p = arg;
-	int ret = check_fmt(file, p->type);
+	return v4l_do_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
+			     file, fh, arg);
+}
 
-	return ret ? ret : ops->vidioc_qbuf(file, fh, p);
+static int v4l_ext_qbuf(const struct v4l2_ioctl_ops *ops,
+			struct file *file, void *fh, void *arg)
+{
+	return v4l_do_ext_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
+				 file, fh, arg);
 }
 
 static int v4l_dqbuf(const struct v4l2_ioctl_ops *ops,
-				struct file *file, void *fh, void *arg)
+		     struct file *file, void *fh, void *arg)
 {
-	struct v4l2_buffer *p = arg;
-	int ret = check_fmt(file, p->type);
+	return v4l_do_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
+			     file, fh, arg);
+}
 
-	return ret ? ret : ops->vidioc_dqbuf(file, fh, p);
+static int v4l_ext_dqbuf(const struct v4l2_ioctl_ops *ops,
+			 struct file *file, void *fh, void *arg)
+{
+	return v4l_do_ext_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
+				 file, fh, arg);
 }
 
 static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
@@ -2507,7 +2750,27 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
 
 	v4l_sanitize_format(&create->format);
 
-	ret = ops->vidioc_create_bufs(file, fh, create);
+	if (ops->vidioc_create_bufs) {
+		ret = ops->vidioc_create_bufs(file, fh, create);
+	} else {
+		struct v4l2_ext_create_buffers ecreate = {
+			.count = create->count,
+			.memory = create->memory,
+		};
+
+		ret = v4l2_format_to_ext_pix_format(&create->format,
+						    &ecreate.format, true);
+		if (ret)
+			return ret;
+
+		ret = ops->vidioc_ext_create_bufs(file, fh, &ecreate);
+		if (ret)
+			return ret;
+
+		create->index = ecreate.index;
+		create->count = ecreate.count;
+		create->capabilities = ecreate.capabilities;
+	}
 
 	if (create->format.type == V4L2_BUF_TYPE_VIDEO_CAPTURE ||
 	    create->format.type == V4L2_BUF_TYPE_VIDEO_OUTPUT)
@@ -2516,13 +2779,60 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
 	return ret;
 }
 
+static int v4l_ext_create_bufs(const struct v4l2_ioctl_ops *ops,
+			       struct file *file, void *fh, void *arg)
+{
+	struct v4l2_ext_create_buffers *ecreate = arg;
+	struct video_device *vdev = video_devdata(file);
+	struct v4l2_create_buffers create = {
+		.count = ecreate->count,
+		.memory = ecreate->memory,
+		.flags = ecreate->flags,
+	};
+	bool mplane_cap;
+	int ret;
+
+	ret = check_fmt(file, ecreate->format.type);
+	if (ret)
+		return ret;
+
+	if (ops->vidioc_ext_create_bufs)
+		return ops->vidioc_ext_create_bufs(file, fh, ecreate);
+
+	mplane_cap = !!(vdev->device_caps &
+			(V4L2_CAP_VIDEO_CAPTURE_MPLANE |
+			 V4L2_CAP_VIDEO_OUTPUT_MPLANE |
+			 V4L2_CAP_VIDEO_M2M_MPLANE));
+	ret = v4l2_ext_pix_format_to_format(&ecreate->format,
+					    &create.format, mplane_cap, true);
+	if (ret)
+		return ret;
+
+	ret = v4l_create_bufs(ops, file, fh, &create);
+	if (ret)
+		return ret;
+
+	ecreate->index = create.index;
+	ecreate->count = create.count;
+	ecreate->capabilities = create.capabilities;
+
+	return 0;
+}
+
 static int v4l_prepare_buf(const struct v4l2_ioctl_ops *ops,
-				struct file *file, void *fh, void *arg)
+			   struct file *file, void *fh, void *arg)
 {
-	struct v4l2_buffer *b = arg;
-	int ret = check_fmt(file, b->type);
+	return v4l_do_buf_op(ops->vidioc_prepare_buf,
+			     ops->vidioc_ext_prepare_buf,
+			     file, fh, arg);
+}
 
-	return ret ? ret : ops->vidioc_prepare_buf(file, fh, b);
+static int v4l_ext_prepare_buf(const struct v4l2_ioctl_ops *ops,
+			       struct file *file, void *fh, void *arg)
+{
+	return v4l_do_ext_buf_op(ops->vidioc_prepare_buf,
+				 ops->vidioc_ext_prepare_buf,
+				 file, fh, arg);
 }
 
 static int v4l_g_parm(const struct v4l2_ioctl_ops *ops,
@@ -3196,12 +3506,15 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
 	IOCTL_INFO(VIDIOC_S_EXT_PIX_FMT, v4l_s_ext_pix_fmt, v4l_print_ext_pix_format, INFO_FL_PRIO),
 	IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
+	IOCTL_INFO(VIDIOC_EXT_QUERYBUF, v4l_ext_querybuf, v4l_print_ext_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_ext_buffer, planes)),
 	IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
 	IOCTL_INFO(VIDIOC_S_FBUF, v4l_stub_s_fbuf, v4l_print_framebuffer, INFO_FL_PRIO),
 	IOCTL_INFO(VIDIOC_OVERLAY, v4l_overlay, v4l_print_u32, INFO_FL_PRIO),
 	IOCTL_INFO(VIDIOC_QBUF, v4l_qbuf, v4l_print_buffer, INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_EXPBUF, v4l_stub_expbuf, v4l_print_exportbuffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_exportbuffer, flags)),
+	IOCTL_INFO(VIDIOC_EXT_QBUF, v4l_ext_qbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_DQBUF, v4l_dqbuf, v4l_print_buffer, INFO_FL_QUEUE),
+	IOCTL_INFO(VIDIOC_EXT_DQBUF, v4l_ext_dqbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_STREAMON, v4l_streamon, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_STREAMOFF, v4l_streamoff, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_G_PARM, v4l_g_parm, v4l_print_streamparm, INFO_FL_CLEAR(v4l2_streamparm, type)),
@@ -3266,7 +3579,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
 	IOCTL_INFO(VIDIOC_SUBSCRIBE_EVENT, v4l_subscribe_event, v4l_print_event_subscription, 0),
 	IOCTL_INFO(VIDIOC_UNSUBSCRIBE_EVENT, v4l_unsubscribe_event, v4l_print_event_subscription, 0),
 	IOCTL_INFO(VIDIOC_CREATE_BUFS, v4l_create_bufs, v4l_print_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
+	IOCTL_INFO(VIDIOC_EXT_CREATE_BUFS, v4l_ext_create_bufs, v4l_print_ext_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_PREPARE_BUF, v4l_prepare_buf, v4l_print_buffer, INFO_FL_QUEUE),
+	IOCTL_INFO(VIDIOC_EXT_PREPARE_BUF, v4l_ext_prepare_buf, v4l_print_ext_buffer, INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_ENUM_DV_TIMINGS, v4l_stub_enum_dv_timings, v4l_print_enum_dv_timings, INFO_FL_CLEAR(v4l2_enum_dv_timings, pad)),
 	IOCTL_INFO(VIDIOC_QUERY_DV_TIMINGS, v4l_stub_query_dv_timings, v4l_print_dv_timings, INFO_FL_ALWAYS_COPY),
 	IOCTL_INFO(VIDIOC_DV_TIMINGS_CAP, v4l_stub_dv_timings_cap, v4l_print_dv_timings_cap, INFO_FL_CLEAR(v4l2_dv_timings_cap, pad)),
diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
index 525ce86725260..524caedebab3d 100644
--- a/include/media/v4l2-ioctl.h
+++ b/include/media/v4l2-ioctl.h
@@ -169,16 +169,26 @@ struct v4l2_fh;
  *	:ref:`VIDIOC_REQBUFS <vidioc_reqbufs>` ioctl
  * @vidioc_querybuf: pointer to the function that implements
  *	:ref:`VIDIOC_QUERYBUF <vidioc_querybuf>` ioctl
+ * @vidioc_ext_querybuf: pointer to the function that implements
+ *	:ref:`VIDIOC_EXT_QUERYBUF <vidioc_ext_querybuf>` ioctl
  * @vidioc_qbuf: pointer to the function that implements
  *	:ref:`VIDIOC_QBUF <vidioc_qbuf>` ioctl
+ * @vidioc_ext_qbuf: pointer to the function that implements
+ *	:ref:`VIDIOC_EXT_QBUF <vidioc_ext_qbuf>` ioctl
  * @vidioc_expbuf: pointer to the function that implements
  *	:ref:`VIDIOC_EXPBUF <vidioc_expbuf>` ioctl
  * @vidioc_dqbuf: pointer to the function that implements
  *	:ref:`VIDIOC_DQBUF <vidioc_qbuf>` ioctl
+ * @vidioc_ext_dqbuf: pointer to the function that implements
+ *	:ref:`VIDIOC_EXT_DQBUF <vidioc_ext_qbuf>` ioctl
  * @vidioc_create_bufs: pointer to the function that implements
  *	:ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl
+ * @vidioc_ext_create_bufs: pointer to the function that implements
+ *	:ref:`VIDIOC_EXT_CREATE_BUFS <vidioc_ext_create_bufs>` ioctl
  * @vidioc_prepare_buf: pointer to the function that implements
  *	:ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl
+ * @vidioc_ext_prepare_buf: pointer to the function that implements
+ *	:ref:`VIDIOC_EXT_PREPARE_BUF <vidioc_ext_prepare_buf>` ioctl
  * @vidioc_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl
  * @vidioc_g_fbuf: pointer to the function that implements
@@ -439,17 +449,27 @@ struct v4l2_ioctl_ops {
 			      struct v4l2_requestbuffers *b);
 	int (*vidioc_querybuf)(struct file *file, void *fh,
 			       struct v4l2_buffer *b);
+	int (*vidioc_ext_querybuf)(struct file *file, void *fh,
+				   struct v4l2_ext_buffer *b);
 	int (*vidioc_qbuf)(struct file *file, void *fh,
 			   struct v4l2_buffer *b);
+	int (*vidioc_ext_qbuf)(struct file *file, void *fh,
+			       struct v4l2_ext_buffer *b);
 	int (*vidioc_expbuf)(struct file *file, void *fh,
 			     struct v4l2_exportbuffer *e);
 	int (*vidioc_dqbuf)(struct file *file, void *fh,
 			    struct v4l2_buffer *b);
+	int (*vidioc_ext_dqbuf)(struct file *file, void *fh,
+				struct v4l2_ext_buffer *b);
 
 	int (*vidioc_create_bufs)(struct file *file, void *fh,
 				  struct v4l2_create_buffers *b);
+	int (*vidioc_ext_create_bufs)(struct file *file, void *fh,
+				      struct v4l2_ext_create_buffers *b);
 	int (*vidioc_prepare_buf)(struct file *file, void *fh,
 				  struct v4l2_buffer *b);
+	int (*vidioc_ext_prepare_buf)(struct file *file, void *fh,
+				      struct v4l2_ext_buffer *b);
 
 	int (*vidioc_overlay)(struct file *file, void *fh, unsigned int i);
 	int (*vidioc_g_fbuf)(struct file *file, void *fh,
@@ -758,6 +778,12 @@ int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
 				  struct v4l2_format *f,
 				  bool mplane_cap, bool strict);
 
+int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
+			      struct v4l2_buffer *b,
+			      bool mplane_cap);
+int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
+			      struct v4l2_ext_buffer *e);
+
 /*
  * The user space interpretation of the 'v4l2_event' differs
  * based on the 'time_t' definition on 32-bit architectures, so
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index fc04c81ce7713..f4906adddc280 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -994,6 +994,37 @@ struct v4l2_plane {
 	__u32			reserved[11];
 };
 
+/**
+ * struct v4l2_ext_plane - extended plane buffer info
+ * @buffer_length:	size of the entire buffer in bytes, should fit
+ *			@offset + @plane_length
+ * @plane_length:	size of the plane in bytes.
+ * @userptr:		when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
+ *			to this plane.
+ * @dmabuf_fd:		when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
+ *			associated with this plane.
+ * @offset:		offset in the memory buffer where the plane starts. If
+ *			V4L2_MEMORY_MMAP is used, then it can be a "cookie" that
+ *			should be passed to mmap() called on the video node.
+ * @reserved:		extra space reserved for future fields, must be set to 0.
+ *
+ *
+ * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
+ * can have one plane for Y, and another for interleaved CbCr components.
+ * Each plane can reside in a separate memory buffer, or even in
+ * a completely separate memory node (e.g. in embedded devices).
+ */
+struct v4l2_ext_plane {
+	__u32 buffer_length;
+	__u32 plane_length;
+	union {
+		__u64 userptr;
+		__s32 dmabuf_fd;
+	} m;
+	__u32 offset;
+	__u32 reserved[4];
+};
+
 /**
  * struct v4l2_buffer - video buffer info
  * @index:	id number of the buffer
@@ -1055,6 +1086,36 @@ struct v4l2_buffer {
 	};
 };
 
+/**
+ * struct v4l2_ext_buffer - extended video buffer info
+ * @index:	id number of the buffer
+ * @type:	V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
+ * @flags:	buffer informational flags
+ * @field:	enum v4l2_field; field order of the image in the buffer
+ * @timestamp:	frame timestamp
+ * @sequence:	sequence count of this frame
+ * @memory:	enum v4l2_memory; the method, in which the actual video data is
+ *		passed
+ * @planes:	per-plane buffer information
+ * @request_fd:	fd of the request that this buffer should use
+ * @reserved:	extra space reserved for future fields, must be set to 0
+ *
+ * Contains data exchanged by application and driver using one of the Streaming
+ * I/O methods.
+ */
+struct v4l2_ext_buffer {
+	__u32 index;
+	__u32 type;
+	__u32 flags;
+	__u32 field;
+	__u64 timestamp;
+	__u32 sequence;
+	__u32 memory;
+	__u32 request_fd;
+	struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
+	__u32 reserved[4];
+};
+
 #ifndef __KERNEL__
 /**
  * v4l2_timeval_to_ns - Convert timeval to nanoseconds
@@ -2520,6 +2581,29 @@ struct v4l2_create_buffers {
 	__u32			reserved[6];
 };
 
+/**
+ * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
+ * @index:	on return, index of the first created buffer
+ * @count:	entry: number of requested buffers,
+ *		return: number of created buffers
+ * @memory:	enum v4l2_memory; buffer memory type
+ * @capabilities: capabilities of this buffer type.
+ * @format:	frame format, for which buffers are requested
+ * @flags:	additional buffer management attributes (ignored unless the
+ *		queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
+ *		and configured for MMAP streaming I/O).
+ * @reserved:	extra space reserved for future fields, must be set to 0
+ */
+struct v4l2_ext_create_buffers {
+	__u32				index;
+	__u32				count;
+	__u32				memory;
+	struct v4l2_ext_pix_format	format;
+	__u32				capabilities;
+	__u32				flags;
+	__u32 reserved[4];
+};
+
 /*
  *	I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
  *
@@ -2623,6 +2707,11 @@ struct v4l2_create_buffers {
 #define VIDIOC_G_EXT_PIX_FMT	_IOWR('V', 104, struct v4l2_ext_pix_format)
 #define VIDIOC_S_EXT_PIX_FMT	_IOWR('V', 105, struct v4l2_ext_pix_format)
 #define VIDIOC_TRY_EXT_PIX_FMT	_IOWR('V', 106, struct v4l2_ext_pix_format)
+#define VIDIOC_EXT_CREATE_BUFS	_IOWR('V', 107, struct v4l2_ext_create_buffers)
+#define VIDIOC_EXT_QUERYBUF	_IOWR('V', 108, struct v4l2_ext_buffer)
+#define VIDIOC_EXT_QBUF		_IOWR('V', 109, struct v4l2_ext_buffer)
+#define VIDIOC_EXT_DQBUF	_IOWR('V', 110, struct v4l2_ext_buffer)
+#define VIDIOC_EXT_PREPARE_BUF	_IOWR('V', 111, struct v4l2_ext_buffer)
 
 /* Reminder: when adding new ioctls please add support for them to
    drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
-- 
2.26.0


^ permalink raw reply related	[flat|nested] 29+ messages in thread

* [PATCH v4 3/6] media: videobuf2: Expose helpers to implement the _ext_fmt and _ext_buf hooks
  2020-07-17 11:54 [PATCH v4 0/6] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
  2020-07-17 11:54 ` [PATCH v4 1/6] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more) Helen Koike
  2020-07-17 11:54 ` [PATCH v4 2/6] media: v4l2: Add extended buffer operations Helen Koike
@ 2020-07-17 11:54 ` Helen Koike
  2020-07-17 11:54 ` [PATCH v4 4/6] media: mediabus: Add helpers to convert a ext_pix format to/from a mbus_fmt Helen Koike
                   ` (5 subsequent siblings)
  8 siblings, 0 replies; 29+ messages in thread
From: Helen Koike @ 2020-07-17 11:54 UTC (permalink / raw)
  To: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus, linux-media
  Cc: Boris Brezillon, Helen Koike, tfiga, hiroh, nicolas,
	Brian.Starkey, kernel, narmstrong, linux-kernel, frkoenig,
	mjourdan, stanimir.varbanov

From: Boris Brezillon <boris.brezillon@collabora.com>

The VB2 layer is used by a lot of drivers. Patch it to support the
_EXT_FMT and _EXT_BUF ioctls in order to ease conversion of existing
drivers to these new APIs.

Note that internally, the VB2 core is now only using ext structs and old
APIs are supported through conversion wrappers.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Signed-off-by: Helen Koike <helen.koike@collabora.com>
---
Changes in v4:
- Update with new format and buffer structs
- Fix some bugs caught by v4l2-compliance
- Rebased on top of media/master (post 5.8-rc1)

Changes in v3:
- Rebased on top of media/master (post 5.4-rc1)

Changes in v2:
- New patch
---
 .../media/common/videobuf2/videobuf2-core.c   |   2 +
 .../media/common/videobuf2/videobuf2-v4l2.c   | 549 ++++++++++--------
 include/media/videobuf2-core.h                |   6 +-
 include/media/videobuf2-v4l2.h                |  21 +-
 4 files changed, 335 insertions(+), 243 deletions(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
index abaf28e057ebe..1e81efdddcf08 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -1253,6 +1253,7 @@ static int __prepare_dmabuf(struct vb2_buffer *vb)
 		vb->planes[plane].length = 0;
 		vb->planes[plane].m.fd = 0;
 		vb->planes[plane].data_offset = 0;
+		vb->planes[plane].dbuf_offset = 0;
 
 		/* Acquire each plane's memory */
 		mem_priv = call_ptr_memop(vb, attach_dmabuf,
@@ -1296,6 +1297,7 @@ static int __prepare_dmabuf(struct vb2_buffer *vb)
 		vb->planes[plane].length = planes[plane].length;
 		vb->planes[plane].m.fd = planes[plane].m.fd;
 		vb->planes[plane].data_offset = planes[plane].data_offset;
+		vb->planes[plane].dbuf_offset = planes[plane].dbuf_offset;
 	}
 
 	if (reacquired) {
diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index 57aa183bd1983..928d754fb89e8 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -29,6 +29,7 @@
 #include <media/v4l2-fh.h>
 #include <media/v4l2-event.h>
 #include <media/v4l2-common.h>
+#include <media/v4l2-ioctl.h>
 
 #include <media/videobuf2-v4l2.h>
 
@@ -56,42 +57,17 @@ module_param(debug, int, 0644);
 				 V4L2_BUF_FLAG_TIMECODE | \
 				 V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF)
 
-/*
- * __verify_planes_array() - verify that the planes array passed in struct
- * v4l2_buffer from userspace can be safely used
- */
-static int __verify_planes_array(struct vb2_buffer *vb, const struct v4l2_buffer *b)
-{
-	if (!V4L2_TYPE_IS_MULTIPLANAR(b->type))
-		return 0;
-
-	/* Is memory for copying plane information present? */
-	if (b->m.planes == NULL) {
-		dprintk(vb->vb2_queue, 1,
-			"multi-planar buffer passed but planes array not provided\n");
-		return -EINVAL;
-	}
-
-	if (b->length < vb->num_planes || b->length > VB2_MAX_PLANES) {
-		dprintk(vb->vb2_queue, 1,
-			"incorrect planes array length, expected %d, got %d\n",
-			vb->num_planes, b->length);
-		return -EINVAL;
-	}
-
-	return 0;
-}
-
 static int __verify_planes_array_core(struct vb2_buffer *vb, const void *pb)
 {
-	return __verify_planes_array(vb, pb);
+	return 0;
 }
 
 /*
  * __verify_length() - Verify that the bytesused value for each plane fits in
  * the plane length and that the data offset doesn't exceed the bytesused value.
  */
-static int __verify_length(struct vb2_buffer *vb, const struct v4l2_buffer *b)
+static int __verify_length(struct vb2_buffer *vb,
+			   const struct v4l2_ext_buffer *b)
 {
 	unsigned int length;
 	unsigned int bytesused;
@@ -100,28 +76,19 @@ static int __verify_length(struct vb2_buffer *vb, const struct v4l2_buffer *b)
 	if (!V4L2_TYPE_IS_OUTPUT(b->type))
 		return 0;
 
-	if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
-		for (plane = 0; plane < vb->num_planes; ++plane) {
-			length = (b->memory == VB2_MEMORY_USERPTR ||
-				  b->memory == VB2_MEMORY_DMABUF)
-			       ? b->m.planes[plane].length
-				: vb->planes[plane].length;
-			bytesused = b->m.planes[plane].bytesused
-				  ? b->m.planes[plane].bytesused : length;
-
-			if (b->m.planes[plane].bytesused > length)
-				return -EINVAL;
-
-			if (b->m.planes[plane].data_offset > 0 &&
-			    b->m.planes[plane].data_offset >= bytesused)
-				return -EINVAL;
-		}
-	} else {
+	for (plane = 0; plane < vb->num_planes; ++plane) {
 		length = (b->memory == VB2_MEMORY_USERPTR ||
-			  b->memory == VB2_MEMORY_DMABUF)
-			? b->length : vb->planes[0].length;
+			  b->memory == VB2_MEMORY_DMABUF) ?
+			 b->planes[plane].buffer_length :
+			 vb->planes[plane].length;
+		bytesused = b->planes[plane].plane_length ?
+			    b->planes[plane].plane_length : length;
 
-		if (b->bytesused > length)
+		if (bytesused > length)
+			return -EINVAL;
+
+		if (b->memory != V4L2_MEMORY_MMAP &&
+		    b->planes[plane].offset + bytesused > length)
 			return -EINVAL;
 	}
 
@@ -140,21 +107,12 @@ static void __init_vb2_v4l2_buffer(struct vb2_buffer *vb)
 
 static void __copy_timestamp(struct vb2_buffer *vb, const void *pb)
 {
-	const struct v4l2_buffer *b = pb;
-	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
+	const struct v4l2_ext_buffer *b = pb;
 	struct vb2_queue *q = vb->vb2_queue;
 
-	if (q->is_output) {
-		/*
-		 * For output buffers copy the timestamp if needed,
-		 * and the timecode field and flag if needed.
-		 */
-		if (q->copy_timestamp)
-			vb->timestamp = v4l2_buffer_get_timestamp(b);
-		vbuf->flags |= b->flags & V4L2_BUF_FLAG_TIMECODE;
-		if (b->flags & V4L2_BUF_FLAG_TIMECODE)
-			vbuf->timecode = b->timecode;
-	}
+	/* For output buffers copy the timestamp if needed. */
+	if (q->is_output && q->copy_timestamp)
+		vb->timestamp = b->timestamp;
 };
 
 static void vb2_warn_zero_bytesused(struct vb2_buffer *vb)
@@ -173,7 +131,8 @@ static void vb2_warn_zero_bytesused(struct vb2_buffer *vb)
 		pr_warn("use the actual size instead.\n");
 }
 
-static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *b)
+static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb,
+				    struct v4l2_ext_buffer *b)
 {
 	struct vb2_queue *q = vb->vb2_queue;
 	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
@@ -203,110 +162,60 @@ static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *b
 	vbuf->request_fd = -1;
 	vbuf->is_held = false;
 
-	if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
-		switch (b->memory) {
-		case VB2_MEMORY_USERPTR:
-			for (plane = 0; plane < vb->num_planes; ++plane) {
-				planes[plane].m.userptr =
-					b->m.planes[plane].m.userptr;
-				planes[plane].length =
-					b->m.planes[plane].length;
-			}
-			break;
-		case VB2_MEMORY_DMABUF:
-			for (plane = 0; plane < vb->num_planes; ++plane) {
-				planes[plane].m.fd =
-					b->m.planes[plane].m.fd;
-				planes[plane].length =
-					b->m.planes[plane].length;
-			}
-			break;
-		default:
-			for (plane = 0; plane < vb->num_planes; ++plane) {
-				planes[plane].m.offset =
-					vb->planes[plane].m.offset;
-				planes[plane].length =
-					vb->planes[plane].length;
-			}
-			break;
+	switch (b->memory) {
+	case VB2_MEMORY_USERPTR:
+		for (plane = 0; plane < vb->num_planes; ++plane) {
+			planes[plane].m.userptr = b->planes[plane].m.userptr;
+			planes[plane].length = b->planes[plane].buffer_length;
 		}
-
-		/* Fill in driver-provided information for OUTPUT types */
-		if (V4L2_TYPE_IS_OUTPUT(b->type)) {
-			/*
-			 * Will have to go up to b->length when API starts
-			 * accepting variable number of planes.
-			 *
-			 * If bytesused == 0 for the output buffer, then fall
-			 * back to the full buffer size. In that case
-			 * userspace clearly never bothered to set it and
-			 * it's a safe assumption that they really meant to
-			 * use the full plane sizes.
-			 *
-			 * Some drivers, e.g. old codec drivers, use bytesused == 0
-			 * as a way to indicate that streaming is finished.
-			 * In that case, the driver should use the
-			 * allow_zero_bytesused flag to keep old userspace
-			 * applications working.
-			 */
-			for (plane = 0; plane < vb->num_planes; ++plane) {
-				struct vb2_plane *pdst = &planes[plane];
-				struct v4l2_plane *psrc = &b->m.planes[plane];
-
-				if (psrc->bytesused == 0)
-					vb2_warn_zero_bytesused(vb);
-
-				if (vb->vb2_queue->allow_zero_bytesused)
-					pdst->bytesused = psrc->bytesused;
-				else
-					pdst->bytesused = psrc->bytesused ?
-						psrc->bytesused : pdst->length;
-				pdst->data_offset = psrc->data_offset;
-			}
+		break;
+	case VB2_MEMORY_DMABUF:
+		for (plane = 0; plane < vb->num_planes; ++plane) {
+			planes[plane].m.fd = b->planes[plane].m.dmabuf_fd;
+			planes[plane].dbuf_offset = b->planes[plane].offset;
+			planes[plane].length = b->planes[plane].buffer_length;
 		}
-	} else {
+		break;
+	default:
+		for (plane = 0; plane < vb->num_planes; ++plane) {
+			planes[plane].m.offset = vb->planes[plane].m.offset;
+			planes[plane].length = vb->planes[plane].length;
+		}
+		break;
+	}
+
+	/* Fill in driver-provided information for OUTPUT types */
+	if (V4L2_TYPE_IS_OUTPUT(b->type)) {
 		/*
-		 * Single-planar buffers do not use planes array,
-		 * so fill in relevant v4l2_buffer struct fields instead.
-		 * In videobuf we use our internal V4l2_planes struct for
-		 * single-planar buffers as well, for simplicity.
+		 * Will have to go up to b->length when API starts
+		 * accepting variable number of planes.
 		 *
-		 * If bytesused == 0 for the output buffer, then fall back
-		 * to the full buffer size as that's a sensible default.
+		 * If bytesused == 0 for the output buffer, then fall
+		 * back to the full buffer size. In that case
+		 * userspace clearly never bothered to set it and
+		 * it's a safe assumption that they really meant to
+		 * use the full plane sizes.
 		 *
-		 * Some drivers, e.g. old codec drivers, use bytesused == 0 as
-		 * a way to indicate that streaming is finished. In that case,
-		 * the driver should use the allow_zero_bytesused flag to keep
-		 * old userspace applications working.
+		 * Some drivers, e.g. old codec drivers, use bytesused == 0
+		 * as a way to indicate that streaming is finished.
+		 * In that case, the driver should use the
+		 * allow_zero_bytesused flag to keep old userspace
+		 * applications working.
 		 */
-		switch (b->memory) {
-		case VB2_MEMORY_USERPTR:
-			planes[0].m.userptr = b->m.userptr;
-			planes[0].length = b->length;
-			break;
-		case VB2_MEMORY_DMABUF:
-			planes[0].m.fd = b->m.fd;
-			planes[0].length = b->length;
-			break;
-		default:
-			planes[0].m.offset = vb->planes[0].m.offset;
-			planes[0].length = vb->planes[0].length;
-			break;
-		}
+		for (plane = 0; plane < vb->num_planes; ++plane) {
+			struct vb2_plane *pdst = &planes[plane];
+			struct v4l2_ext_plane *psrc = &b->planes[plane];
 
-		planes[0].data_offset = 0;
-		if (V4L2_TYPE_IS_OUTPUT(b->type)) {
-			if (b->bytesused == 0)
+			if (psrc->plane_length == 0)
 				vb2_warn_zero_bytesused(vb);
 
 			if (vb->vb2_queue->allow_zero_bytesused)
-				planes[0].bytesused = b->bytesused;
+				pdst->bytesused = psrc->plane_length;
 			else
-				planes[0].bytesused = b->bytesused ?
-					b->bytesused : planes[0].length;
-		} else
-			planes[0].bytesused = 0;
-
+				pdst->bytesused = psrc->plane_length ?
+						  psrc->plane_length :
+						  pdst->length;
+		}
 	}
 
 	/* Zero flags that we handle */
@@ -343,7 +252,7 @@ static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *b
 
 static void set_buffer_cache_hints(struct vb2_queue *q,
 				   struct vb2_buffer *vb,
-				   struct v4l2_buffer *b)
+				   struct v4l2_ext_buffer *b)
 {
 	/*
 	 * DMA exporter should take care of cache syncs, so we can avoid
@@ -388,8 +297,34 @@ static void set_buffer_cache_hints(struct vb2_queue *q,
 		vb->need_cache_sync_on_prepare = 0;
 }
 
+static enum v4l2_buf_type vb2_ext_qtype(struct vb2_queue *q)
+{
+	if (!q->is_multiplanar)
+		return q->type;
+
+	if (q->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
+		return V4L2_BUF_TYPE_VIDEO_CAPTURE;
+	else if (q->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE)
+		return V4L2_BUF_TYPE_VIDEO_OUTPUT;
+
+	return q->type;
+}
+
+static enum v4l2_buf_type vb2_ext_type(unsigned int type, bool is_multiplanar)
+{
+	if (!is_multiplanar)
+		return type;
+
+	if (type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
+	else if (type == V4L2_BUF_TYPE_VIDEO_OUTPUT)
+		return V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
+
+	return type;
+}
+
 static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
-				    struct v4l2_buffer *b, bool is_prepare,
+				    struct v4l2_ext_buffer *b, bool is_prepare,
 				    struct media_request **p_req)
 {
 	const char *opname = is_prepare ? "prepare_buf" : "qbuf";
@@ -398,7 +333,7 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct media_device *md
 	struct vb2_buffer *vb;
 	int ret;
 
-	if (b->type != q->type) {
+	if (b->type != vb2_ext_qtype(q)) {
 		dprintk(q, 1, "%s: invalid buffer type\n", opname);
 		return -EINVAL;
 	}
@@ -421,9 +356,6 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct media_device *md
 
 	vb = q->bufs[b->index];
 	vbuf = to_vb2_v4l2_buffer(vb);
-	ret = __verify_planes_array(vb, b);
-	if (ret)
-		return ret;
 
 	if (!is_prepare && (b->flags & V4L2_BUF_FLAG_REQUEST_FD) &&
 	    vb->state != VB2_BUF_STATE_DEQUEUED) {
@@ -487,11 +419,6 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct media_device *md
 		    !q->ops->buf_out_validate))
 		return -EINVAL;
 
-	if (b->request_fd < 0) {
-		dprintk(q, 1, "%s: request_fd < 0\n", opname);
-		return -EINVAL;
-	}
-
 	req = media_request_get_by_fd(mdev, b->request_fd);
 	if (IS_ERR(req)) {
 		dprintk(q, 1, "%s: invalid request_fd\n", opname);
@@ -516,64 +443,47 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct media_device *md
 }
 
 /*
- * __fill_v4l2_buffer() - fill in a struct v4l2_buffer with information to be
- * returned to userspace
+ * __fill_v4l2_buffer() - fill in a struct v4l2_ext_buffer with information to
+ * be returned to userspace
  */
 static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
 {
-	struct v4l2_buffer *b = pb;
+	struct v4l2_ext_buffer *b = pb;
 	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
 	struct vb2_queue *q = vb->vb2_queue;
 	unsigned int plane;
 
 	/* Copy back data such as timestamp, flags, etc. */
 	b->index = vb->index;
-	b->type = vb->type;
+	b->type = vb2_ext_qtype(q);
 	b->memory = vb->memory;
-	b->bytesused = 0;
 
 	b->flags = vbuf->flags;
 	b->field = vbuf->field;
-	v4l2_buffer_set_timestamp(b, vb->timestamp);
-	b->timecode = vbuf->timecode;
+	b->timestamp = vb->timestamp;
 	b->sequence = vbuf->sequence;
-	b->reserved2 = 0;
 	b->request_fd = 0;
+	memset(b->reserved, 0, sizeof(b->reserved));
 
-	if (q->is_multiplanar) {
-		/*
-		 * Fill in plane-related data if userspace provided an array
-		 * for it. The caller has already verified memory and size.
-		 */
-		b->length = vb->num_planes;
-		for (plane = 0; plane < vb->num_planes; ++plane) {
-			struct v4l2_plane *pdst = &b->m.planes[plane];
-			struct vb2_plane *psrc = &vb->planes[plane];
-
-			pdst->bytesused = psrc->bytesused;
-			pdst->length = psrc->length;
-			if (q->memory == VB2_MEMORY_MMAP)
-				pdst->m.mem_offset = psrc->m.offset;
-			else if (q->memory == VB2_MEMORY_USERPTR)
-				pdst->m.userptr = psrc->m.userptr;
-			else if (q->memory == VB2_MEMORY_DMABUF)
-				pdst->m.fd = psrc->m.fd;
-			pdst->data_offset = psrc->data_offset;
-			memset(pdst->reserved, 0, sizeof(pdst->reserved));
+	/*
+	 * Fill in plane-related data if userspace provided an array
+	 * for it. The caller has already verified memory and size.
+	 */
+	for (plane = 0; plane < vb->num_planes; ++plane) {
+		struct v4l2_ext_plane *pdst = &b->planes[plane];
+		struct vb2_plane *psrc = &vb->planes[plane];
+
+		pdst->plane_length = psrc->bytesused;
+		pdst->buffer_length = psrc->length;
+		if (q->memory == VB2_MEMORY_MMAP) {
+			pdst->offset = psrc->m.offset;
+		} else if (q->memory == VB2_MEMORY_USERPTR) {
+			pdst->m.userptr = psrc->m.userptr;
+		} else if (q->memory == VB2_MEMORY_DMABUF) {
+			pdst->m.dmabuf_fd = psrc->m.fd;
+			pdst->offset = psrc->dbuf_offset;
 		}
-	} else {
-		/*
-		 * We use length and offset in v4l2_planes array even for
-		 * single-planar buffers, but userspace does not.
-		 */
-		b->length = vb->planes[0].length;
-		b->bytesused = vb->planes[0].bytesused;
-		if (q->memory == VB2_MEMORY_MMAP)
-			b->m.offset = vb->planes[0].m.offset;
-		else if (q->memory == VB2_MEMORY_USERPTR)
-			b->m.userptr = vb->planes[0].m.userptr;
-		else if (q->memory == VB2_MEMORY_DMABUF)
-			b->m.fd = vb->planes[0].m.fd;
+		memset(pdst->reserved, 0, sizeof(pdst->reserved));
 	}
 
 	/*
@@ -668,6 +578,35 @@ int vb2_find_timestamp(const struct vb2_queue *q, u64 timestamp,
 }
 EXPORT_SYMBOL_GPL(vb2_find_timestamp);
 
+#define vb2_buf_to_ext_buf_op(_name, ...)				\
+({									\
+	int ret;							\
+									\
+	ret = v4l2_buffer_to_ext_buffer(b, &eb);			\
+	if (!ret)							\
+		ret = _name(__VA_ARGS__);				\
+	if (!ret)							\
+		ret = v4l2_ext_buffer_to_buffer(&eb, b,			\
+						q->is_multiplanar);	\
+	ret;								\
+})
+
+int vb2_ext_querybuf(struct vb2_queue *q, struct v4l2_ext_buffer *b)
+{
+	if (b->type != vb2_ext_qtype(q)) {
+		dprintk(q, 1, "wrong buffer type\n");
+		return -EINVAL;
+	}
+
+	if (b->index >= q->num_buffers) {
+		dprintk(q, 1, "buffer index out of range\n");
+		return -EINVAL;
+	}
+	vb2_core_querybuf(q, b->index, b);
+	return 0;
+}
+EXPORT_SYMBOL(vb2_ext_querybuf);
+
 /*
  * vb2_querybuf() - query video buffer information
  * @q:		videobuf queue
@@ -683,23 +622,9 @@ EXPORT_SYMBOL_GPL(vb2_find_timestamp);
  */
 int vb2_querybuf(struct vb2_queue *q, struct v4l2_buffer *b)
 {
-	struct vb2_buffer *vb;
-	int ret;
+	struct v4l2_ext_buffer eb;
 
-	if (b->type != q->type) {
-		dprintk(q, 1, "wrong buffer type\n");
-		return -EINVAL;
-	}
-
-	if (b->index >= q->num_buffers) {
-		dprintk(q, 1, "buffer index out of range\n");
-		return -EINVAL;
-	}
-	vb = q->bufs[b->index];
-	ret = __verify_planes_array(vb, b);
-	if (!ret)
-		vb2_core_querybuf(q, b->index, b);
-	return ret;
+	return vb2_buf_to_ext_buf_op(vb2_ext_querybuf, q, &eb);
 }
 EXPORT_SYMBOL(vb2_querybuf);
 
@@ -741,8 +666,8 @@ int vb2_reqbufs(struct vb2_queue *q, struct v4l2_requestbuffers *req)
 }
 EXPORT_SYMBOL_GPL(vb2_reqbufs);
 
-int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
-		    struct v4l2_buffer *b)
+int vb2_ext_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
+			struct v4l2_ext_buffer *b)
 {
 	int ret;
 
@@ -758,16 +683,59 @@ int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
 
 	return ret ? ret : vb2_core_prepare_buf(q, b->index, b);
 }
+EXPORT_SYMBOL_GPL(vb2_ext_prepare_buf);
+
+int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
+		    struct v4l2_buffer *b)
+{
+	struct v4l2_ext_buffer eb;
+
+	return vb2_buf_to_ext_buf_op(vb2_ext_prepare_buf, q, mdev, &eb);
+}
 EXPORT_SYMBOL_GPL(vb2_prepare_buf);
 
+int vb2_ext_create_bufs(struct vb2_queue *q,
+			struct v4l2_ext_create_buffers *create)
+{
+	unsigned int requested_sizes[VIDEO_MAX_PLANES];
+	struct v4l2_ext_pix_format *f = &create->format;
+	int ret = vb2_verify_memory_type(q, create->memory,
+				vb2_ext_type(f->type, q->is_multiplanar));
+	unsigned i;
+
+	if (create->format.type != V4L2_BUF_TYPE_VIDEO_CAPTURE &&
+	    create->format.type != V4L2_BUF_TYPE_VIDEO_OUTPUT)
+		return -EINVAL;
+
+	fill_buf_caps(q, &create->capabilities);
+	clear_consistency_attr(q, create->memory, &create->flags);
+	create->index = q->num_buffers;
+	if (create->count == 0)
+		return ret != -EBUSY ? ret : 0;
+
+	if (!f->plane_fmt[0].sizeimage)
+		return -EINVAL;
+	for (i = 0; i < VIDEO_MAX_PLANES &&
+		    f->plane_fmt[i].sizeimage; i++)
+		requested_sizes[i] = f->plane_fmt[i].sizeimage;
+
+	return ret ? ret : vb2_core_create_bufs(q, create->memory,
+						create->flags,
+						&create->count,
+						i,
+						requested_sizes);
+}
+EXPORT_SYMBOL_GPL(vb2_ext_create_bufs);
+
 int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create)
 {
+	struct v4l2_format *f = &create->format;
 	unsigned requested_planes = 1;
 	unsigned requested_sizes[VIDEO_MAX_PLANES];
-	struct v4l2_format *f = &create->format;
 	int ret = vb2_verify_memory_type(q, create->memory, f->type);
 	unsigned i;
 
+
 	fill_buf_caps(q, &create->capabilities);
 	clear_consistency_attr(q, create->memory, &create->flags);
 	create->index = q->num_buffers;
@@ -777,18 +745,29 @@ int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create)
 	switch (f->type) {
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
-		requested_planes = f->fmt.pix_mp.num_planes;
-		if (requested_planes == 0 ||
-		    requested_planes > VIDEO_MAX_PLANES)
-			return -EINVAL;
-		for (i = 0; i < requested_planes; i++)
-			requested_sizes[i] =
-				f->fmt.pix_mp.plane_fmt[i].sizeimage;
-		break;
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
-	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
-		requested_sizes[0] = f->fmt.pix.sizeimage;
-		break;
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT: {
+		struct v4l2_ext_create_buffers ecreate = {
+			.count = create->count,
+			.memory = create->memory,
+			.flags = create->flags,
+		};
+
+		ret = v4l2_format_to_ext_pix_format(&create->format,
+						    &ecreate.format, true);
+		if (ret)
+			return ret;
+
+		ret = vb2_ext_create_bufs(q, &ecreate);
+		if (ret)
+			return ret;
+
+		create->index = ecreate.index;
+		create->count = ecreate.count;
+		create->flags = ecreate.flags;
+		create->capabilities = ecreate.capabilities;
+		return 0;
+	}
 	case V4L2_BUF_TYPE_VBI_CAPTURE:
 	case V4L2_BUF_TYPE_VBI_OUTPUT:
 		requested_sizes[0] = f->fmt.vbi.samples_per_line *
@@ -820,8 +799,8 @@ int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create)
 }
 EXPORT_SYMBOL_GPL(vb2_create_bufs);
 
-int vb2_qbuf(struct vb2_queue *q, struct media_device *mdev,
-	     struct v4l2_buffer *b)
+int vb2_ext_qbuf(struct vb2_queue *q, struct media_device *mdev,
+		 struct v4l2_ext_buffer *b)
 {
 	struct media_request *req = NULL;
 	int ret;
@@ -839,9 +818,19 @@ int vb2_qbuf(struct vb2_queue *q, struct media_device *mdev,
 		media_request_put(req);
 	return ret;
 }
+EXPORT_SYMBOL_GPL(vb2_ext_qbuf);
+
+int vb2_qbuf(struct vb2_queue *q, struct media_device *mdev,
+	     struct v4l2_buffer *b)
+{
+	struct v4l2_ext_buffer eb;
+
+	return vb2_buf_to_ext_buf_op(vb2_ext_qbuf, q, mdev, &eb);
+}
 EXPORT_SYMBOL_GPL(vb2_qbuf);
 
-int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, bool nonblocking)
+int vb2_ext_dqbuf(struct vb2_queue *q, struct v4l2_ext_buffer *b,
+		  bool nonblocking)
 {
 	int ret;
 
@@ -850,7 +839,7 @@ int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, bool nonblocking)
 		return -EBUSY;
 	}
 
-	if (b->type != q->type) {
+	if (b->type != vb2_ext_qtype(q)) {
 		dprintk(q, 1, "invalid buffer type\n");
 		return -EINVAL;
 	}
@@ -870,6 +859,14 @@ int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, bool nonblocking)
 
 	return ret;
 }
+EXPORT_SYMBOL_GPL(vb2_ext_dqbuf);
+
+int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, bool nonblocking)
+{
+	struct v4l2_ext_buffer eb;
+
+	return vb2_buf_to_ext_buf_op(vb2_ext_dqbuf, q, &eb, nonblocking);
+}
 EXPORT_SYMBOL_GPL(vb2_dqbuf);
 
 int vb2_streamon(struct vb2_queue *q, enum v4l2_buf_type type)
@@ -1040,6 +1037,33 @@ int vb2_ioctl_create_bufs(struct file *file, void *priv,
 }
 EXPORT_SYMBOL_GPL(vb2_ioctl_create_bufs);
 
+int vb2_ioctl_ext_create_bufs(struct file *file, void *priv,
+			      struct v4l2_ext_create_buffers *p)
+{
+	struct video_device *vdev = video_devdata(file);
+	int res = vb2_verify_memory_type(vdev->queue, p->memory,
+		vb2_ext_type(p->format.type, vdev->queue->is_multiplanar));
+
+	p->index = vdev->queue->num_buffers;
+	fill_buf_caps(vdev->queue, &p->capabilities);
+	/*
+	 * If count == 0, then just check if memory and type are valid.
+	 * Any -EBUSY result from vb2_verify_memory_type can be mapped to 0.
+	 */
+	if (p->count == 0)
+		return res != -EBUSY ? res : 0;
+	if (res)
+		return res;
+	if (vb2_queue_is_busy(vdev, file))
+		return -EBUSY;
+
+	res = vb2_ext_create_bufs(vdev->queue, p);
+	if (res == 0)
+		vdev->queue->owner = file->private_data;
+	return res;
+}
+EXPORT_SYMBOL_GPL(vb2_ioctl_ext_create_bufs);
+
 int vb2_ioctl_prepare_buf(struct file *file, void *priv,
 			  struct v4l2_buffer *p)
 {
@@ -1051,6 +1075,17 @@ int vb2_ioctl_prepare_buf(struct file *file, void *priv,
 }
 EXPORT_SYMBOL_GPL(vb2_ioctl_prepare_buf);
 
+int vb2_ioctl_ext_prepare_buf(struct file *file, void *priv,
+			      struct v4l2_ext_buffer *p)
+{
+	struct video_device *vdev = video_devdata(file);
+
+	if (vb2_queue_is_busy(vdev, file))
+		return -EBUSY;
+	return vb2_ext_prepare_buf(vdev->queue, vdev->v4l2_dev->mdev, p);
+}
+EXPORT_SYMBOL_GPL(vb2_ioctl_ext_prepare_buf);
+
 int vb2_ioctl_querybuf(struct file *file, void *priv, struct v4l2_buffer *p)
 {
 	struct video_device *vdev = video_devdata(file);
@@ -1060,6 +1095,16 @@ int vb2_ioctl_querybuf(struct file *file, void *priv, struct v4l2_buffer *p)
 }
 EXPORT_SYMBOL_GPL(vb2_ioctl_querybuf);
 
+int vb2_ioctl_ext_querybuf(struct file *file, void *priv,
+			   struct v4l2_ext_buffer *p)
+{
+	struct video_device *vdev = video_devdata(file);
+
+	/* No need to call vb2_queue_is_busy(), anyone can query buffers. */
+	return vb2_ext_querybuf(vdev->queue, p);
+}
+EXPORT_SYMBOL_GPL(vb2_ioctl_ext_querybuf);
+
 int vb2_ioctl_qbuf(struct file *file, void *priv, struct v4l2_buffer *p)
 {
 	struct video_device *vdev = video_devdata(file);
@@ -1070,6 +1115,17 @@ int vb2_ioctl_qbuf(struct file *file, void *priv, struct v4l2_buffer *p)
 }
 EXPORT_SYMBOL_GPL(vb2_ioctl_qbuf);
 
+int vb2_ioctl_ext_qbuf(struct file *file, void *priv,
+		       struct v4l2_ext_buffer *p)
+{
+	struct video_device *vdev = video_devdata(file);
+
+	if (vb2_queue_is_busy(vdev, file))
+		return -EBUSY;
+	return vb2_ext_qbuf(vdev->queue, vdev->v4l2_dev->mdev, p);
+}
+EXPORT_SYMBOL_GPL(vb2_ioctl_ext_qbuf);
+
 int vb2_ioctl_dqbuf(struct file *file, void *priv, struct v4l2_buffer *p)
 {
 	struct video_device *vdev = video_devdata(file);
@@ -1080,6 +1136,17 @@ int vb2_ioctl_dqbuf(struct file *file, void *priv, struct v4l2_buffer *p)
 }
 EXPORT_SYMBOL_GPL(vb2_ioctl_dqbuf);
 
+int vb2_ioctl_ext_dqbuf(struct file *file, void *priv,
+			struct v4l2_ext_buffer *p)
+{
+	struct video_device *vdev = video_devdata(file);
+
+	if (vb2_queue_is_busy(vdev, file))
+		return -EBUSY;
+	return vb2_ext_dqbuf(vdev->queue, p, file->f_flags & O_NONBLOCK);
+}
+EXPORT_SYMBOL_GPL(vb2_ioctl_ext_dqbuf);
+
 int vb2_ioctl_streamon(struct file *file, void *priv, enum v4l2_buf_type i)
 {
 	struct video_device *vdev = video_devdata(file);
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index 52ef92049073e..f334767785bd9 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -152,6 +152,8 @@ struct vb2_mem_ops {
  * @mem_priv:	private data with this plane.
  * @dbuf:	dma_buf - shared buffer object.
  * @dbuf_mapped:	flag to show whether dbuf is mapped or not
+ * @dbuf_offset: offset where the plane starts. Usually 0, unless the buffer
+ *		 is shared by all planes of a multi-planar format.
  * @bytesused:	number of bytes occupied by data in the plane (payload).
  * @length:	size of this plane (NOT the payload) in bytes.
  * @min_length:	minimum required size of this plane (NOT the payload) in bytes.
@@ -175,6 +177,7 @@ struct vb2_plane {
 	void			*mem_priv;
 	struct dma_buf		*dbuf;
 	unsigned int		dbuf_mapped;
+	unsigned int		dbuf_offset;
 	unsigned int		bytesused;
 	unsigned int		length;
 	unsigned int		min_length;
@@ -446,7 +449,8 @@ struct vb2_ops {
  *			struct vb2_buffer.
  *			For V4L2 this is a &struct vb2_v4l2_buffer.
  * @fill_user_buffer:	given a &vb2_buffer fill in the userspace structure.
- *			For V4L2 this is a &struct v4l2_buffer.
+ *			For V4L2 this is a &struct v4l2_buffer or
+ *			&struct v4l2_ext_buffer.
  * @fill_vb2_buffer:	given a userspace structure, fill in the &vb2_buffer.
  *			If the userspace structure is invalid, then this op
  *			will return an error.
diff --git a/include/media/videobuf2-v4l2.h b/include/media/videobuf2-v4l2.h
index b7b5a9cb5a280..1bfde6e391bf0 100644
--- a/include/media/videobuf2-v4l2.h
+++ b/include/media/videobuf2-v4l2.h
@@ -37,7 +37,7 @@
  * @planes:	plane information (userptr/fd, length, bytesused, data_offset).
  *
  * Should contain enough information to be able to cover all the fields
- * of &struct v4l2_buffer at ``videodev2.h``.
+ * of &struct v4l2_buffer and &struct v4l2_ext_buffer at ``videodev2.h``.
  */
 struct vb2_v4l2_buffer {
 	struct vb2_buffer	vb2_buf;
@@ -77,6 +77,7 @@ int vb2_find_timestamp(const struct vb2_queue *q, u64 timestamp,
 		       unsigned int start_idx);
 
 int vb2_querybuf(struct vb2_queue *q, struct v4l2_buffer *b);
+int vb2_ext_querybuf(struct vb2_queue *q, struct v4l2_ext_buffer *b);
 
 /**
  * vb2_reqbufs() - Wrapper for vb2_core_reqbufs() that also verifies
@@ -97,6 +98,8 @@ int vb2_reqbufs(struct vb2_queue *q, struct v4l2_requestbuffers *req);
  *		&v4l2_ioctl_ops->vidioc_create_bufs handler in driver
  */
 int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create);
+int vb2_ext_create_bufs(struct vb2_queue *q,
+			struct v4l2_ext_create_buffers *create);
 
 /**
  * vb2_prepare_buf() - Pass ownership of a buffer from userspace to the kernel
@@ -122,6 +125,8 @@ int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create);
  */
 int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
 		    struct v4l2_buffer *b);
+int vb2_ext_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
+			struct v4l2_ext_buffer *b);
 
 /**
  * vb2_qbuf() - Queue a buffer from userspace
@@ -148,6 +153,8 @@ int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
  */
 int vb2_qbuf(struct vb2_queue *q, struct media_device *mdev,
 	     struct v4l2_buffer *b);
+int vb2_ext_qbuf(struct vb2_queue *q, struct media_device *mdev,
+		 struct v4l2_ext_buffer *b);
 
 /**
  * vb2_expbuf() - Export a buffer as a file descriptor
@@ -185,6 +192,8 @@ int vb2_expbuf(struct vb2_queue *q, struct v4l2_exportbuffer *eb);
  * from &v4l2_ioctl_ops->vidioc_dqbuf handler in driver.
  */
 int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, bool nonblocking);
+int vb2_ext_dqbuf(struct vb2_queue *q, struct v4l2_ext_buffer *b,
+		  bool nonblocking);
 
 /**
  * vb2_streamon - start streaming
@@ -294,11 +303,21 @@ int vb2_ioctl_reqbufs(struct file *file, void *priv,
 			  struct v4l2_requestbuffers *p);
 int vb2_ioctl_create_bufs(struct file *file, void *priv,
 			  struct v4l2_create_buffers *p);
+int vb2_ioctl_ext_create_bufs(struct file *file, void *priv,
+			      struct v4l2_ext_create_buffers *p);
 int vb2_ioctl_prepare_buf(struct file *file, void *priv,
 			  struct v4l2_buffer *p);
+int vb2_ioctl_ext_prepare_buf(struct file *file, void *priv,
+			      struct v4l2_ext_buffer *p);
 int vb2_ioctl_querybuf(struct file *file, void *priv, struct v4l2_buffer *p);
+int vb2_ioctl_ext_querybuf(struct file *file, void *priv,
+			   struct v4l2_ext_buffer *p);
 int vb2_ioctl_qbuf(struct file *file, void *priv, struct v4l2_buffer *p);
+int vb2_ioctl_ext_qbuf(struct file *file, void *priv,
+		       struct v4l2_ext_buffer *p);
 int vb2_ioctl_dqbuf(struct file *file, void *priv, struct v4l2_buffer *p);
+int vb2_ioctl_ext_dqbuf(struct file *file, void *priv,
+			struct v4l2_ext_buffer *p);
 int vb2_ioctl_streamon(struct file *file, void *priv, enum v4l2_buf_type i);
 int vb2_ioctl_streamoff(struct file *file, void *priv, enum v4l2_buf_type i);
 int vb2_ioctl_expbuf(struct file *file, void *priv,
-- 
2.26.0


^ permalink raw reply related	[flat|nested] 29+ messages in thread

* [PATCH v4 4/6] media: mediabus: Add helpers to convert a ext_pix format to/from a mbus_fmt
  2020-07-17 11:54 [PATCH v4 0/6] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
                   ` (2 preceding siblings ...)
  2020-07-17 11:54 ` [PATCH v4 3/6] media: videobuf2: Expose helpers to implement the _ext_fmt and _ext_buf hooks Helen Koike
@ 2020-07-17 11:54 ` Helen Koike
  2020-07-17 11:54 ` [PATCH v4 5/6] media: vivid: Convert the capture and output drivers to EXT_FMT/EXT_BUF Helen Koike
                   ` (4 subsequent siblings)
  8 siblings, 0 replies; 29+ messages in thread
From: Helen Koike @ 2020-07-17 11:54 UTC (permalink / raw)
  To: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus, linux-media
  Cc: Boris Brezillon, Helen Koike, tfiga, hiroh, nicolas,
	Brian.Starkey, kernel, narmstrong, linux-kernel, frkoenig,
	mjourdan, stanimir.varbanov

From: Boris Brezillon <boris.brezillon@collabora.com>

Just a new version of v4l2_fill_mbus_format() and v4l2_fill_ext_pix_format()
to deal with the new v4l2_ext_pix_format struct.
This is needed to convert the VIMC driver to the EXT_FMT/EXT_BUF iocts.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Signed-off-by: Helen Koike <helen.koike@collabora.com>
---
Changes in v4:
- Add helper v4l2_fill_ext_pix_format()
- Rebased on top of media/master (post 5.8-rc1)

Changes in v3:
- Rebased on top of media/master (post 5.4-rc1)

Changes in v2:
- New patch
---
 include/media/v4l2-mediabus.h | 42 +++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)

diff --git a/include/media/v4l2-mediabus.h b/include/media/v4l2-mediabus.h
index 45f88f0248c4e..8133407377f7d 100644
--- a/include/media/v4l2-mediabus.h
+++ b/include/media/v4l2-mediabus.h
@@ -119,6 +119,26 @@ v4l2_fill_pix_format(struct v4l2_pix_format *pix_fmt,
 	pix_fmt->xfer_func = mbus_fmt->xfer_func;
 }
 
+/**
+ * v4l2_fill_ext_pix_format - Ancillary routine that fills a &struct
+ *	v4l2_ext_pix_format fields from a &struct v4l2_mbus_framefmt.
+ *
+ * @pix_fmt:	pointer to &struct v4l2_ext_pix_format to be filled
+ * @mbus_fmt:	pointer to &struct v4l2_mbus_framefmt to be used as model
+ */
+static inline void
+v4l2_fill_ext_pix_format(struct v4l2_ext_pix_format *pix_fmt,
+		     const struct v4l2_mbus_framefmt *mbus_fmt)
+{
+	pix_fmt->width = mbus_fmt->width;
+	pix_fmt->height = mbus_fmt->height;
+	pix_fmt->field = mbus_fmt->field;
+	pix_fmt->colorspace = mbus_fmt->colorspace;
+	pix_fmt->ycbcr_enc = mbus_fmt->ycbcr_enc;
+	pix_fmt->quantization = mbus_fmt->quantization;
+	pix_fmt->xfer_func = mbus_fmt->xfer_func;
+}
+
 /**
  * v4l2_fill_pix_format - Ancillary routine that fills a &struct
  *	v4l2_mbus_framefmt from a &struct v4l2_pix_format and a
@@ -182,4 +202,26 @@ v4l2_fill_mbus_format_mplane(struct v4l2_mbus_framefmt *mbus_fmt,
 	mbus_fmt->xfer_func = pix_mp_fmt->xfer_func;
 }
 
+/**
+ * v4l2_fill_mbus_format_ext - Ancillary routine that fills a &struct
+ *	v4l2_mbus_framefmt from a &struct v4l2_ext_pix_format.
+ *
+ * @mbus_fmt:	pointer to &struct v4l2_mbus_framefmt to be filled
+ * @pix_fmt:	pointer to &struct v4l2_ext_pix_format to be used as model
+ * @code:	data format code (from &enum v4l2_mbus_pixelcode)
+ */
+static inline void
+v4l2_fill_mbus_format_ext(struct v4l2_mbus_framefmt *mbus_fmt,
+			  const struct v4l2_ext_pix_format *pix_fmt, u32 code)
+{
+	mbus_fmt->width = pix_fmt->width;
+	mbus_fmt->height = pix_fmt->height;
+	mbus_fmt->field = pix_fmt->field;
+	mbus_fmt->colorspace = pix_fmt->colorspace;
+	mbus_fmt->ycbcr_enc = pix_fmt->ycbcr_enc;
+	mbus_fmt->quantization = pix_fmt->quantization;
+	mbus_fmt->xfer_func = pix_fmt->xfer_func;
+	mbus_fmt->code = code;
+}
+
 #endif
-- 
2.26.0


^ permalink raw reply related	[flat|nested] 29+ messages in thread

* [PATCH v4 5/6] media: vivid: Convert the capture and output drivers to EXT_FMT/EXT_BUF
  2020-07-17 11:54 [PATCH v4 0/6] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
                   ` (3 preceding siblings ...)
  2020-07-17 11:54 ` [PATCH v4 4/6] media: mediabus: Add helpers to convert a ext_pix format to/from a mbus_fmt Helen Koike
@ 2020-07-17 11:54 ` Helen Koike
  2020-07-17 11:54 ` [PATCH v4 6/6] media: vimc: Implement the ext_fmt and ext_buf hooks Helen Koike
                   ` (3 subsequent siblings)
  8 siblings, 0 replies; 29+ messages in thread
From: Helen Koike @ 2020-07-17 11:54 UTC (permalink / raw)
  To: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus, linux-media
  Cc: Boris Brezillon, Helen Koike, tfiga, hiroh, nicolas,
	Brian.Starkey, kernel, narmstrong, linux-kernel, frkoenig,
	mjourdan, stanimir.varbanov

From: Boris Brezillon <boris.brezillon@collabora.com>

This should simplify things a bit as we now have a single
implementation instead of the MPLANE and !MPLANE versions.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Signed-off-by: Helen Koike <helen.koike@collabora.com>
---
Changes in v4:
- Update with new format and buffer structs
- Rebased on top of media/master (post 5.8-rc1)

Changes in v3:
- Rebased on top of media/master (post 5.4-rc1)

Changes in v2:
- New patch
---
 drivers/media/test-drivers/vivid/vivid-core.c |  70 ++-----
 .../test-drivers/vivid/vivid-touch-cap.c      |  26 +--
 .../test-drivers/vivid/vivid-touch-cap.h      |   3 +-
 .../media/test-drivers/vivid/vivid-vid-cap.c  | 169 +++++----------
 .../media/test-drivers/vivid/vivid-vid-cap.h  |  15 +-
 .../media/test-drivers/vivid/vivid-vid-out.c  | 193 ++++++------------
 .../media/test-drivers/vivid/vivid-vid-out.h  |  15 +-
 7 files changed, 154 insertions(+), 337 deletions(-)

diff --git a/drivers/media/test-drivers/vivid/vivid-core.c b/drivers/media/test-drivers/vivid/vivid-core.c
index f7ee37e9508db..69b9433487150 100644
--- a/drivers/media/test-drivers/vivid/vivid-core.c
+++ b/drivers/media/test-drivers/vivid/vivid-core.c
@@ -488,63 +488,33 @@ static int vivid_enum_fmt_cap(struct file *file, void  *priv,
 }
 
 static int vivid_g_fmt_cap(struct file *file, void *priv,
-			   struct v4l2_format *f)
+			   struct v4l2_ext_pix_format *f)
 {
 	struct video_device *vdev = video_devdata(file);
 
 	if (vdev->vfl_type == VFL_TYPE_TOUCH)
 		return vivid_g_fmt_tch(file, priv, f);
-	return vidioc_g_fmt_vid_cap(file, priv, f);
+	return vivid_g_fmt_vid_cap(file, priv, f);
 }
 
 static int vivid_try_fmt_cap(struct file *file, void *priv,
-			     struct v4l2_format *f)
+			     struct v4l2_ext_pix_format *f)
 {
 	struct video_device *vdev = video_devdata(file);
 
 	if (vdev->vfl_type == VFL_TYPE_TOUCH)
 		return vivid_g_fmt_tch(file, priv, f);
-	return vidioc_try_fmt_vid_cap(file, priv, f);
+	return vivid_try_fmt_vid_cap(file, priv, f);
 }
 
 static int vivid_s_fmt_cap(struct file *file, void *priv,
-			   struct v4l2_format *f)
+			   struct v4l2_ext_pix_format *f)
 {
 	struct video_device *vdev = video_devdata(file);
 
 	if (vdev->vfl_type == VFL_TYPE_TOUCH)
 		return vivid_g_fmt_tch(file, priv, f);
-	return vidioc_s_fmt_vid_cap(file, priv, f);
-}
-
-static int vivid_g_fmt_cap_mplane(struct file *file, void *priv,
-				  struct v4l2_format *f)
-{
-	struct video_device *vdev = video_devdata(file);
-
-	if (vdev->vfl_type == VFL_TYPE_TOUCH)
-		return vivid_g_fmt_tch_mplane(file, priv, f);
-	return vidioc_g_fmt_vid_cap_mplane(file, priv, f);
-}
-
-static int vivid_try_fmt_cap_mplane(struct file *file, void *priv,
-				    struct v4l2_format *f)
-{
-	struct video_device *vdev = video_devdata(file);
-
-	if (vdev->vfl_type == VFL_TYPE_TOUCH)
-		return vivid_g_fmt_tch_mplane(file, priv, f);
-	return vidioc_try_fmt_vid_cap_mplane(file, priv, f);
-}
-
-static int vivid_s_fmt_cap_mplane(struct file *file, void *priv,
-				  struct v4l2_format *f)
-{
-	struct video_device *vdev = video_devdata(file);
-
-	if (vdev->vfl_type == VFL_TYPE_TOUCH)
-		return vivid_g_fmt_tch_mplane(file, priv, f);
-	return vidioc_s_fmt_vid_cap_mplane(file, priv, f);
+	return vivid_s_fmt_vid_cap(file, priv, f);
 }
 
 static bool vivid_is_in_use(struct video_device *vdev)
@@ -640,20 +610,14 @@ static const struct v4l2_ioctl_ops vivid_ioctl_ops = {
 	.vidioc_querycap		= vidioc_querycap,
 
 	.vidioc_enum_fmt_vid_cap	= vivid_enum_fmt_cap,
-	.vidioc_g_fmt_vid_cap		= vivid_g_fmt_cap,
-	.vidioc_try_fmt_vid_cap		= vivid_try_fmt_cap,
-	.vidioc_s_fmt_vid_cap		= vivid_s_fmt_cap,
-	.vidioc_g_fmt_vid_cap_mplane	= vivid_g_fmt_cap_mplane,
-	.vidioc_try_fmt_vid_cap_mplane	= vivid_try_fmt_cap_mplane,
-	.vidioc_s_fmt_vid_cap_mplane	= vivid_s_fmt_cap_mplane,
+	.vidioc_g_ext_pix_fmt_vid_cap	= vivid_g_fmt_cap,
+	.vidioc_try_ext_pix_fmt_vid_cap	= vivid_try_fmt_cap,
+	.vidioc_s_ext_pix_fmt_vid_cap	= vivid_s_fmt_cap,
 
 	.vidioc_enum_fmt_vid_out	= vivid_enum_fmt_vid,
-	.vidioc_g_fmt_vid_out		= vidioc_g_fmt_vid_out,
-	.vidioc_try_fmt_vid_out		= vidioc_try_fmt_vid_out,
-	.vidioc_s_fmt_vid_out		= vidioc_s_fmt_vid_out,
-	.vidioc_g_fmt_vid_out_mplane	= vidioc_g_fmt_vid_out_mplane,
-	.vidioc_try_fmt_vid_out_mplane	= vidioc_try_fmt_vid_out_mplane,
-	.vidioc_s_fmt_vid_out_mplane	= vidioc_s_fmt_vid_out_mplane,
+	.vidioc_g_ext_pix_fmt_vid_out	= vivid_g_fmt_vid_out,
+	.vidioc_try_ext_pix_fmt_vid_out	= vivid_try_fmt_vid_out,
+	.vidioc_s_ext_pix_fmt_vid_out	= vivid_s_fmt_vid_out,
 
 	.vidioc_g_selection		= vidioc_g_selection,
 	.vidioc_s_selection		= vidioc_s_selection,
@@ -698,11 +662,11 @@ static const struct v4l2_ioctl_ops vivid_ioctl_ops = {
 	.vidioc_s_fbuf			= vidioc_s_fbuf,
 
 	.vidioc_reqbufs			= vb2_ioctl_reqbufs,
-	.vidioc_create_bufs		= vb2_ioctl_create_bufs,
-	.vidioc_prepare_buf		= vb2_ioctl_prepare_buf,
-	.vidioc_querybuf		= vb2_ioctl_querybuf,
-	.vidioc_qbuf			= vb2_ioctl_qbuf,
-	.vidioc_dqbuf			= vb2_ioctl_dqbuf,
+	.vidioc_ext_create_bufs		= vb2_ioctl_ext_create_bufs,
+	.vidioc_ext_prepare_buf		= vb2_ioctl_ext_prepare_buf,
+	.vidioc_ext_querybuf		= vb2_ioctl_ext_querybuf,
+	.vidioc_ext_qbuf		= vb2_ioctl_ext_qbuf,
+	.vidioc_ext_dqbuf		= vb2_ioctl_ext_dqbuf,
 	.vidioc_expbuf			= vb2_ioctl_expbuf,
 	.vidioc_streamon		= vb2_ioctl_streamon,
 	.vidioc_streamoff		= vb2_ioctl_streamoff,
diff --git a/drivers/media/test-drivers/vivid/vivid-touch-cap.c b/drivers/media/test-drivers/vivid/vivid-touch-cap.c
index ebb00b128030c..6bbe7616e91a1 100644
--- a/drivers/media/test-drivers/vivid/vivid-touch-cap.c
+++ b/drivers/media/test-drivers/vivid/vivid-touch-cap.c
@@ -3,6 +3,8 @@
  * vivid-touch-cap.c - touch support functions.
  */
 
+#include <media/v4l2-ioctl.h>
+
 #include "vivid-core.h"
 #include "vivid-kthread-touch.h"
 #include "vivid-vid-common.h"
@@ -126,27 +128,19 @@ int vivid_enum_fmt_tch(struct file *file, void  *priv, struct v4l2_fmtdesc *f)
 	return 0;
 }
 
-int vivid_g_fmt_tch(struct file *file, void *priv, struct v4l2_format *f)
+int vivid_g_fmt_tch(struct file *file, void *priv, struct v4l2_ext_pix_format *e)
 {
 	struct vivid_dev *dev = video_drvdata(file);
+	struct v4l2_format f;
+	int ret;
 
-	if (dev->multiplanar)
-		return -ENOTTY;
-	f->fmt.pix = dev->tch_format;
-	return 0;
-}
+	ret = v4l2_ext_pix_format_to_format(e, &f, dev->multiplanar, false);
+	if (ret)
+		return ret;
 
-int vivid_g_fmt_tch_mplane(struct file *file, void *priv, struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-	struct v4l2_format sp_fmt;
+	f.fmt.pix = dev->tch_format;
 
-	if (!dev->multiplanar)
-		return -ENOTTY;
-	sp_fmt.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
-	sp_fmt.fmt.pix = dev->tch_format;
-	fmt_sp2mp(&sp_fmt, f);
-	return 0;
+	return v4l2_format_to_ext_pix_format(&f, e, false);
 }
 
 int vivid_g_parm_tch(struct file *file, void *priv,
diff --git a/drivers/media/test-drivers/vivid/vivid-touch-cap.h b/drivers/media/test-drivers/vivid/vivid-touch-cap.h
index 07e514046ae80..fccdf0a46824b 100644
--- a/drivers/media/test-drivers/vivid/vivid-touch-cap.h
+++ b/drivers/media/test-drivers/vivid/vivid-touch-cap.h
@@ -27,8 +27,7 @@ enum vivid_tch_test {
 extern const struct vb2_ops vivid_touch_cap_qops;
 
 int vivid_enum_fmt_tch(struct file *file, void  *priv, struct v4l2_fmtdesc *f);
-int vivid_g_fmt_tch(struct file *file, void *priv, struct v4l2_format *f);
-int vivid_g_fmt_tch_mplane(struct file *file, void *priv, struct v4l2_format *f);
+int vivid_g_fmt_tch(struct file *file, void *priv, struct v4l2_ext_pix_format *e);
 int vivid_enum_input_tch(struct file *file, void *priv, struct v4l2_input *inp);
 int vivid_g_input_tch(struct file *file, void *priv, unsigned int *i);
 int vivid_s_input_tch(struct file *file, void *priv, unsigned int i);
diff --git a/drivers/media/test-drivers/vivid/vivid-vid-cap.c b/drivers/media/test-drivers/vivid/vivid-vid-cap.c
index e94beef008c8e..dcd616c4719bb 100644
--- a/drivers/media/test-drivers/vivid/vivid-vid-cap.c
+++ b/drivers/media/test-drivers/vivid/vivid-vid-cap.c
@@ -521,28 +521,26 @@ static unsigned vivid_quantization_cap(struct vivid_dev *dev)
 }
 
 int vivid_g_fmt_vid_cap(struct file *file, void *priv,
-					struct v4l2_format *f)
+			struct v4l2_ext_pix_format *f)
 {
 	struct vivid_dev *dev = video_drvdata(file);
-	struct v4l2_pix_format_mplane *mp = &f->fmt.pix_mp;
 	unsigned p;
 
-	mp->width        = dev->fmt_cap_rect.width;
-	mp->height       = dev->fmt_cap_rect.height;
-	mp->field        = dev->field_cap;
-	mp->pixelformat  = dev->fmt_cap->fourcc;
-	mp->colorspace   = vivid_colorspace_cap(dev);
-	mp->xfer_func    = vivid_xfer_func_cap(dev);
+	f->width = dev->fmt_cap_rect.width;
+	f->height = dev->fmt_cap_rect.height;
+	f->field = dev->field_cap;
+	f->pixelformat = dev->fmt_cap->fourcc;
+	f->colorspace = vivid_colorspace_cap(dev);
+	f->xfer_func = vivid_xfer_func_cap(dev);
 	if (dev->fmt_cap->color_enc == TGP_COLOR_ENC_HSV)
-		mp->hsv_enc    = vivid_hsv_enc_cap(dev);
+		f->hsv_enc = vivid_hsv_enc_cap(dev);
 	else
-		mp->ycbcr_enc    = vivid_ycbcr_enc_cap(dev);
-	mp->quantization = vivid_quantization_cap(dev);
-	mp->num_planes = dev->fmt_cap->buffers;
-	for (p = 0; p < mp->num_planes; p++) {
-		mp->plane_fmt[p].bytesperline = tpg_g_bytesperline(&dev->tpg, p);
-		mp->plane_fmt[p].sizeimage =
-			(tpg_g_line_width(&dev->tpg, p) * mp->height) /
+		f->ycbcr_enc    = vivid_ycbcr_enc_cap(dev);
+	f->quantization = vivid_quantization_cap(dev);
+	for (p = 0; p < dev->fmt_cap->buffers; p++) {
+		f->plane_fmt[p].bytesperline = tpg_g_bytesperline(&dev->tpg, p);
+		f->plane_fmt[p].sizeimage =
+			(tpg_g_line_width(&dev->tpg, p) * f->height) /
 			dev->fmt_cap->vdownsampling[p] +
 			dev->fmt_cap->data_offset[p];
 	}
@@ -550,31 +548,30 @@ int vivid_g_fmt_vid_cap(struct file *file, void *priv,
 }
 
 int vivid_try_fmt_vid_cap(struct file *file, void *priv,
-			struct v4l2_format *f)
+			  struct v4l2_ext_pix_format *f)
 {
-	struct v4l2_pix_format_mplane *mp = &f->fmt.pix_mp;
-	struct v4l2_plane_pix_format *pfmt = mp->plane_fmt;
 	struct vivid_dev *dev = video_drvdata(file);
+	struct v4l2_plane_ext_pix_format *pfmt = f->plane_fmt;
 	const struct vivid_fmt *fmt;
 	unsigned bytesperline, max_bpl;
 	unsigned factor = 1;
 	unsigned w, h;
 	unsigned p;
 
-	fmt = vivid_get_format(dev, mp->pixelformat);
+	fmt = vivid_get_format(dev, f->pixelformat);
 	if (!fmt) {
 		dprintk(dev, 1, "Fourcc format (0x%08x) unknown.\n",
-			mp->pixelformat);
-		mp->pixelformat = V4L2_PIX_FMT_YUYV;
-		fmt = vivid_get_format(dev, mp->pixelformat);
+			f->pixelformat);
+		f->pixelformat = V4L2_PIX_FMT_YUYV;
+		fmt = vivid_get_format(dev, f->pixelformat);
 	}
 
-	mp->field = vivid_field_cap(dev, mp->field);
+	f->field = vivid_field_cap(dev, f->field);
 	if (vivid_is_webcam(dev)) {
 		const struct v4l2_frmsize_discrete *sz =
 			v4l2_find_nearest_size(webcam_sizes,
 					       VIVID_WEBCAM_SIZES, width,
-					       height, mp->width, mp->height);
+					       height, f->width, f->height);
 
 		w = sz->width;
 		h = sz->height;
@@ -585,14 +582,14 @@ int vivid_try_fmt_vid_cap(struct file *file, void *priv,
 		w = dev->src_rect.width;
 		h = dev->src_rect.height;
 	}
-	if (V4L2_FIELD_HAS_T_OR_B(mp->field))
+	if (V4L2_FIELD_HAS_T_OR_B(f->field))
 		factor = 2;
 	if (vivid_is_webcam(dev) ||
 	    (!dev->has_scaler_cap && !dev->has_crop_cap && !dev->has_compose_cap)) {
-		mp->width = w;
-		mp->height = h / factor;
+		f->width = w;
+		f->height = h / factor;
 	} else {
-		struct v4l2_rect r = { 0, 0, mp->width, mp->height * factor };
+		struct v4l2_rect r = { 0, 0, f->width, f->height * factor };
 
 		v4l2_rect_set_min_size(&r, &vivid_min_rect);
 		v4l2_rect_set_max_size(&r, &vivid_max_rect);
@@ -605,16 +602,15 @@ int vivid_try_fmt_vid_cap(struct file *file, void *priv,
 		} else if (!dev->has_scaler_cap && !dev->has_crop_cap) {
 			v4l2_rect_set_min_size(&r, &dev->src_rect);
 		}
-		mp->width = r.width;
-		mp->height = r.height / factor;
+		f->width = r.width;
+		f->height = r.height / factor;
 	}
 
 	/* This driver supports custom bytesperline values */
 
-	mp->num_planes = fmt->buffers;
 	for (p = 0; p < fmt->buffers; p++) {
 		/* Calculate the minimum supported bytesperline value */
-		bytesperline = (mp->width * fmt->bit_depth[p]) >> 3;
+		bytesperline = (f->width * fmt->bit_depth[p]) >> 3;
 		/* Calculate the maximum supported bytesperline value */
 		max_bpl = (MAX_ZOOM * MAX_WIDTH * fmt->bit_depth[p]) >> 3;
 
@@ -623,31 +619,27 @@ int vivid_try_fmt_vid_cap(struct file *file, void *priv,
 		if (pfmt[p].bytesperline < bytesperline)
 			pfmt[p].bytesperline = bytesperline;
 
-		pfmt[p].sizeimage = (pfmt[p].bytesperline * mp->height) /
+		pfmt[p].sizeimage = (pfmt[p].bytesperline * f->height) /
 				fmt->vdownsampling[p] + fmt->data_offset[p];
-
-		memset(pfmt[p].reserved, 0, sizeof(pfmt[p].reserved));
 	}
 	for (p = fmt->buffers; p < fmt->planes; p++)
-		pfmt[0].sizeimage += (pfmt[0].bytesperline * mp->height *
+		pfmt[0].sizeimage += (pfmt[0].bytesperline * f->height *
 			(fmt->bit_depth[p] / fmt->vdownsampling[p])) /
 			(fmt->bit_depth[0] / fmt->vdownsampling[0]);
 
-	mp->colorspace = vivid_colorspace_cap(dev);
+	f->colorspace = vivid_colorspace_cap(dev);
 	if (fmt->color_enc == TGP_COLOR_ENC_HSV)
-		mp->hsv_enc = vivid_hsv_enc_cap(dev);
+		f->hsv_enc = vivid_hsv_enc_cap(dev);
 	else
-		mp->ycbcr_enc = vivid_ycbcr_enc_cap(dev);
-	mp->xfer_func = vivid_xfer_func_cap(dev);
-	mp->quantization = vivid_quantization_cap(dev);
-	memset(mp->reserved, 0, sizeof(mp->reserved));
+		f->ycbcr_enc = vivid_ycbcr_enc_cap(dev);
+	f->xfer_func = vivid_xfer_func_cap(dev);
+	f->quantization = vivid_quantization_cap(dev);
 	return 0;
 }
 
 int vivid_s_fmt_vid_cap(struct file *file, void *priv,
-					struct v4l2_format *f)
+			struct v4l2_ext_pix_format *f)
 {
-	struct v4l2_pix_format_mplane *mp = &f->fmt.pix_mp;
 	struct vivid_dev *dev = video_drvdata(file);
 	struct v4l2_rect *crop = &dev->crop_cap;
 	struct v4l2_rect *compose = &dev->compose_cap;
@@ -665,20 +657,21 @@ int vivid_s_fmt_vid_cap(struct file *file, void *priv,
 		return -EBUSY;
 	}
 
-	if (dev->overlay_cap_owner && dev->fb_cap.fmt.pixelformat != mp->pixelformat) {
+	if (dev->overlay_cap_owner &&
+	    dev->fb_cap.fmt.pixelformat != f->pixelformat) {
 		dprintk(dev, 1, "overlay is active, can't change pixelformat\n");
 		return -EBUSY;
 	}
 
-	dev->fmt_cap = vivid_get_format(dev, mp->pixelformat);
-	if (V4L2_FIELD_HAS_T_OR_B(mp->field))
+	dev->fmt_cap = vivid_get_format(dev, f->pixelformat);
+	if (V4L2_FIELD_HAS_T_OR_B(f->field))
 		factor = 2;
 
 	/* Note: the webcam input doesn't support scaling, cropping or composing */
 
 	if (!vivid_is_webcam(dev) &&
 	    (dev->has_scaler_cap || dev->has_crop_cap || dev->has_compose_cap)) {
-		struct v4l2_rect r = { 0, 0, mp->width, mp->height };
+		struct v4l2_rect r = { 0, 0, f->width, f->height };
 
 		if (dev->has_scaler_cap) {
 			if (dev->has_compose_cap)
@@ -739,99 +732,39 @@ int vivid_s_fmt_vid_cap(struct file *file, void *priv,
 	} else if (vivid_is_webcam(dev)) {
 		/* Guaranteed to be a match */
 		for (i = 0; i < ARRAY_SIZE(webcam_sizes); i++)
-			if (webcam_sizes[i].width == mp->width &&
-					webcam_sizes[i].height == mp->height)
+			if (webcam_sizes[i].width == f->width &&
+			    webcam_sizes[i].height == f->height)
 				break;
 		dev->webcam_size_idx = i;
 		if (dev->webcam_ival_idx >= 2 * (VIVID_WEBCAM_SIZES - i))
 			dev->webcam_ival_idx = 2 * (VIVID_WEBCAM_SIZES - i) - 1;
 		vivid_update_format_cap(dev, false);
 	} else {
-		struct v4l2_rect r = { 0, 0, mp->width, mp->height };
+		struct v4l2_rect r = { 0, 0, f->width, f->height };
 
 		v4l2_rect_set_size_to(compose, &r);
 		r.height *= factor;
 		v4l2_rect_set_size_to(crop, &r);
 	}
 
-	dev->fmt_cap_rect.width = mp->width;
-	dev->fmt_cap_rect.height = mp->height;
-	tpg_s_buf_height(&dev->tpg, mp->height);
+	dev->fmt_cap_rect.width = f->width;
+	dev->fmt_cap_rect.height = f->height;
+	tpg_s_buf_height(&dev->tpg, f->height);
 	tpg_s_fourcc(&dev->tpg, dev->fmt_cap->fourcc);
 	for (p = 0; p < tpg_g_buffers(&dev->tpg); p++)
-		tpg_s_bytesperline(&dev->tpg, p, mp->plane_fmt[p].bytesperline);
-	dev->field_cap = mp->field;
+		tpg_s_bytesperline(&dev->tpg, p, f->plane_fmt[p].bytesperline);
+	dev->field_cap = f->field;
 	if (dev->field_cap == V4L2_FIELD_ALTERNATE)
 		tpg_s_field(&dev->tpg, V4L2_FIELD_TOP, true);
 	else
 		tpg_s_field(&dev->tpg, dev->field_cap, false);
 	tpg_s_crop_compose(&dev->tpg, &dev->crop_cap, &dev->compose_cap);
 	if (vivid_is_sdtv_cap(dev))
-		dev->tv_field_cap = mp->field;
+		dev->tv_field_cap = f->field;
 	tpg_update_mv_step(&dev->tpg);
 	return 0;
 }
 
-int vidioc_g_fmt_vid_cap_mplane(struct file *file, void *priv,
-					struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (!dev->multiplanar)
-		return -ENOTTY;
-	return vivid_g_fmt_vid_cap(file, priv, f);
-}
-
-int vidioc_try_fmt_vid_cap_mplane(struct file *file, void *priv,
-			struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (!dev->multiplanar)
-		return -ENOTTY;
-	return vivid_try_fmt_vid_cap(file, priv, f);
-}
-
-int vidioc_s_fmt_vid_cap_mplane(struct file *file, void *priv,
-			struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (!dev->multiplanar)
-		return -ENOTTY;
-	return vivid_s_fmt_vid_cap(file, priv, f);
-}
-
-int vidioc_g_fmt_vid_cap(struct file *file, void *priv,
-					struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (dev->multiplanar)
-		return -ENOTTY;
-	return fmt_sp2mp_func(file, priv, f, vivid_g_fmt_vid_cap);
-}
-
-int vidioc_try_fmt_vid_cap(struct file *file, void *priv,
-			struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (dev->multiplanar)
-		return -ENOTTY;
-	return fmt_sp2mp_func(file, priv, f, vivid_try_fmt_vid_cap);
-}
-
-int vidioc_s_fmt_vid_cap(struct file *file, void *priv,
-			struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (dev->multiplanar)
-		return -ENOTTY;
-	return fmt_sp2mp_func(file, priv, f, vivid_s_fmt_vid_cap);
-}
-
 int vivid_vid_cap_g_selection(struct file *file, void *priv,
 			      struct v4l2_selection *sel)
 {
diff --git a/drivers/media/test-drivers/vivid/vivid-vid-cap.h b/drivers/media/test-drivers/vivid/vivid-vid-cap.h
index 1e422a59eeabf..7c9fc5c787b59 100644
--- a/drivers/media/test-drivers/vivid/vivid-vid-cap.h
+++ b/drivers/media/test-drivers/vivid/vivid-vid-cap.h
@@ -17,15 +17,12 @@ extern const char * const vivid_ctrl_standard_strings[];
 
 extern const struct vb2_ops vivid_vid_cap_qops;
 
-int vivid_g_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f);
-int vivid_try_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f);
-int vivid_s_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_g_fmt_vid_cap_mplane(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_try_fmt_vid_cap_mplane(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_s_fmt_vid_cap_mplane(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_g_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_try_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_s_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f);
+int vivid_g_fmt_vid_cap(struct file *file, void *priv,
+			struct v4l2_ext_pix_format *f);
+int vivid_try_fmt_vid_cap(struct file *file, void *priv,
+			  struct v4l2_ext_pix_format *f);
+int vivid_s_fmt_vid_cap(struct file *file, void *priv,
+			struct v4l2_ext_pix_format *f);
 int vivid_vid_cap_g_selection(struct file *file, void *priv, struct v4l2_selection *sel);
 int vivid_vid_cap_s_selection(struct file *file, void *fh, struct v4l2_selection *s);
 int vivid_vid_cap_g_pixelaspect(struct file *file, void *priv, int type, struct v4l2_fract *f);
diff --git a/drivers/media/test-drivers/vivid/vivid-vid-out.c b/drivers/media/test-drivers/vivid/vivid-vid-out.c
index ee3446e3217cc..1da6bd5a8d121 100644
--- a/drivers/media/test-drivers/vivid/vivid-vid-out.c
+++ b/drivers/media/test-drivers/vivid/vivid-vid-out.c
@@ -315,59 +315,56 @@ static enum tpg_pixel_aspect vivid_get_pixel_aspect(const struct vivid_dev *dev)
 }
 
 int vivid_g_fmt_vid_out(struct file *file, void *priv,
-					struct v4l2_format *f)
+			struct v4l2_ext_pix_format *f)
 {
 	struct vivid_dev *dev = video_drvdata(file);
-	struct v4l2_pix_format_mplane *mp = &f->fmt.pix_mp;
 	const struct vivid_fmt *fmt = dev->fmt_out;
 	unsigned p;
 
-	mp->width        = dev->fmt_out_rect.width;
-	mp->height       = dev->fmt_out_rect.height;
-	mp->field        = dev->field_out;
-	mp->pixelformat  = fmt->fourcc;
-	mp->colorspace   = dev->colorspace_out;
-	mp->xfer_func    = dev->xfer_func_out;
-	mp->ycbcr_enc    = dev->ycbcr_enc_out;
-	mp->quantization = dev->quantization_out;
-	mp->num_planes = fmt->buffers;
-	for (p = 0; p < mp->num_planes; p++) {
-		mp->plane_fmt[p].bytesperline = dev->bytesperline_out[p];
-		mp->plane_fmt[p].sizeimage =
-			mp->plane_fmt[p].bytesperline * mp->height +
+	f->width        = dev->fmt_out_rect.width;
+	f->height       = dev->fmt_out_rect.height;
+	f->field        = dev->field_out;
+	f->pixelformat  = fmt->fourcc;
+	f->colorspace   = dev->colorspace_out;
+	f->xfer_func    = dev->xfer_func_out;
+	f->ycbcr_enc    = dev->ycbcr_enc_out;
+	f->quantization = dev->quantization_out;
+	for (p = 0; p < fmt->buffers; p++) {
+		f->plane_fmt[p].bytesperline = dev->bytesperline_out[p];
+		f->plane_fmt[p].sizeimage =
+			f->plane_fmt[p].bytesperline * f->height +
 			fmt->data_offset[p];
 	}
 	for (p = fmt->buffers; p < fmt->planes; p++) {
 		unsigned stride = dev->bytesperline_out[p];
 
-		mp->plane_fmt[0].sizeimage +=
-			(stride * mp->height) / fmt->vdownsampling[p];
+		f->plane_fmt[0].sizeimage +=
+			(stride * f->height) / fmt->vdownsampling[p];
 	}
 	return 0;
 }
 
 int vivid_try_fmt_vid_out(struct file *file, void *priv,
-			struct v4l2_format *f)
+			  struct v4l2_ext_pix_format *f)
 {
 	struct vivid_dev *dev = video_drvdata(file);
+	struct v4l2_plane_ext_pix_format *pfmt = f->plane_fmt;
 	struct v4l2_bt_timings *bt = &dev->dv_timings_out.bt;
-	struct v4l2_pix_format_mplane *mp = &f->fmt.pix_mp;
-	struct v4l2_plane_pix_format *pfmt = mp->plane_fmt;
 	const struct vivid_fmt *fmt;
 	unsigned bytesperline, max_bpl;
 	unsigned factor = 1;
 	unsigned w, h;
 	unsigned p;
 
-	fmt = vivid_get_format(dev, mp->pixelformat);
+	fmt = vivid_get_format(dev, f->pixelformat);
 	if (!fmt) {
 		dprintk(dev, 1, "Fourcc format (0x%08x) unknown.\n",
-			mp->pixelformat);
-		mp->pixelformat = V4L2_PIX_FMT_YUYV;
-		fmt = vivid_get_format(dev, mp->pixelformat);
+			f->pixelformat);
+		f->pixelformat = V4L2_PIX_FMT_YUYV;
+		fmt = vivid_get_format(dev, f->pixelformat);
 	}
 
-	mp->field = vivid_field_out(dev, mp->field);
+	f->field = vivid_field_out(dev, f->field);
 	if (vivid_is_svid_out(dev)) {
 		w = 720;
 		h = (dev->std_out & V4L2_STD_525_60) ? 480 : 576;
@@ -375,13 +372,13 @@ int vivid_try_fmt_vid_out(struct file *file, void *priv,
 		w = dev->sink_rect.width;
 		h = dev->sink_rect.height;
 	}
-	if (V4L2_FIELD_HAS_T_OR_B(mp->field))
+	if (V4L2_FIELD_HAS_T_OR_B(f->field))
 		factor = 2;
 	if (!dev->has_scaler_out && !dev->has_crop_out && !dev->has_compose_out) {
-		mp->width = w;
-		mp->height = h / factor;
+		f->width = w;
+		f->height = h / factor;
 	} else {
-		struct v4l2_rect r = { 0, 0, mp->width, mp->height * factor };
+		struct v4l2_rect r = { 0, 0, f->width, f->height * factor };
 
 		v4l2_rect_set_min_size(&r, &vivid_min_rect);
 		v4l2_rect_set_max_size(&r, &vivid_max_rect);
@@ -394,16 +391,15 @@ int vivid_try_fmt_vid_out(struct file *file, void *priv,
 		} else if (!dev->has_scaler_out && !dev->has_compose_out) {
 			v4l2_rect_set_min_size(&r, &dev->sink_rect);
 		}
-		mp->width = r.width;
-		mp->height = r.height / factor;
+		f->width = r.width;
+		f->height = r.height / factor;
 	}
 
 	/* This driver supports custom bytesperline values */
 
-	mp->num_planes = fmt->buffers;
 	for (p = 0; p < fmt->buffers; p++) {
 		/* Calculate the minimum supported bytesperline value */
-		bytesperline = (mp->width * fmt->bit_depth[p]) >> 3;
+		bytesperline = (f->width * fmt->bit_depth[p]) >> 3;
 		/* Calculate the maximum supported bytesperline value */
 		max_bpl = (MAX_ZOOM * MAX_WIDTH * fmt->bit_depth[p]) >> 3;
 
@@ -412,42 +408,39 @@ int vivid_try_fmt_vid_out(struct file *file, void *priv,
 		if (pfmt[p].bytesperline < bytesperline)
 			pfmt[p].bytesperline = bytesperline;
 
-		pfmt[p].sizeimage = (pfmt[p].bytesperline * mp->height) /
+		pfmt[p].sizeimage = (pfmt[p].bytesperline * f->height) /
 				fmt->vdownsampling[p] + fmt->data_offset[p];
 
-		memset(pfmt[p].reserved, 0, sizeof(pfmt[p].reserved));
 	}
 	for (p = fmt->buffers; p < fmt->planes; p++)
-		pfmt[0].sizeimage += (pfmt[0].bytesperline * mp->height *
+		pfmt[0].sizeimage += (pfmt[0].bytesperline * f->height *
 			(fmt->bit_depth[p] / fmt->vdownsampling[p])) /
 			(fmt->bit_depth[0] / fmt->vdownsampling[0]);
 
-	mp->xfer_func = V4L2_XFER_FUNC_DEFAULT;
-	mp->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
-	mp->quantization = V4L2_QUANTIZATION_DEFAULT;
+	f->xfer_func = V4L2_XFER_FUNC_DEFAULT;
+	f->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
+	f->quantization = V4L2_QUANTIZATION_DEFAULT;
 	if (vivid_is_svid_out(dev)) {
-		mp->colorspace = V4L2_COLORSPACE_SMPTE170M;
+		f->colorspace = V4L2_COLORSPACE_SMPTE170M;
 	} else if (dev->dvi_d_out || !(bt->flags & V4L2_DV_FL_IS_CE_VIDEO)) {
-		mp->colorspace = V4L2_COLORSPACE_SRGB;
+		f->colorspace = V4L2_COLORSPACE_SRGB;
 		if (dev->dvi_d_out)
-			mp->quantization = V4L2_QUANTIZATION_LIM_RANGE;
+			f->quantization = V4L2_QUANTIZATION_LIM_RANGE;
 	} else if (bt->width == 720 && bt->height <= 576) {
-		mp->colorspace = V4L2_COLORSPACE_SMPTE170M;
-	} else if (mp->colorspace != V4L2_COLORSPACE_SMPTE170M &&
-		   mp->colorspace != V4L2_COLORSPACE_REC709 &&
-		   mp->colorspace != V4L2_COLORSPACE_OPRGB &&
-		   mp->colorspace != V4L2_COLORSPACE_BT2020 &&
-		   mp->colorspace != V4L2_COLORSPACE_SRGB) {
-		mp->colorspace = V4L2_COLORSPACE_REC709;
+		f->colorspace = V4L2_COLORSPACE_SMPTE170M;
+	} else if (f->colorspace != V4L2_COLORSPACE_SMPTE170M &&
+		   f->colorspace != V4L2_COLORSPACE_REC709 &&
+		   f->colorspace != V4L2_COLORSPACE_OPRGB &&
+		   f->colorspace != V4L2_COLORSPACE_BT2020 &&
+		   f->colorspace != V4L2_COLORSPACE_SRGB) {
+		f->colorspace = V4L2_COLORSPACE_REC709;
 	}
-	memset(mp->reserved, 0, sizeof(mp->reserved));
 	return 0;
 }
 
 int vivid_s_fmt_vid_out(struct file *file, void *priv,
-					struct v4l2_format *f)
+			struct v4l2_ext_pix_format *f)
 {
-	struct v4l2_pix_format_mplane *mp = &f->fmt.pix_mp;
 	struct vivid_dev *dev = video_drvdata(file);
 	struct v4l2_rect *crop = &dev->crop_out;
 	struct v4l2_rect *compose = &dev->compose_out;
@@ -461,10 +454,10 @@ int vivid_s_fmt_vid_out(struct file *file, void *priv,
 
 	if (vb2_is_busy(q) &&
 	    (vivid_is_svid_out(dev) ||
-	     mp->width != dev->fmt_out_rect.width ||
-	     mp->height != dev->fmt_out_rect.height ||
-	     mp->pixelformat != dev->fmt_out->fourcc ||
-	     mp->field != dev->field_out)) {
+	     f->width != dev->fmt_out_rect.width ||
+	     f->height != dev->fmt_out_rect.height ||
+	     f->pixelformat != dev->fmt_out->fourcc ||
+	     f->field != dev->field_out)) {
 		dprintk(dev, 1, "%s device busy\n", __func__);
 		return -EBUSY;
 	}
@@ -477,12 +470,12 @@ int vivid_s_fmt_vid_out(struct file *file, void *priv,
 	if (vb2_is_busy(q))
 		goto set_colorspace;
 
-	dev->fmt_out = vivid_get_format(dev, mp->pixelformat);
-	if (V4L2_FIELD_HAS_T_OR_B(mp->field))
+	dev->fmt_out = vivid_get_format(dev, f->pixelformat);
+	if (V4L2_FIELD_HAS_T_OR_B(f->field))
 		factor = 2;
 
 	if (dev->has_scaler_out || dev->has_crop_out || dev->has_compose_out) {
-		struct v4l2_rect r = { 0, 0, mp->width, mp->height };
+		struct v4l2_rect r = { 0, 0, f->width, f->height };
 
 		if (dev->has_scaler_out) {
 			if (dev->has_crop_out)
@@ -541,30 +534,30 @@ int vivid_s_fmt_vid_out(struct file *file, void *priv,
 			crop->height /= factor;
 		}
 	} else {
-		struct v4l2_rect r = { 0, 0, mp->width, mp->height };
+		struct v4l2_rect r = { 0, 0, f->width, f->height };
 
 		v4l2_rect_set_size_to(crop, &r);
 		r.height /= factor;
 		v4l2_rect_set_size_to(compose, &r);
 	}
 
-	dev->fmt_out_rect.width = mp->width;
-	dev->fmt_out_rect.height = mp->height;
-	for (p = 0; p < mp->num_planes; p++)
-		dev->bytesperline_out[p] = mp->plane_fmt[p].bytesperline;
+	dev->fmt_out_rect.width = f->width;
+	dev->fmt_out_rect.height = f->height;
+	for (p = 0; p < VIDEO_MAX_PLANES && f->plane_fmt[p].sizeimage; p++)
+		dev->bytesperline_out[p] = f->plane_fmt[p].bytesperline;
 	for (p = dev->fmt_out->buffers; p < dev->fmt_out->planes; p++)
 		dev->bytesperline_out[p] =
 			(dev->bytesperline_out[0] * dev->fmt_out->bit_depth[p]) /
 			dev->fmt_out->bit_depth[0];
-	dev->field_out = mp->field;
+	dev->field_out = f->field;
 	if (vivid_is_svid_out(dev))
-		dev->tv_field_out = mp->field;
+		dev->tv_field_out = f->field;
 
 set_colorspace:
-	dev->colorspace_out = mp->colorspace;
-	dev->xfer_func_out = mp->xfer_func;
-	dev->ycbcr_enc_out = mp->ycbcr_enc;
-	dev->quantization_out = mp->quantization;
+	dev->colorspace_out = f->colorspace;
+	dev->xfer_func_out = f->xfer_func;
+	dev->ycbcr_enc_out = f->ycbcr_enc;
+	dev->quantization_out = f->quantization;
 	if (dev->loop_video) {
 		vivid_send_source_change(dev, SVID);
 		vivid_send_source_change(dev, HDMI);
@@ -572,66 +565,6 @@ int vivid_s_fmt_vid_out(struct file *file, void *priv,
 	return 0;
 }
 
-int vidioc_g_fmt_vid_out_mplane(struct file *file, void *priv,
-					struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (!dev->multiplanar)
-		return -ENOTTY;
-	return vivid_g_fmt_vid_out(file, priv, f);
-}
-
-int vidioc_try_fmt_vid_out_mplane(struct file *file, void *priv,
-			struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (!dev->multiplanar)
-		return -ENOTTY;
-	return vivid_try_fmt_vid_out(file, priv, f);
-}
-
-int vidioc_s_fmt_vid_out_mplane(struct file *file, void *priv,
-			struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (!dev->multiplanar)
-		return -ENOTTY;
-	return vivid_s_fmt_vid_out(file, priv, f);
-}
-
-int vidioc_g_fmt_vid_out(struct file *file, void *priv,
-					struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (dev->multiplanar)
-		return -ENOTTY;
-	return fmt_sp2mp_func(file, priv, f, vivid_g_fmt_vid_out);
-}
-
-int vidioc_try_fmt_vid_out(struct file *file, void *priv,
-			struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (dev->multiplanar)
-		return -ENOTTY;
-	return fmt_sp2mp_func(file, priv, f, vivid_try_fmt_vid_out);
-}
-
-int vidioc_s_fmt_vid_out(struct file *file, void *priv,
-			struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (dev->multiplanar)
-		return -ENOTTY;
-	return fmt_sp2mp_func(file, priv, f, vivid_s_fmt_vid_out);
-}
-
 int vivid_vid_out_g_selection(struct file *file, void *priv,
 			      struct v4l2_selection *sel)
 {
diff --git a/drivers/media/test-drivers/vivid/vivid-vid-out.h b/drivers/media/test-drivers/vivid/vivid-vid-out.h
index 8d56314f4ea1f..b84dc578af362 100644
--- a/drivers/media/test-drivers/vivid/vivid-vid-out.h
+++ b/drivers/media/test-drivers/vivid/vivid-vid-out.h
@@ -12,15 +12,12 @@ extern const struct vb2_ops vivid_vid_out_qops;
 
 void vivid_update_format_out(struct vivid_dev *dev);
 
-int vivid_g_fmt_vid_out(struct file *file, void *priv, struct v4l2_format *f);
-int vivid_try_fmt_vid_out(struct file *file, void *priv, struct v4l2_format *f);
-int vivid_s_fmt_vid_out(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_g_fmt_vid_out_mplane(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_try_fmt_vid_out_mplane(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_s_fmt_vid_out_mplane(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_g_fmt_vid_out(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_try_fmt_vid_out(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_s_fmt_vid_out(struct file *file, void *priv, struct v4l2_format *f);
+int vivid_g_fmt_vid_out(struct file *file, void *priv,
+			struct v4l2_ext_pix_format *f);
+int vivid_try_fmt_vid_out(struct file *file, void *priv,
+			  struct v4l2_ext_pix_format *f);
+int vivid_s_fmt_vid_out(struct file *file, void *priv,
+			struct v4l2_ext_pix_format *f);
 int vivid_vid_out_g_selection(struct file *file, void *priv, struct v4l2_selection *sel);
 int vivid_vid_out_s_selection(struct file *file, void *fh, struct v4l2_selection *s);
 int vivid_vid_out_g_pixelaspect(struct file *file, void *priv, int type, struct v4l2_fract *f);
-- 
2.26.0


^ permalink raw reply related	[flat|nested] 29+ messages in thread

* [PATCH v4 6/6] media: vimc: Implement the ext_fmt and ext_buf hooks
  2020-07-17 11:54 [PATCH v4 0/6] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
                   ` (4 preceding siblings ...)
  2020-07-17 11:54 ` [PATCH v4 5/6] media: vivid: Convert the capture and output drivers to EXT_FMT/EXT_BUF Helen Koike
@ 2020-07-17 11:54 ` Helen Koike
  2020-07-20 11:57 ` [PATCH v4 0/6] media: v4l2: Add extended fmt and buffer ioctls Tomasz Figa
                   ` (2 subsequent siblings)
  8 siblings, 0 replies; 29+ messages in thread
From: Helen Koike @ 2020-07-17 11:54 UTC (permalink / raw)
  To: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus, linux-media
  Cc: Boris Brezillon, Helen Koike, tfiga, hiroh, nicolas,
	Brian.Starkey, kernel, narmstrong, linux-kernel, frkoenig,
	mjourdan, stanimir.varbanov

From: Boris Brezillon <boris.brezillon@collabora.com>

Convert the driver to the _ext_fmt and _ext_buf API.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Signed-off-by: Helen Koike <helen.koike@collabora.com>
---
Changes in v4:
- Update with new format and buffer structs
- Rebased on top of media/master (post 5.8-rc1)

Changes in v3:
- Rebased on top of media/master (post 5.4-rc1)

Changes in v2:
- New patch
---
 .../media/test-drivers/vimc/vimc-capture.c    | 61 +++++++++----------
 drivers/media/test-drivers/vimc/vimc-common.c |  6 +-
 drivers/media/test-drivers/vimc/vimc-common.h |  2 +-
 3 files changed, 34 insertions(+), 35 deletions(-)

diff --git a/drivers/media/test-drivers/vimc/vimc-capture.c b/drivers/media/test-drivers/vimc/vimc-capture.c
index c63496b17b9a5..0c8b86f0dd3ec 100644
--- a/drivers/media/test-drivers/vimc/vimc-capture.c
+++ b/drivers/media/test-drivers/vimc/vimc-capture.c
@@ -15,7 +15,7 @@
 struct vimc_cap_device {
 	struct vimc_ent_device ved;
 	struct video_device vdev;
-	struct v4l2_pix_format format;
+	struct v4l2_ext_pix_format format;
 	struct vb2_queue queue;
 	struct list_head buf_list;
 	/*
@@ -32,7 +32,7 @@ struct vimc_cap_device {
 	struct media_pad pad;
 };
 
-static const struct v4l2_pix_format fmt_default = {
+static const struct v4l2_ext_pix_format fmt_default = {
 	.width = 640,
 	.height = 480,
 	.pixelformat = V4L2_PIX_FMT_RGB24,
@@ -63,7 +63,7 @@ static int vimc_cap_querycap(struct file *file, void *priv,
 }
 
 static void vimc_cap_get_format(struct vimc_ent_device *ved,
-				struct v4l2_pix_format *fmt)
+				struct v4l2_ext_pix_format *fmt)
 {
 	struct vimc_cap_device *vcap = container_of(ved, struct vimc_cap_device,
 						    ved);
@@ -72,19 +72,18 @@ static void vimc_cap_get_format(struct vimc_ent_device *ved,
 }
 
 static int vimc_cap_g_fmt_vid_cap(struct file *file, void *priv,
-				  struct v4l2_format *f)
+				  struct v4l2_ext_pix_format *f)
 {
 	struct vimc_cap_device *vcap = video_drvdata(file);
 
-	f->fmt.pix = vcap->format;
+	*f = vcap->format;
 
 	return 0;
 }
 
 static int vimc_cap_try_fmt_vid_cap(struct file *file, void *priv,
-				    struct v4l2_format *f)
+				    struct v4l2_ext_pix_format *format)
 {
-	struct v4l2_pix_format *format = &f->fmt.pix;
 	const struct vimc_pix_map *vpix;
 
 	format->width = clamp_t(u32, format->width, VIMC_FRAME_MIN_WIDTH,
@@ -99,8 +98,9 @@ static int vimc_cap_try_fmt_vid_cap(struct file *file, void *priv,
 		vpix = vimc_pix_map_by_pixelformat(format->pixelformat);
 	}
 	/* TODO: Add support for custom bytesperline values */
-	format->bytesperline = format->width * vpix->bpp;
-	format->sizeimage = format->bytesperline * format->height;
+	format->plane_fmt[0].bytesperline = format->width * vpix->bpp;
+	format->plane_fmt[0].sizeimage = format->plane_fmt[0].bytesperline *
+					 format->height;
 
 	if (format->field == V4L2_FIELD_ANY)
 		format->field = fmt_default.field;
@@ -114,7 +114,7 @@ static int vimc_cap_try_fmt_vid_cap(struct file *file, void *priv,
 }
 
 static int vimc_cap_s_fmt_vid_cap(struct file *file, void *priv,
-				  struct v4l2_format *f)
+				  struct v4l2_ext_pix_format *f)
 {
 	struct vimc_cap_device *vcap = video_drvdata(file);
 	int ret;
@@ -136,12 +136,10 @@ static int vimc_cap_s_fmt_vid_cap(struct file *file, void *priv,
 		vcap->format.quantization, vcap->format.xfer_func,
 		vcap->format.ycbcr_enc,
 		/* new */
-		f->fmt.pix.width, f->fmt.pix.height,
-		f->fmt.pix.pixelformat,	f->fmt.pix.colorspace,
-		f->fmt.pix.quantization, f->fmt.pix.xfer_func,
-		f->fmt.pix.ycbcr_enc);
+		f->width, f->height, f->pixelformat, f->colorspace,
+		f->quantization, f->xfer_func, f->ycbcr_enc);
 
-	vcap->format = f->fmt.pix;
+	vcap->format = *f;
 
 	return 0;
 }
@@ -205,18 +203,18 @@ static const struct v4l2_file_operations vimc_cap_fops = {
 static const struct v4l2_ioctl_ops vimc_cap_ioctl_ops = {
 	.vidioc_querycap = vimc_cap_querycap,
 
-	.vidioc_g_fmt_vid_cap = vimc_cap_g_fmt_vid_cap,
-	.vidioc_s_fmt_vid_cap = vimc_cap_s_fmt_vid_cap,
-	.vidioc_try_fmt_vid_cap = vimc_cap_try_fmt_vid_cap,
+	.vidioc_g_ext_pix_fmt_vid_cap = vimc_cap_g_fmt_vid_cap,
+	.vidioc_s_ext_pix_fmt_vid_cap = vimc_cap_s_fmt_vid_cap,
+	.vidioc_try_ext_pix_fmt_vid_cap = vimc_cap_try_fmt_vid_cap,
 	.vidioc_enum_fmt_vid_cap = vimc_cap_enum_fmt_vid_cap,
 	.vidioc_enum_framesizes = vimc_cap_enum_framesizes,
 
 	.vidioc_reqbufs = vb2_ioctl_reqbufs,
-	.vidioc_create_bufs = vb2_ioctl_create_bufs,
-	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
-	.vidioc_querybuf = vb2_ioctl_querybuf,
-	.vidioc_qbuf = vb2_ioctl_qbuf,
-	.vidioc_dqbuf = vb2_ioctl_dqbuf,
+	.vidioc_ext_create_bufs = vb2_ioctl_ext_create_bufs,
+	.vidioc_ext_prepare_buf = vb2_ioctl_ext_prepare_buf,
+	.vidioc_ext_querybuf = vb2_ioctl_ext_querybuf,
+	.vidioc_ext_qbuf = vb2_ioctl_ext_qbuf,
+	.vidioc_ext_dqbuf = vb2_ioctl_ext_dqbuf,
 	.vidioc_expbuf = vb2_ioctl_expbuf,
 	.vidioc_streamon = vb2_ioctl_streamon,
 	.vidioc_streamoff = vb2_ioctl_streamoff,
@@ -298,10 +296,11 @@ static int vimc_cap_queue_setup(struct vb2_queue *vq, unsigned int *nbuffers,
 	struct vimc_cap_device *vcap = vb2_get_drv_priv(vq);
 
 	if (*nplanes)
-		return sizes[0] < vcap->format.sizeimage ? -EINVAL : 0;
+		return sizes[0] < vcap->format.plane_fmt[0].sizeimage ?
+		       -EINVAL : 0;
 	/* We don't support multiplanes for now */
 	*nplanes = 1;
-	sizes[0] = vcap->format.sizeimage;
+	sizes[0] = vcap->format.plane_fmt[0].sizeimage;
 
 	return 0;
 }
@@ -309,7 +308,7 @@ static int vimc_cap_queue_setup(struct vb2_queue *vq, unsigned int *nbuffers,
 static int vimc_cap_buffer_prepare(struct vb2_buffer *vb)
 {
 	struct vimc_cap_device *vcap = vb2_get_drv_priv(vb->vb2_queue);
-	unsigned long size = vcap->format.sizeimage;
+	unsigned long size = vcap->format.plane_fmt[0].sizeimage;
 
 	if (vb2_plane_size(vb, 0) < size) {
 		dev_err(vcap->ved.dev, "%s: buffer too small (%lu < %lu)\n",
@@ -385,11 +384,11 @@ static void *vimc_cap_process_frame(struct vimc_ent_device *ved,
 
 	vbuf = vb2_plane_vaddr(&vimc_buf->vb2.vb2_buf, 0);
 
-	memcpy(vbuf, frame, vcap->format.sizeimage);
+	memcpy(vbuf, frame, vcap->format.plane_fmt[0].sizeimage);
 
 	/* Set it as ready */
 	vb2_set_plane_payload(&vimc_buf->vb2.vb2_buf, 0,
-			      vcap->format.sizeimage);
+			      vcap->format.plane_fmt[0].sizeimage);
 	vb2_buffer_done(&vimc_buf->vb2.vb2_buf, VB2_BUF_STATE_DONE);
 	return NULL;
 }
@@ -447,9 +446,9 @@ static struct vimc_ent_device *vimc_cap_add(struct vimc_device *vimc,
 	/* Set default frame format */
 	vcap->format = fmt_default;
 	vpix = vimc_pix_map_by_pixelformat(vcap->format.pixelformat);
-	vcap->format.bytesperline = vcap->format.width * vpix->bpp;
-	vcap->format.sizeimage = vcap->format.bytesperline *
-				 vcap->format.height;
+	vcap->format.plane_fmt[0].bytesperline = vcap->format.width * vpix->bpp;
+	vcap->format.plane_fmt[0].sizeimage = vcap->format.plane_fmt[0].bytesperline *
+					      vcap->format.height;
 
 	/* Fill the vimc_ent_device struct */
 	vcap->ved.ent = &vcap->vdev.entity;
diff --git a/drivers/media/test-drivers/vimc/vimc-common.c b/drivers/media/test-drivers/vimc/vimc-common.c
index 7b27153c0728b..854f88be3e93e 100644
--- a/drivers/media/test-drivers/vimc/vimc-common.c
+++ b/drivers/media/test-drivers/vimc/vimc-common.c
@@ -236,7 +236,7 @@ const struct vimc_pix_map *vimc_pix_map_by_pixelformat(u32 pixelformat)
 }
 
 static int vimc_get_pix_format(struct media_pad *pad,
-			       struct v4l2_pix_format *fmt)
+			       struct v4l2_ext_pix_format *fmt)
 {
 	if (is_media_entity_v4l2_subdev(pad->entity)) {
 		struct v4l2_subdev *sd =
@@ -252,7 +252,7 @@ static int vimc_get_pix_format(struct media_pad *pad,
 		if (ret)
 			return ret;
 
-		v4l2_fill_pix_format(fmt, &sd_fmt.format);
+		v4l2_fill_ext_pix_format(fmt, &sd_fmt.format);
 		pix_map = vimc_pix_map_by_code(sd_fmt.format.code);
 		fmt->pixelformat = pix_map->pixelformat;
 	} else if (is_media_entity_v4l2_video_device(pad->entity)) {
@@ -274,7 +274,7 @@ static int vimc_get_pix_format(struct media_pad *pad,
 
 int vimc_vdev_link_validate(struct media_link *link)
 {
-	struct v4l2_pix_format source_fmt, sink_fmt;
+	struct v4l2_ext_pix_format source_fmt, sink_fmt;
 	int ret;
 
 	ret = vimc_get_pix_format(link->source, &source_fmt);
diff --git a/drivers/media/test-drivers/vimc/vimc-common.h b/drivers/media/test-drivers/vimc/vimc-common.h
index ae163dec2459c..f9a915d064e12 100644
--- a/drivers/media/test-drivers/vimc/vimc-common.h
+++ b/drivers/media/test-drivers/vimc/vimc-common.h
@@ -103,7 +103,7 @@ struct vimc_ent_device {
 	void * (*process_frame)(struct vimc_ent_device *ved,
 				const void *frame);
 	void (*vdev_get_format)(struct vimc_ent_device *ved,
-			      struct v4l2_pix_format *fmt);
+				struct v4l2_ext_pix_format *fmt);
 };
 
 /**
-- 
2.26.0


^ permalink raw reply related	[flat|nested] 29+ messages in thread

* Re: [PATCH v4 0/6] media: v4l2: Add extended fmt and buffer ioctls
  2020-07-17 11:54 [PATCH v4 0/6] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
                   ` (5 preceding siblings ...)
  2020-07-17 11:54 ` [PATCH v4 6/6] media: vimc: Implement the ext_fmt and ext_buf hooks Helen Koike
@ 2020-07-20 11:57 ` Tomasz Figa
  2020-07-20 19:47   ` Helen Koike
  2020-07-21 10:24 ` Hans Verkuil
  2020-07-21 11:15 ` Boris Brezillon
  8 siblings, 1 reply; 29+ messages in thread
From: Tomasz Figa @ 2020-07-20 11:57 UTC (permalink / raw)
  To: Helen Koike
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Laurent Pinchart,
	Sakari Ailus, Linux Media Mailing List, Hirokazu Honda,
	Nicolas Dufresne, Brian Starkey, kernel, Boris Brezillon,
	Neil Armstrong, Linux Kernel Mailing List, frkoenig,
	Maxime Jourdan, Stanimir Varbanov

Hi Helen,

On Fri, Jul 17, 2020 at 1:55 PM Helen Koike <helen.koike@collabora.com> wrote:
>
> Hi,
>
> I'm sorry for taking too long to submit v4.
>
> It is not perfect, not all v4l2-compliance tests passes, but I'd like a review,
> specially on the API and potential problems, so I can focus on improving implementation
> and maybe drop the RFC tag for next version.
>
> Follow below what changed in v4 and some items I'd like to discuss:
>
>
> * Ioctl to replace v4l2_pix_format
> ---------------------------------------------------------------------------------
> During last media summit, we agreed to create ioctls that replace the v4l2_pix_format
> struct and leave the other structs in the v4l2_format union alone.
> Thus I refactored the code to receive struct v4l2_ext_pix_format, and I renamed the
> ioctls, so now we have:
>
> int ioctl(int fd, VIDIOC_G_EXT_FMT, struct v4l2_ext_pix_format *argp);
> int ioctl(int fd, VIDIOC_S_EXT_FMT, struct v4l2_ext_pix_format *argp);
> int ioctl(int fd, VIDIOC_TRY_EXT_FMT, struct v4l2_ext_pix_format *argp);
>
> The only valid types are V4L2_BUF_TYPE_VIDEO_CAPTURE and V4L2_BUF_TYPE_VIDEO_OUTPUT,
> all the other types are invalid with this API.
>
>
> * Modifiers
> ---------------------------------------------------------------------------------
> I understand that unifying DRM and V4L2 pixel formats is not possible, but I'd like
> to unify the modifiers [1].
>
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/drm/drm_fourcc.h#n290
>
> Should we use the DRM modifiers directly in the V4L2 API?
> Or should we move this header to a common place and change the prefix? (which requires
> us to sync with DRM community).
> Or should we create a v4l2 header, defining V4L2_ prefixed macros mapping to DRM_
> macros?
>
> For now, patch 1/6 includes drm/drm_fourcc.h and it is using DRM_FORMAT_MOD_*
>
> As discussed before, It would be nice to have documentation describing DRM fourcc
> equivalents (I'm not sure if someone started this already), listing the number of
> planes per format.
>
> We should also document which pixelformats are valid for the EXT_API, since multiplanar
> and tile versions like V4L2_PIX_FMT_NV12MT_16X16 (which seems equivalent to
> DRM_FORMAT_MOD_SAMSUNG_16_16_TILE, and could have a more generic name) should be
> replaced by a modifier.
>
> Regarding flags [2] field in struct v4l2_pix_format_mplane [3]:
> The only defined flag is V4L2_PIX_FMT_FLAG_PREMUL_ALPHA, and it is only used by vsp1 driver.
> Which I believe could be replaced by a modifier, to avoid another field that changes
> pixel formats, so I removed it from the EXT API (we can always add it back later with
> the reserved fields).
>
> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/pixfmt-reserved.html#format-flags
> [3] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/pixfmt-v4l2-mplane.html?highlight=v4l2_pix_format_mplane#c.v4l2_pix_format_mplane
>
> We also discussed to add a new ENUM_FMT_EXT ioctl to return all pixelformats + modifiers
> combinations. I still didn't add it in this version, but I don't think it affects
> what is in this RFC and it can be added later.
>
>
> * Buffers/Plane offset
> ---------------------------------------------------------------------------------
>
> My understanding is that inside a memory buffer we can have multiple planes in random
> offsets.
> I was comparing with the DRM API [4], where it can have the same dmabuf for multiple
> planes in different offsets, and I started to think we could simplify our API, so
> I took the liberty to do some more changes, please review struct v4l2_ext_plane in
> this RFC.
>
> I removed the data_offset, since it is unused (See Laurent's RFC repurposing this
> field [5]). And comparing to the DRM API, it seems to me we only need a single offset
> field.
>
> We could also check about overlapping planes in a memory buffer, but this is complicated
> if we use the same memory buffer with different v4l2_ext_buffer objects. We can also leave
> to the driver to check situations that may cause HW errors.
>
> [4] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/drm/drm_mode.h#n489
> [5] https://patchwork.linuxtv.org/patch/29177/
>
>
> * Multistream Channels
> ---------------------------------------------------------------------------------
> During last media summit, we discussed about adding a channel number to the API to
> support multistreams. i.e, to have multiple queues through a single video node.
>
> Use cases:
>
>     - Blitters: can take multiple streams as input, which would require multiple OUTPUT queues.
>
>     As Nicolas was explaining me:
>     "The blitters comes with a lot of variation between hardware. Most blitters at
>     least support 3 frames buffer. 2 inputs and one output. The second input is usually
>     optional, as the output buffer data is not always overwritten (e.g. SRC_OVER
>     blend or 1 input). Some of them have additional solid color or pattern that can
>     be used too. Advanced blitters will have composition feature, and may support more
>     input buffers to reduce the added latency that would be normally done through cascading
>     the operations. Note that each input can have different size and different cropping
>     region. Many blitters can scale and render to a sub-region of the CAPTURE buffer."
>
>     - Multis-calers: can produce multiple streams, which would require multiple CAPTURE queues.
>
>     As Nicolas was explaining me:
>     "This type of HW (or soft IP) is commonly found on HW used to produce internet
>     streams for fragmented and scalable protocols (HLS, DASH).  Basically they are
>     used to transform one stream into multiple sized streams prior from being encoded."
>
> Modeling as channels allows the API to have synchronized Start/Stop between queues,
> and also avoid the complexity of using the Media API in a topology with multiple video
> nodes, which complicates userspace.
>
> This requires adding a new channel id in ioctls for formats (G_FMT/S_FMT/TRY_FMT), and
> also for buffers (QBUF/DBUF).
> We also need a mechanism to enumerate channels and their properties.
> Since we don't have a clear view how this would work, for now I'm leaving reserved bits
> in the structs, so we can add them later.
>
>
> * Timecode
> ---------------------------------------------------------------------------------
> During last media summit, we discussed to return the v4l2_timecode field to the API,
> since Nicolas mentioned that, even if it is not used by any upstreamed driver, it
> is used by out-of-tree drivers.
>
> I've been discussing with Nicolas about this, and we can avoid adding too many metadata
> to the buffer struct by using the Read-Only Request API [6] for retrieving more information
> when required, similar to HDR.
>
> The RO Request API has the ability to read a control using a request that has already
> completed, the control value lives as long as the request object. If it's not read
> (or if there was no request), the data is simply ignored/discard.
>
> Since no upstream driver uses the timecode field, there are no conversions that need
> to be done.
>
> [6] https://patchwork.kernel.org/cover/11635927/
>
>
> * Other changes (and some questions) in this version:
> ---------------------------------------------------------------------------------
> - Added reserved fields to struct
>
> - The only difference between previously proposed VIDIOC_EXT_EXPBUF and VIDIOC_EXPBUF,
> was that with VIDIOC_EXT_EXPBUF we can export multiple planes at once. I think we
> can add this later, so I removed it from this RFC to simplify it.
>
> - v4l2_buffer [7] has a memory field (enum v4l2_memory [8]). We kept this field in
> struct v4l2_ext_buffer, buf I was wondering if this shouldn't be in struct v4l2_ext_plane
> instead.
>
> [7] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/buffer.html?highlight=v4l2_buffer#c.v4l2_buffer
> [8] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/buffer.html?highlight=v4l2_memory#c.v4l2_memory
>
> - In struct v4l2_ext_pix_format, we have:
>
>         struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];
>
> The number of planes can be deducted from plane_fmt[i].sizeimage != 0, so I removed
> the num_planes field. Please let me know if we can't use sizeimage for this.
> In DRM, we know the number of planes from drm_mode_fb_cmd2 by the number of handle
> args passed which are not 0.
> This also avoids num_planes to be bigger then VIDEO_MAX_PLANES.
>
> - Added flags field to struct v4l2_ext_create_buffers
>
>
> * Fixed bugs here and there
> ---------------------------------------------------------------------------------
> I fixed some bugs found with v4l2-compliance (not all of them yet),
> through script v4l-utils/contrib/test/test-media.
>
> I adapted what Boris did for v4l-utils in previous version to this version:
> https://gitlab.collabora.com/koike/v4l-utils/-/tree/ext-api/wip
>
> Boris' questions regarding DMABUF in last version still holds [9].
>
> [9] https://patchwork.linuxtv.org/project/linux-media/cover/20191008091119.7294-1-boris.brezillon@collabora.com/
>
>
> Please, let me know your feedback,
> Helen
>
>
> Boris Brezillon (5):
>   media: v4l2: Extend pixel formats to unify single/multi-planar
>     handling (and more)
>   media: videobuf2: Expose helpers to implement the _ext_fmt and
>     _ext_buf hooks
>   media: mediabus: Add helpers to convert a ext_pix format to/from a
>     mbus_fmt
>   media: vivid: Convert the capture and output drivers to
>     EXT_FMT/EXT_BUF
>   media: vimc: Implement the ext_fmt and ext_buf hooks
>
> Hans Verkuil (1):
>   media: v4l2: Add extended buffer operations
>
>  .../media/common/videobuf2/videobuf2-core.c   |   2 +
>  .../media/common/videobuf2/videobuf2-v4l2.c   | 549 +++++-----
>  .../media/test-drivers/vimc/vimc-capture.c    |  61 +-
>  drivers/media/test-drivers/vimc/vimc-common.c |   6 +-
>  drivers/media/test-drivers/vimc/vimc-common.h |   2 +-
>  drivers/media/test-drivers/vivid/vivid-core.c |  70 +-
>  .../test-drivers/vivid/vivid-touch-cap.c      |  26 +-
>  .../test-drivers/vivid/vivid-touch-cap.h      |   3 +-
>  .../media/test-drivers/vivid/vivid-vid-cap.c  | 169 +---
>  .../media/test-drivers/vivid/vivid-vid-cap.h  |  15 +-
>  .../media/test-drivers/vivid/vivid-vid-out.c  | 193 ++--
>  .../media/test-drivers/vivid/vivid-vid-out.h  |  15 +-
>  drivers/media/v4l2-core/v4l2-dev.c            |  50 +-
>  drivers/media/v4l2-core/v4l2-ioctl.c          | 934 ++++++++++++++++--
>  include/media/v4l2-ioctl.h                    |  60 ++
>  include/media/v4l2-mediabus.h                 |  42 +
>  include/media/videobuf2-core.h                |   6 +-
>  include/media/videobuf2-v4l2.h                |  21 +-
>  include/uapi/linux/videodev2.h                | 144 +++
>  19 files changed, 1650 insertions(+), 718 deletions(-)

I don't see any documentation being added by this series. I think it's
been a long standing issue with this series and makes it difficult to
review the UAPI itself, in separation from the kernel implementation
details, which is especially important for any non-kernel developers
willing to provide feedback.

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v4 0/6] media: v4l2: Add extended fmt and buffer ioctls
  2020-07-20 11:57 ` [PATCH v4 0/6] media: v4l2: Add extended fmt and buffer ioctls Tomasz Figa
@ 2020-07-20 19:47   ` Helen Koike
  0 siblings, 0 replies; 29+ messages in thread
From: Helen Koike @ 2020-07-20 19:47 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Laurent Pinchart,
	Sakari Ailus, Linux Media Mailing List, Hirokazu Honda,
	Nicolas Dufresne, Brian Starkey, kernel, Boris Brezillon,
	Neil Armstrong, Linux Kernel Mailing List, frkoenig,
	Maxime Jourdan, Stanimir Varbanov



On 7/20/20 8:57 AM, Tomasz Figa wrote:
> Hi Helen,
> 
> On Fri, Jul 17, 2020 at 1:55 PM Helen Koike <helen.koike@collabora.com> wrote:
>>
>> Hi,
>>
>> I'm sorry for taking too long to submit v4.
>>
>> It is not perfect, not all v4l2-compliance tests passes, but I'd like a review,
>> specially on the API and potential problems, so I can focus on improving implementation
>> and maybe drop the RFC tag for next version.
>>
>> Follow below what changed in v4 and some items I'd like to discuss:
>>
>>
>> * Ioctl to replace v4l2_pix_format
>> ---------------------------------------------------------------------------------
>> During last media summit, we agreed to create ioctls that replace the v4l2_pix_format
>> struct and leave the other structs in the v4l2_format union alone.
>> Thus I refactored the code to receive struct v4l2_ext_pix_format, and I renamed the
>> ioctls, so now we have:
>>
>> int ioctl(int fd, VIDIOC_G_EXT_FMT, struct v4l2_ext_pix_format *argp);
>> int ioctl(int fd, VIDIOC_S_EXT_FMT, struct v4l2_ext_pix_format *argp);
>> int ioctl(int fd, VIDIOC_TRY_EXT_FMT, struct v4l2_ext_pix_format *argp);
>>
>> The only valid types are V4L2_BUF_TYPE_VIDEO_CAPTURE and V4L2_BUF_TYPE_VIDEO_OUTPUT,
>> all the other types are invalid with this API.
>>
>>
>> * Modifiers
>> ---------------------------------------------------------------------------------
>> I understand that unifying DRM and V4L2 pixel formats is not possible, but I'd like
>> to unify the modifiers [1].
>>
>> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/drm/drm_fourcc.h#n290
>>
>> Should we use the DRM modifiers directly in the V4L2 API?
>> Or should we move this header to a common place and change the prefix? (which requires
>> us to sync with DRM community).
>> Or should we create a v4l2 header, defining V4L2_ prefixed macros mapping to DRM_
>> macros?
>>
>> For now, patch 1/6 includes drm/drm_fourcc.h and it is using DRM_FORMAT_MOD_*
>>
>> As discussed before, It would be nice to have documentation describing DRM fourcc
>> equivalents (I'm not sure if someone started this already), listing the number of
>> planes per format.
>>
>> We should also document which pixelformats are valid for the EXT_API, since multiplanar
>> and tile versions like V4L2_PIX_FMT_NV12MT_16X16 (which seems equivalent to
>> DRM_FORMAT_MOD_SAMSUNG_16_16_TILE, and could have a more generic name) should be
>> replaced by a modifier.
>>
>> Regarding flags [2] field in struct v4l2_pix_format_mplane [3]:
>> The only defined flag is V4L2_PIX_FMT_FLAG_PREMUL_ALPHA, and it is only used by vsp1 driver.
>> Which I believe could be replaced by a modifier, to avoid another field that changes
>> pixel formats, so I removed it from the EXT API (we can always add it back later with
>> the reserved fields).
>>
>> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/pixfmt-reserved.html#format-flags
>> [3] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/pixfmt-v4l2-mplane.html?highlight=v4l2_pix_format_mplane#c.v4l2_pix_format_mplane
>>
>> We also discussed to add a new ENUM_FMT_EXT ioctl to return all pixelformats + modifiers
>> combinations. I still didn't add it in this version, but I don't think it affects
>> what is in this RFC and it can be added later.
>>
>>
>> * Buffers/Plane offset
>> ---------------------------------------------------------------------------------
>>
>> My understanding is that inside a memory buffer we can have multiple planes in random
>> offsets.
>> I was comparing with the DRM API [4], where it can have the same dmabuf for multiple
>> planes in different offsets, and I started to think we could simplify our API, so
>> I took the liberty to do some more changes, please review struct v4l2_ext_plane in
>> this RFC.
>>
>> I removed the data_offset, since it is unused (See Laurent's RFC repurposing this
>> field [5]). And comparing to the DRM API, it seems to me we only need a single offset
>> field.
>>
>> We could also check about overlapping planes in a memory buffer, but this is complicated
>> if we use the same memory buffer with different v4l2_ext_buffer objects. We can also leave
>> to the driver to check situations that may cause HW errors.
>>
>> [4] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/drm/drm_mode.h#n489
>> [5] https://patchwork.linuxtv.org/patch/29177/
>>
>>
>> * Multistream Channels
>> ---------------------------------------------------------------------------------
>> During last media summit, we discussed about adding a channel number to the API to
>> support multistreams. i.e, to have multiple queues through a single video node.
>>
>> Use cases:
>>
>>     - Blitters: can take multiple streams as input, which would require multiple OUTPUT queues.
>>
>>     As Nicolas was explaining me:
>>     "The blitters comes with a lot of variation between hardware. Most blitters at
>>     least support 3 frames buffer. 2 inputs and one output. The second input is usually
>>     optional, as the output buffer data is not always overwritten (e.g. SRC_OVER
>>     blend or 1 input). Some of them have additional solid color or pattern that can
>>     be used too. Advanced blitters will have composition feature, and may support more
>>     input buffers to reduce the added latency that would be normally done through cascading
>>     the operations. Note that each input can have different size and different cropping
>>     region. Many blitters can scale and render to a sub-region of the CAPTURE buffer."
>>
>>     - Multis-calers: can produce multiple streams, which would require multiple CAPTURE queues.
>>
>>     As Nicolas was explaining me:
>>     "This type of HW (or soft IP) is commonly found on HW used to produce internet
>>     streams for fragmented and scalable protocols (HLS, DASH).  Basically they are
>>     used to transform one stream into multiple sized streams prior from being encoded."
>>
>> Modeling as channels allows the API to have synchronized Start/Stop between queues,
>> and also avoid the complexity of using the Media API in a topology with multiple video
>> nodes, which complicates userspace.
>>
>> This requires adding a new channel id in ioctls for formats (G_FMT/S_FMT/TRY_FMT), and
>> also for buffers (QBUF/DBUF).
>> We also need a mechanism to enumerate channels and their properties.
>> Since we don't have a clear view how this would work, for now I'm leaving reserved bits
>> in the structs, so we can add them later.
>>
>>
>> * Timecode
>> ---------------------------------------------------------------------------------
>> During last media summit, we discussed to return the v4l2_timecode field to the API,
>> since Nicolas mentioned that, even if it is not used by any upstreamed driver, it
>> is used by out-of-tree drivers.
>>
>> I've been discussing with Nicolas about this, and we can avoid adding too many metadata
>> to the buffer struct by using the Read-Only Request API [6] for retrieving more information
>> when required, similar to HDR.
>>
>> The RO Request API has the ability to read a control using a request that has already
>> completed, the control value lives as long as the request object. If it's not read
>> (or if there was no request), the data is simply ignored/discard.
>>
>> Since no upstream driver uses the timecode field, there are no conversions that need
>> to be done.
>>
>> [6] https://patchwork.kernel.org/cover/11635927/
>>
>>
>> * Other changes (and some questions) in this version:
>> ---------------------------------------------------------------------------------
>> - Added reserved fields to struct
>>
>> - The only difference between previously proposed VIDIOC_EXT_EXPBUF and VIDIOC_EXPBUF,
>> was that with VIDIOC_EXT_EXPBUF we can export multiple planes at once. I think we
>> can add this later, so I removed it from this RFC to simplify it.
>>
>> - v4l2_buffer [7] has a memory field (enum v4l2_memory [8]). We kept this field in
>> struct v4l2_ext_buffer, buf I was wondering if this shouldn't be in struct v4l2_ext_plane
>> instead.
>>
>> [7] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/buffer.html?highlight=v4l2_buffer#c.v4l2_buffer
>> [8] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/buffer.html?highlight=v4l2_memory#c.v4l2_memory
>>
>> - In struct v4l2_ext_pix_format, we have:
>>
>>         struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];
>>
>> The number of planes can be deducted from plane_fmt[i].sizeimage != 0, so I removed
>> the num_planes field. Please let me know if we can't use sizeimage for this.
>> In DRM, we know the number of planes from drm_mode_fb_cmd2 by the number of handle
>> args passed which are not 0.
>> This also avoids num_planes to be bigger then VIDEO_MAX_PLANES.
>>
>> - Added flags field to struct v4l2_ext_create_buffers
>>
>>
>> * Fixed bugs here and there
>> ---------------------------------------------------------------------------------
>> I fixed some bugs found with v4l2-compliance (not all of them yet),
>> through script v4l-utils/contrib/test/test-media.
>>
>> I adapted what Boris did for v4l-utils in previous version to this version:
>> https://gitlab.collabora.com/koike/v4l-utils/-/tree/ext-api/wip
>>
>> Boris' questions regarding DMABUF in last version still holds [9].
>>
>> [9] https://patchwork.linuxtv.org/project/linux-media/cover/20191008091119.7294-1-boris.brezillon@collabora.com/
>>
>>
>> Please, let me know your feedback,
>> Helen
>>
>>
>> Boris Brezillon (5):
>>   media: v4l2: Extend pixel formats to unify single/multi-planar
>>     handling (and more)
>>   media: videobuf2: Expose helpers to implement the _ext_fmt and
>>     _ext_buf hooks
>>   media: mediabus: Add helpers to convert a ext_pix format to/from a
>>     mbus_fmt
>>   media: vivid: Convert the capture and output drivers to
>>     EXT_FMT/EXT_BUF
>>   media: vimc: Implement the ext_fmt and ext_buf hooks
>>
>> Hans Verkuil (1):
>>   media: v4l2: Add extended buffer operations
>>
>>  .../media/common/videobuf2/videobuf2-core.c   |   2 +
>>  .../media/common/videobuf2/videobuf2-v4l2.c   | 549 +++++-----
>>  .../media/test-drivers/vimc/vimc-capture.c    |  61 +-
>>  drivers/media/test-drivers/vimc/vimc-common.c |   6 +-
>>  drivers/media/test-drivers/vimc/vimc-common.h |   2 +-
>>  drivers/media/test-drivers/vivid/vivid-core.c |  70 +-
>>  .../test-drivers/vivid/vivid-touch-cap.c      |  26 +-
>>  .../test-drivers/vivid/vivid-touch-cap.h      |   3 +-
>>  .../media/test-drivers/vivid/vivid-vid-cap.c  | 169 +---
>>  .../media/test-drivers/vivid/vivid-vid-cap.h  |  15 +-
>>  .../media/test-drivers/vivid/vivid-vid-out.c  | 193 ++--
>>  .../media/test-drivers/vivid/vivid-vid-out.h  |  15 +-
>>  drivers/media/v4l2-core/v4l2-dev.c            |  50 +-
>>  drivers/media/v4l2-core/v4l2-ioctl.c          | 934 ++++++++++++++++--
>>  include/media/v4l2-ioctl.h                    |  60 ++
>>  include/media/v4l2-mediabus.h                 |  42 +
>>  include/media/videobuf2-core.h                |   6 +-
>>  include/media/videobuf2-v4l2.h                |  21 +-
>>  include/uapi/linux/videodev2.h                | 144 +++
>>  19 files changed, 1650 insertions(+), 718 deletions(-)
> 
> I don't see any documentation being added by this series. I think it's
> been a long standing issue with this series and makes it difficult to
> review the UAPI itself, in separation from the kernel implementation
> details, which is especially important for any non-kernel developers
> willing to provide feedback.

Agreed, I'll re-spin a new version with docs.

Regards,
Helen

> 
> Best regards,
> Tomasz
> 

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v4 0/6] media: v4l2: Add extended fmt and buffer ioctls
  2020-07-17 11:54 [PATCH v4 0/6] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
                   ` (6 preceding siblings ...)
  2020-07-20 11:57 ` [PATCH v4 0/6] media: v4l2: Add extended fmt and buffer ioctls Tomasz Figa
@ 2020-07-21 10:24 ` Hans Verkuil
  2020-07-21 14:23   ` Helen Koike
  2020-07-21 11:15 ` Boris Brezillon
  8 siblings, 1 reply; 29+ messages in thread
From: Hans Verkuil @ 2020-07-21 10:24 UTC (permalink / raw)
  To: Helen Koike, mchehab, hans.verkuil, laurent.pinchart,
	sakari.ailus, linux-media
  Cc: tfiga, hiroh, nicolas, Brian.Starkey, kernel, boris.brezillon,
	narmstrong, linux-kernel, frkoenig, mjourdan, stanimir.varbanov

On 17/07/2020 13:54, Helen Koike wrote:
> Hi,
> 
> I'm sorry for taking too long to submit v4.
> 
> It is not perfect, not all v4l2-compliance tests passes, but I'd like a review,
> specially on the API and potential problems, so I can focus on improving implementation
> and maybe drop the RFC tag for next version.
> 
> Follow below what changed in v4 and some items I'd like to discuss:
> 
> 
> * Ioctl to replace v4l2_pix_format
> ---------------------------------------------------------------------------------
> During last media summit, we agreed to create ioctls that replace the v4l2_pix_format
> struct and leave the other structs in the v4l2_format union alone.
> Thus I refactored the code to receive struct v4l2_ext_pix_format, and I renamed the
> ioctls, so now we have:
> 
> int ioctl(int fd, VIDIOC_G_EXT_FMT, struct v4l2_ext_pix_format *argp);
> int ioctl(int fd, VIDIOC_S_EXT_FMT, struct v4l2_ext_pix_format *argp);
> int ioctl(int fd, VIDIOC_TRY_EXT_FMT, struct v4l2_ext_pix_format *argp);
> 
> The only valid types are V4L2_BUF_TYPE_VIDEO_CAPTURE and V4L2_BUF_TYPE_VIDEO_OUTPUT,
> all the other types are invalid with this API.
> 
> 
> * Modifiers
> ---------------------------------------------------------------------------------
> I understand that unifying DRM and V4L2 pixel formats is not possible, but I'd like
> to unify the modifiers [1].
> 
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/drm/drm_fourcc.h#n290
> 
> Should we use the DRM modifiers directly in the V4L2 API?

For now, yes. Most of the modifier work is done in DRM, it is only fairly recent
that the media subsystem starts to have a need for it. So for now just use the drm
header and prefixes.

> Or should we move this header to a common place and change the prefix? (which requires
> us to sync with DRM community).
> Or should we create a v4l2 header, defining V4L2_ prefixed macros mapping to DRM_
> macros?
> 
> For now, patch 1/6 includes drm/drm_fourcc.h and it is using DRM_FORMAT_MOD_*
> 
> As discussed before, It would be nice to have documentation describing DRM fourcc
> equivalents (I'm not sure if someone started this already), listing the number of
> planes per format.
> 
> We should also document which pixelformats are valid for the EXT_API, since multiplanar
> and tile versions like V4L2_PIX_FMT_NV12MT_16X16 (which seems equivalent to
> DRM_FORMAT_MOD_SAMSUNG_16_16_TILE, and could have a more generic name) should be
> replaced by a modifier.
> 
> Regarding flags [2] field in struct v4l2_pix_format_mplane [3]:
> The only defined flag is V4L2_PIX_FMT_FLAG_PREMUL_ALPHA, and it is only used by vsp1 driver.
> Which I believe could be replaced by a modifier, to avoid another field that changes
> pixel formats, so I removed it from the EXT API (we can always add it back later with
> the reserved fields).

The colorspace series that Dafna is working on will add a V4L2_PIX_FMT_FLAG_SET_CSC
flag, so this flags field will be needed.

> 
> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/pixfmt-reserved.html#format-flags
> [3] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/pixfmt-v4l2-mplane.html?highlight=v4l2_pix_format_mplane#c.v4l2_pix_format_mplane
> 
> We also discussed to add a new ENUM_FMT_EXT ioctl to return all pixelformats + modifiers
> combinations. I still didn't add it in this version, but I don't think it affects
> what is in this RFC and it can be added later.
> 
> 
> * Buffers/Plane offset
> ---------------------------------------------------------------------------------
> 
> My understanding is that inside a memory buffer we can have multiple planes in random
> offsets.
> I was comparing with the DRM API [4], where it can have the same dmabuf for multiple
> planes in different offsets, and I started to think we could simplify our API, so
> I took the liberty to do some more changes, please review struct v4l2_ext_plane in
> this RFC.
> 
> I removed the data_offset, since it is unused (See Laurent's RFC repurposing this
> field [5]). And comparing to the DRM API, it seems to me we only need a single offset
> field.
> 
> We could also check about overlapping planes in a memory buffer, but this is complicated
> if we use the same memory buffer with different v4l2_ext_buffer objects. We can also leave
> to the driver to check situations that may cause HW errors.
> 
> [4] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/drm/drm_mode.h#n489
> [5] https://patchwork.linuxtv.org/patch/29177/
> 
> 
> * Multistream Channels
> ---------------------------------------------------------------------------------
> During last media summit, we discussed about adding a channel number to the API to
> support multistreams. i.e, to have multiple queues through a single video node.
> 
> Use cases:
> 
>     - Blitters: can take multiple streams as input, which would require multiple OUTPUT queues.
> 
>     As Nicolas was explaining me:
>     "The blitters comes with a lot of variation between hardware. Most blitters at
>     least support 3 frames buffer. 2 inputs and one output. The second input is usually
>     optional, as the output buffer data is not always overwritten (e.g. SRC_OVER
>     blend or 1 input). Some of them have additional solid color or pattern that can
>     be used too. Advanced blitters will have composition feature, and may support more
>     input buffers to reduce the added latency that would be normally done through cascading
>     the operations. Note that each input can have different size and different cropping
>     region. Many blitters can scale and render to a sub-region of the CAPTURE buffer."
> 
>     - Multis-calers: can produce multiple streams, which would require multiple CAPTURE queues.
> 
>     As Nicolas was explaining me:
>     "This type of HW (or soft IP) is commonly found on HW used to produce internet
>     streams for fragmented and scalable protocols (HLS, DASH).  Basically they are
>     used to transform one stream into multiple sized streams prior from being encoded."
> 
> Modeling as channels allows the API to have synchronized Start/Stop between queues,
> and also avoid the complexity of using the Media API in a topology with multiple video
> nodes, which complicates userspace.
> 
> This requires adding a new channel id in ioctls for formats (G_FMT/S_FMT/TRY_FMT), and
> also for buffers (QBUF/DBUF).
> We also need a mechanism to enumerate channels and their properties.
> Since we don't have a clear view how this would work, for now I'm leaving reserved bits
> in the structs, so we can add them later.
> 
> 
> * Timecode
> ---------------------------------------------------------------------------------
> During last media summit, we discussed to return the v4l2_timecode field to the API,
> since Nicolas mentioned that, even if it is not used by any upstreamed driver, it
> is used by out-of-tree drivers.
> 
> I've been discussing with Nicolas about this, and we can avoid adding too many metadata
> to the buffer struct by using the Read-Only Request API [6] for retrieving more information
> when required, similar to HDR.
> 
> The RO Request API has the ability to read a control using a request that has already
> completed, the control value lives as long as the request object. If it's not read
> (or if there was no request), the data is simply ignored/discard.
> 
> Since no upstream driver uses the timecode field, there are no conversions that need
> to be done.

That's a reasonable solution.

> 
> [6] https://patchwork.kernel.org/cover/11635927/
> 
> 
> * Other changes (and some questions) in this version:
> ---------------------------------------------------------------------------------
> - Added reserved fields to struct
> 
> - The only difference between previously proposed VIDIOC_EXT_EXPBUF and VIDIOC_EXPBUF,
> was that with VIDIOC_EXT_EXPBUF we can export multiple planes at once. I think we
> can add this later, so I removed it from this RFC to simplify it.
> 
> - v4l2_buffer [7] has a memory field (enum v4l2_memory [8]). We kept this field in
> struct v4l2_ext_buffer, buf I was wondering if this shouldn't be in struct v4l2_ext_plane
> instead.

This pops up every so often. The only use-case I can think of is when you return both
video planes and metadata planes where the metadata might be MMAP and the video planes
DMABUF. But it would add quite a bit of complexity, I suspect.

> 
> [7] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/buffer.html?highlight=v4l2_buffer#c.v4l2_buffer
> [8] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/buffer.html?highlight=v4l2_memory#c.v4l2_memory
> 
> - In struct v4l2_ext_pix_format, we have:
> 
>         struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];
> 
> The number of planes can be deducted from plane_fmt[i].sizeimage != 0, so I removed
> the num_planes field. Please let me know if we can't use sizeimage for this.
> In DRM, we know the number of planes from drm_mode_fb_cmd2 by the number of handle
> args passed which are not 0.
> This also avoids num_planes to be bigger then VIDEO_MAX_PLANES.

I have no objection to this. You do probably need to add a note about there not
being holes, e.g. plane_fmt[0].sizeimage is != 0, so is plane_fmt[2].sizeimage,
but plane_fmt[1].sizeimage == 0. That's likely something you don't want.

Regards,

	Hans

> 
> - Added flags field to struct v4l2_ext_create_buffers
> 
> 
> * Fixed bugs here and there
> ---------------------------------------------------------------------------------
> I fixed some bugs found with v4l2-compliance (not all of them yet),
> through script v4l-utils/contrib/test/test-media.
> 
> I adapted what Boris did for v4l-utils in previous version to this version:
> https://gitlab.collabora.com/koike/v4l-utils/-/tree/ext-api/wip
> 
> Boris' questions regarding DMABUF in last version still holds [9].
> 
> [9] https://patchwork.linuxtv.org/project/linux-media/cover/20191008091119.7294-1-boris.brezillon@collabora.com/
> 
> 
> Please, let me know your feedback,
> Helen
> 
> 
> Boris Brezillon (5):
>   media: v4l2: Extend pixel formats to unify single/multi-planar
>     handling (and more)
>   media: videobuf2: Expose helpers to implement the _ext_fmt and
>     _ext_buf hooks
>   media: mediabus: Add helpers to convert a ext_pix format to/from a
>     mbus_fmt
>   media: vivid: Convert the capture and output drivers to
>     EXT_FMT/EXT_BUF
>   media: vimc: Implement the ext_fmt and ext_buf hooks
> 
> Hans Verkuil (1):
>   media: v4l2: Add extended buffer operations
> 
>  .../media/common/videobuf2/videobuf2-core.c   |   2 +
>  .../media/common/videobuf2/videobuf2-v4l2.c   | 549 +++++-----
>  .../media/test-drivers/vimc/vimc-capture.c    |  61 +-
>  drivers/media/test-drivers/vimc/vimc-common.c |   6 +-
>  drivers/media/test-drivers/vimc/vimc-common.h |   2 +-
>  drivers/media/test-drivers/vivid/vivid-core.c |  70 +-
>  .../test-drivers/vivid/vivid-touch-cap.c      |  26 +-
>  .../test-drivers/vivid/vivid-touch-cap.h      |   3 +-
>  .../media/test-drivers/vivid/vivid-vid-cap.c  | 169 +---
>  .../media/test-drivers/vivid/vivid-vid-cap.h  |  15 +-
>  .../media/test-drivers/vivid/vivid-vid-out.c  | 193 ++--
>  .../media/test-drivers/vivid/vivid-vid-out.h  |  15 +-
>  drivers/media/v4l2-core/v4l2-dev.c            |  50 +-
>  drivers/media/v4l2-core/v4l2-ioctl.c          | 934 ++++++++++++++++--
>  include/media/v4l2-ioctl.h                    |  60 ++
>  include/media/v4l2-mediabus.h                 |  42 +
>  include/media/videobuf2-core.h                |   6 +-
>  include/media/videobuf2-v4l2.h                |  21 +-
>  include/uapi/linux/videodev2.h                | 144 +++
>  19 files changed, 1650 insertions(+), 718 deletions(-)
> 


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v4 1/6] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)
  2020-07-17 11:54 ` [PATCH v4 1/6] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more) Helen Koike
@ 2020-07-21 10:37   ` Hans Verkuil
  2020-07-28 15:18     ` Helen Koike
  0 siblings, 1 reply; 29+ messages in thread
From: Hans Verkuil @ 2020-07-21 10:37 UTC (permalink / raw)
  To: Helen Koike, mchehab, hans.verkuil, laurent.pinchart,
	sakari.ailus, linux-media
  Cc: Boris Brezillon, tfiga, hiroh, nicolas, Brian.Starkey, kernel,
	narmstrong, linux-kernel, frkoenig, mjourdan, stanimir.varbanov

On 17/07/2020 13:54, Helen Koike wrote:
> From: Boris Brezillon <boris.brezillon@collabora.com>
> 
> This is part of the multiplanar and singleplanar unification process.
> v4l2_ext_pix_format is supposed to work for both cases.
> 
> We also add the concept of modifiers already employed in DRM to expose
> HW-specific formats (like tiled or compressed formats) and allow
> exchanging this information with the DRM subsystem in a consistent way.
> 
> Note that only V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] are accepted in
> v4l2_ext_format, other types will be rejected if you use the {G,S,TRY}EXT_FMT
> ioctls.
> 
> New hooks have been added to v4l2_ioctl_ops to support those new ioctls
> in drivers, but, in the meantime, the core takes care of converting
> {S,G,TRY}_EXT_FMT requests into {S,G,TRY}_FMT so that old drivers can
> still work if the userspace app/lib uses the new ioctls.
> The conversion is also done the other around to allow userspace
> apps/libs using {S,G,TRY}_FMT to work with drivers implementing the
> _ext_ hooks.
> 
> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> ---
> Changes in v4:
> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
> - Add reserved fields
> - Removed num_planes from struct v4l2_ext_pix_format
> - Removed flag field from struct v4l2_ext_pix_format, since the only
>   defined value is V4L2_PIX_FMT_FLAG_PREMUL_ALPHA only used by vsp1,
>   where we can use modifiers, or add it back later through the reserved
>   bits.
> - In v4l2_ext_format_to_format(), check if modifier is != MOD_LINEAR &&
>   != MOD_INVALID
> - Fix type assignment in v4l_g_fmt_ext_pix()
> - Rebased on top of media/master (post 5.8-rc1)
> 
> Changes in v3:
> - Rebased on top of media/master (post 5.4-rc1)
> 
> Changes in v2:
> - Move the modifier in v4l2_ext_format (was formerly placed in
>   v4l2_ext_plane)
> - Fix a few bugs in the converters and add a strict parameter to
>   allow conversion of uninitialized/mis-initialized objects
> ---
>  drivers/media/v4l2-core/v4l2-dev.c   |  21 +-
>  drivers/media/v4l2-core/v4l2-ioctl.c | 585 +++++++++++++++++++++++----
>  include/media/v4l2-ioctl.h           |  34 ++
>  include/uapi/linux/videodev2.h       |  55 +++
>  4 files changed, 614 insertions(+), 81 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
> index a593ea0598b55..e1829906bc086 100644
> --- a/drivers/media/v4l2-core/v4l2-dev.c
> +++ b/drivers/media/v4l2-core/v4l2-dev.c
> @@ -607,25 +607,37 @@ static void determine_valid_ioctls(struct video_device *vdev)
>  			set_bit(_IOC_NR(VIDIOC_ENUM_FMT), valid_ioctls);
>  		if ((is_rx && (ops->vidioc_g_fmt_vid_cap ||
>  			       ops->vidioc_g_fmt_vid_cap_mplane ||
> +			       ops->vidioc_g_ext_pix_fmt_vid_cap ||
>  			       ops->vidioc_g_fmt_vid_overlay)) ||
>  		    (is_tx && (ops->vidioc_g_fmt_vid_out ||
>  			       ops->vidioc_g_fmt_vid_out_mplane ||
> -			       ops->vidioc_g_fmt_vid_out_overlay)))
> +			       ops->vidioc_g_ext_pix_fmt_vid_out ||
> +			       ops->vidioc_g_fmt_vid_out_overlay))) {
>  			 set_bit(_IOC_NR(VIDIOC_G_FMT), valid_ioctls);
> +			 set_bit(_IOC_NR(VIDIOC_G_EXT_PIX_FMT), valid_ioctls);
> +		}
>  		if ((is_rx && (ops->vidioc_s_fmt_vid_cap ||
>  			       ops->vidioc_s_fmt_vid_cap_mplane ||
> +			       ops->vidioc_s_ext_pix_fmt_vid_cap ||
>  			       ops->vidioc_s_fmt_vid_overlay)) ||
>  		    (is_tx && (ops->vidioc_s_fmt_vid_out ||
>  			       ops->vidioc_s_fmt_vid_out_mplane ||
> -			       ops->vidioc_s_fmt_vid_out_overlay)))
> +			       ops->vidioc_s_ext_pix_fmt_vid_out ||
> +			       ops->vidioc_s_fmt_vid_out_overlay))) {
>  			 set_bit(_IOC_NR(VIDIOC_S_FMT), valid_ioctls);
> +			 set_bit(_IOC_NR(VIDIOC_S_EXT_PIX_FMT), valid_ioctls);
> +		}
>  		if ((is_rx && (ops->vidioc_try_fmt_vid_cap ||
>  			       ops->vidioc_try_fmt_vid_cap_mplane ||
> +			       ops->vidioc_try_ext_pix_fmt_vid_cap ||
>  			       ops->vidioc_try_fmt_vid_overlay)) ||
>  		    (is_tx && (ops->vidioc_try_fmt_vid_out ||
>  			       ops->vidioc_try_fmt_vid_out_mplane ||
> -			       ops->vidioc_try_fmt_vid_out_overlay)))
> +			       ops->vidioc_try_ext_pix_fmt_vid_out ||
> +			       ops->vidioc_try_fmt_vid_out_overlay))) {
>  			 set_bit(_IOC_NR(VIDIOC_TRY_FMT), valid_ioctls);
> +			 set_bit(_IOC_NR(VIDIOC_TRY_EXT_PIX_FMT), valid_ioctls);
> +		}
>  		SET_VALID_IOCTL(ops, VIDIOC_OVERLAY, vidioc_overlay);
>  		SET_VALID_IOCTL(ops, VIDIOC_G_FBUF, vidioc_g_fbuf);
>  		SET_VALID_IOCTL(ops, VIDIOC_S_FBUF, vidioc_s_fbuf);
> @@ -682,8 +694,11 @@ static void determine_valid_ioctls(struct video_device *vdev)
>  		/* touch specific ioctls */
>  		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FMT, vidioc_enum_fmt_vid_cap);
>  		SET_VALID_IOCTL(ops, VIDIOC_G_FMT, vidioc_g_fmt_vid_cap);
> +		SET_VALID_IOCTL(ops, VIDIOC_G_EXT_PIX_FMT, vidioc_g_fmt_vid_cap);
>  		SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_vid_cap);
> +		SET_VALID_IOCTL(ops, VIDIOC_S_EXT_PIX_FMT, vidioc_s_fmt_vid_cap);
>  		SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_vid_cap);
> +		SET_VALID_IOCTL(ops, VIDIOC_TRY_EXT_PIX_FMT, vidioc_try_fmt_vid_cap);
>  		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMESIZES, vidioc_enum_framesizes);
>  		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMEINTERVALS, vidioc_enum_frameintervals);
>  		SET_VALID_IOCTL(ops, VIDIOC_ENUMINPUT, vidioc_enum_input);
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 02bfef0da76da..3b77433f6c32b 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -17,6 +17,8 @@
>  
>  #include <linux/videodev2.h>
>  
> +#include <drm/drm_fourcc.h>
> +
>  #include <media/v4l2-common.h>
>  #include <media/v4l2-ioctl.h>
>  #include <media/v4l2-ctrls.h>
> @@ -378,6 +380,27 @@ static void v4l_print_format(const void *arg, bool write_only)
>  	}
>  }
>  
> +static void v4l_print_ext_pix_format(const void *arg, bool write_only)
> +{
> +	const struct v4l2_ext_pix_format *pix = arg;
> +	unsigned int i;
> +
> +	pr_cont("type=%s, width=%u, height=%u, format=%c%c%c%c, modifier %llx, field=%s, colorspace=%d, ycbcr_enc=%u, quantization=%u, xfer_func=%u\n",
> +		prt_names(pix->type, v4l2_type_names),
> +		pix->width, pix->height,
> +		(pix->pixelformat & 0xff),
> +		(pix->pixelformat >>  8) & 0xff,
> +		(pix->pixelformat >> 16) & 0xff,
> +		(pix->pixelformat >> 24) & 0xff,
> +		pix->modifier, prt_names(pix->field, v4l2_field_names),
> +		pix->colorspace, pix->ycbcr_enc,
> +		pix->quantization, pix->xfer_func);
> +	for (i = 0; i < VIDEO_MAX_PLANES && pix->plane_fmt[i].sizeimage; i++)
> +		pr_debug("plane %u: bytesperline=%u sizeimage=%u\n",
> +			 i, pix->plane_fmt[i].bytesperline,
> +			 pix->plane_fmt[i].sizeimage);
> +}
> +
>  static void v4l_print_framebuffer(const void *arg, bool write_only)
>  {
>  	const struct v4l2_framebuffer *p = arg;
> @@ -958,11 +981,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>  	switch (type) {
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>  		if ((is_vid || is_tch) && is_rx &&
> -		    (ops->vidioc_g_fmt_vid_cap || ops->vidioc_g_fmt_vid_cap_mplane))
> +		    (ops->vidioc_g_fmt_vid_cap ||
> +		     ops->vidioc_g_ext_pix_fmt_vid_cap ||
> +		     ops->vidioc_g_fmt_vid_cap_mplane))
>  			return 0;
>  		break;
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> -		if ((is_vid || is_tch) && is_rx && ops->vidioc_g_fmt_vid_cap_mplane)
> +		if ((is_vid || is_tch) && is_rx &&
> +		    (ops->vidioc_g_fmt_vid_cap_mplane ||
> +		     ops->vidioc_g_ext_pix_fmt_vid_cap))
>  			return 0;
>  		break;
>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
> @@ -971,11 +998,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>  		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>  		if (is_vid && is_tx &&
> -		    (ops->vidioc_g_fmt_vid_out || ops->vidioc_g_fmt_vid_out_mplane))
> +		    (ops->vidioc_g_fmt_vid_out ||
> +		     ops->vidioc_g_ext_pix_fmt_vid_out ||
> +		     ops->vidioc_g_fmt_vid_out_mplane))
>  			return 0;
>  		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> -		if (is_vid && is_tx && ops->vidioc_g_fmt_vid_out_mplane)
> +		if (is_vid && is_tx &&
> +		    (ops->vidioc_g_ext_pix_fmt_vid_out ||
> +		     ops->vidioc_g_fmt_vid_out_mplane))
>  			return 0;
>  		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
> @@ -1055,6 +1086,134 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>  	       sizeof(fmt->fmt.pix) - offset);
>  }
>  
> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
> +				  struct v4l2_format *f, bool mplane_cap,
> +				  bool strict)
> +{
> +	const struct v4l2_plane_ext_pix_format *pe;
> +	struct v4l2_plane_pix_format *p;
> +	unsigned int i;
> +
> +	memset(f, 0, sizeof(*f));
> +
> +	/*
> +	 * Make sure no modifier is required before doing the
> +	 * conversion.
> +	 */
> +	if (e->modifier && strict &&
> +	    e->modifier != DRM_FORMAT_MOD_LINEAR &&
> +	    e->modifier != DRM_FORMAT_MOD_INVALID)
> +		return -EINVAL;
> +
> +	if (!e->plane_fmt[0].sizeimage && strict)
> +		return -EINVAL;
> +
> +	if (e->plane_fmt[1].sizeimage && !mplane_cap && strict)
> +		return 0;
> +
> +	if (!mplane_cap) {
> +		f->fmt.pix.width = e->width;
> +		f->fmt.pix.height = e->height;
> +		f->fmt.pix.pixelformat = e->pixelformat;
> +		f->fmt.pix.field = e->field;
> +		f->fmt.pix.colorspace = e->colorspace;
> +		f->fmt.pix.ycbcr_enc = e->ycbcr_enc;
> +		f->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +		f->fmt.pix.quantization = e->quantization;
> +		pe = &e->plane_fmt[0];
> +		f->fmt.pix.bytesperline = pe->bytesperline;
> +		f->fmt.pix.sizeimage = pe->sizeimage;
> +		f->type = e->type;
> +		return 0;
> +	}
> +
> +	f->fmt.pix_mp.width = e->width;
> +	f->fmt.pix_mp.height = e->height;
> +	f->fmt.pix_mp.pixelformat = e->pixelformat;
> +	f->fmt.pix_mp.field = e->field;
> +	f->fmt.pix_mp.colorspace = e->colorspace;
> +	f->fmt.pix_mp.ycbcr_enc = e->ycbcr_enc;
> +	f->fmt.pix_mp.quantization = e->quantization;
> +	if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +		f->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
> +	else
> +		f->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
> +
> +	for (i = 0; i < VIDEO_MAX_PLANES; i++) {
> +		pe = &e->plane_fmt[i];
> +		p = &f->fmt.pix_mp.plane_fmt[i];
> +		p->bytesperline = pe->bytesperline;
> +		p->sizeimage = pe->sizeimage;
> +	}
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_ext_pix_format_to_format);
> +
> +int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
> +				  struct v4l2_ext_pix_format *e, bool strict)
> +{
> +	const struct v4l2_plane_pix_format *p;
> +	struct v4l2_plane_ext_pix_format *pe;
> +	unsigned int i;
> +
> +	memset(e, 0, sizeof(*e));
> +
> +	switch (f->type) {
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +		e->width = f->fmt.pix.width;
> +		e->height = f->fmt.pix.height;
> +		e->pixelformat = f->fmt.pix.pixelformat;
> +		e->field = f->fmt.pix.field;
> +		e->colorspace = f->fmt.pix.colorspace;
> +		if (f->fmt.pix.flags)
> +			pr_warn("Ignoring pixelformat flags 0x%x\n",
> +				f->fmt.pix.flags);
> +		e->ycbcr_enc = f->fmt.pix.ycbcr_enc;
> +		e->quantization = f->fmt.pix.quantization;
> +		e->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
> +		e->plane_fmt[0].sizeimage = f->fmt.pix.sizeimage;
> +		e->type = f->type;
> +		break;
> +
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> +		if ((f->fmt.pix_mp.num_planes > VIDEO_MAX_PLANES ||
> +		     !f->fmt.pix_mp.num_planes) && strict)
> +			return -EINVAL;
> +
> +		e->width = f->fmt.pix_mp.width;
> +		e->height = f->fmt.pix_mp.height;
> +		e->pixelformat = f->fmt.pix_mp.pixelformat;
> +		e->field = f->fmt.pix_mp.field;
> +		e->colorspace = f->fmt.pix_mp.colorspace;
> +		if (f->fmt.pix.flags)
> +			pr_warn("Ignoring pixelformat flags 0x%x\n",
> +				f->fmt.pix.flags);
> +		e->ycbcr_enc = f->fmt.pix_mp.ycbcr_enc;
> +		e->quantization = f->fmt.pix_mp.quantization;
> +		if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
> +			e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +		else
> +			e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
> +
> +		for (i = 0; i < VIDEO_MAX_PLANES; i++) {
> +			pe = &e->plane_fmt[i];
> +			p = &f->fmt.pix_mp.plane_fmt[i];
> +			pe->bytesperline = p->bytesperline;
> +			pe->sizeimage = p->sizeimage;
> +		}
> +		break;
> +
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_format_to_ext_pix_format);
> +
>  static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
>  				struct file *file, void *fh, void *arg)
>  {
> @@ -1558,6 +1717,38 @@ static void v4l_pix_format_touch(struct v4l2_pix_format *p)
>  	p->xfer_func = 0;
>  }
>  
> +static int v4l_g_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
> +			     struct file *file, void *fh,
> +			     struct v4l2_format *f)
> +{
> +	struct v4l2_ext_pix_format ef;
> +	int ret;
> +
> +	switch (f->type) {
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> +		ef.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +		ret = ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, &ef);
> +		break;
> +
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> +		ef.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
> +		ret = ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, &ef);
> +		break;
> +
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	if (ret)
> +		return ret;
> +
> +	return v4l2_ext_pix_format_to_format(&ef, f,
> +					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
> +					     true);
> +}
> +
>  static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>  				struct file *file, void *fh, void *arg)
>  {
> @@ -1594,17 +1785,27 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>  
>  	switch (p->type) {
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> -		if (unlikely(!ops->vidioc_g_fmt_vid_cap))
> -			break;
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -		ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
> -		/* just in case the driver zeroed it again */
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -		if (vfd->vfl_type == VFL_TYPE_TOUCH)
> -			v4l_pix_format_touch(&p->fmt.pix);
> -		return ret;
> +		if (ops->vidioc_g_fmt_vid_cap) {
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +			ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
> +			/* just in case the driver zeroed it again */
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
> +				v4l_pix_format_touch(&p->fmt.pix);
> +			return ret;
> +		} else if (ops->vidioc_g_ext_pix_fmt_vid_cap) {
> +			ret = v4l_g_fmt_ext_pix(ops, file, fh, p);
> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
> +				v4l_pix_format_touch(&p->fmt.pix);
> +			return ret;
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> -		return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
> +		if (ops->vidioc_g_fmt_vid_cap_mplane)
> +			return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
> +		else if (ops->vidioc_g_ext_pix_fmt_vid_cap)
> +			return v4l_g_fmt_ext_pix(ops, file, fh, p);
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>  		return ops->vidioc_g_fmt_vid_overlay(file, fh, arg);
>  	case V4L2_BUF_TYPE_VBI_CAPTURE:
> @@ -1612,15 +1813,22 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>  	case V4L2_BUF_TYPE_SLICED_VBI_CAPTURE:
>  		return ops->vidioc_g_fmt_sliced_vbi_cap(file, fh, arg);
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> -		if (unlikely(!ops->vidioc_g_fmt_vid_out))
> -			break;
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -		ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
> -		/* just in case the driver zeroed it again */
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -		return ret;
> +		if (ops->vidioc_g_fmt_vid_out) {
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +			ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
> +			/* just in case the driver zeroed it again */
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +			return ret;
> +		} else if (ops->vidioc_g_ext_pix_fmt_vid_out) {
> +			return v4l_g_fmt_ext_pix(ops, file, fh, p);
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> -		return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
> +		if (ops->vidioc_g_fmt_vid_out_mplane)
> +			return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
> +		else if (ops->vidioc_g_ext_pix_fmt_vid_out)
> +			return v4l_g_fmt_ext_pix(ops, file, fh, p);
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>  		return ops->vidioc_g_fmt_vid_out_overlay(file, fh, arg);
>  	case V4L2_BUF_TYPE_VBI_OUTPUT:
> @@ -1639,6 +1847,76 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>  	return -EINVAL;
>  }
>  
> +static int v4l_g_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
> +			     struct file *file, void *fh, void *arg)
> +{
> +	struct v4l2_ext_pix_format *ef = arg;
> +	struct v4l2_format f = {
> +		.type = ef->type,
> +	};
> +	int ret;
> +
> +	ret = check_fmt(file, ef->type);
> +	if (ret)
> +		return ret;
> +
> +	memset(ef, 0, sizeof(*ef));
> +	ef->type = f.type;
> +
> +	switch (f.type) {
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +		if (ops->vidioc_g_ext_pix_fmt_vid_cap)
> +			return ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, ef);
> +		break;
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +		if (ops->vidioc_g_ext_pix_fmt_vid_out)
> +			return ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, ef);
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	ret = v4l_g_fmt(ops, file, fh, &f);
> +	if (ret)
> +		return ret;
> +
> +	return v4l2_format_to_ext_pix_format(&f, ef, true);
> +}
> +
> +static int v4l_s_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
> +			     struct file *file, void *fh,
> +			     struct v4l2_format *f)
> +{
> +	struct v4l2_ext_pix_format ef;
> +	int ret;
> +
> +	ret = v4l2_format_to_ext_pix_format(f, &ef, false);
> +	if (ret)
> +		return ret;
> +
> +	switch (f->type) {
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> +		ret = ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, &ef);
> +		break;
> +
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> +		ret = ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, &ef);
> +		break;
> +
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	if (ret)
> +		return ret;
> +
> +	return v4l2_ext_pix_format_to_format(&ef, f,
> +					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
> +					     true);
> +}
> +
>  static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>  				struct file *file, void *fh, void *arg)
>  {
> @@ -1657,23 +1935,31 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>  
>  	switch (p->type) {
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> -		if (unlikely(!ops->vidioc_s_fmt_vid_cap))
> +		if (ops->vidioc_s_fmt_vid_cap) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix);
> +			ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
> +			/* just in case the driver zeroed it again */
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
> +			ret = v4l_s_fmt_ext_pix(ops, file, fh, arg);
> +		} else {
>  			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix);
> -		ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
> -		/* just in case the driver zeroed it again */
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +		}
> +
>  		if (vfd->vfl_type == VFL_TYPE_TOUCH)
>  			v4l_pix_format_touch(&p->fmt.pix);
>  		return ret;
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> -		if (unlikely(!ops->vidioc_s_fmt_vid_cap_mplane))
> -			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> -					  bytesperline);
> -		return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
> +		if (ops->vidioc_s_fmt_vid_cap_mplane) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> +						  bytesperline);
> +			return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
> +			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>  		if (unlikely(!ops->vidioc_s_fmt_vid_overlay))
>  			break;
> @@ -1690,21 +1976,27 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>  		CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
>  		return ops->vidioc_s_fmt_sliced_vbi_cap(file, fh, arg);
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> -		if (unlikely(!ops->vidioc_s_fmt_vid_out))
> -			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix);
> -		ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
> -		/* just in case the driver zeroed it again */
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -		return ret;
> +		if (ops->vidioc_s_fmt_vid_out) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix);
> +			ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
> +			/* just in case the driver zeroed it again */
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +			return ret;
> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
> +			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> -		if (unlikely(!ops->vidioc_s_fmt_vid_out_mplane))
> -			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> -					  bytesperline);
> -		return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
> +		if (ops->vidioc_s_fmt_vid_out_mplane) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> +						  bytesperline);
> +			return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
> +			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>  		if (unlikely(!ops->vidioc_s_fmt_vid_out_overlay))
>  			break;
> @@ -1744,6 +2036,82 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>  	return -EINVAL;
>  }
>  
> +static int v4l_s_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
> +			     struct file *file, void *fh, void *arg)
> +{
> +	struct video_device *vfd = video_devdata(file);
> +	struct v4l2_ext_pix_format *ef = arg;
> +	struct v4l2_format f;
> +	int ret;
> +
> +	ret = check_fmt(file, ef->type);
> +	if (ret)
> +		return ret;
> +
> +	switch (ef->type) {
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +		if (ops->vidioc_s_ext_pix_fmt_vid_cap)
> +			return ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, ef);
> +		break;
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +		if (ops->vidioc_s_ext_pix_fmt_vid_out)
> +			return ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, ef);
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	ret = v4l2_ext_pix_format_to_format(ef, &f,
> +					    vfd->device_caps &
> +					    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> +					     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
> +					     V4L2_CAP_VIDEO_M2M_MPLANE),
> +					    false);
> +	if (ret)
> +		return ret;
> +
> +	ret = v4l_s_fmt(ops, file, fh, &f);
> +	if (ret)
> +		return ret;
> +
> +	return v4l2_format_to_ext_pix_format(&f, ef, true);
> +}
> +
> +static int v4l_try_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
> +			       struct file *file, void *fh,
> +			       struct v4l2_format *f)
> +{
> +	struct v4l2_ext_pix_format ef;
> +	int ret;
> +
> +	ret = v4l2_format_to_ext_pix_format(f, &ef, false);
> +	if (ret)
> +		return ret;
> +
> +	switch (f->type) {
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> +		ret = ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh, &ef);
> +		break;
> +
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> +		ret = ops->vidioc_try_ext_pix_fmt_vid_out(file, fh, &ef);
> +		break;
> +
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	if (ret)
> +		return ret;
> +
> +	return v4l2_ext_pix_format_to_format(&ef, f,
> +					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
> +					     true);
> +}
> +
> +
>  static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>  				struct file *file, void *fh, void *arg)
>  {
> @@ -1759,23 +2127,32 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>  
>  	switch (p->type) {
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> -		if (unlikely(!ops->vidioc_try_fmt_vid_cap))
> -			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix);
> -		ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
> -		/* just in case the driver zeroed it again */
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -		if (vfd->vfl_type == VFL_TYPE_TOUCH)
> -			v4l_pix_format_touch(&p->fmt.pix);
> -		return ret;
> +		if (ops->vidioc_try_fmt_vid_cap) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix);
> +			ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
> +			/* just in case the driver zeroed it again */
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
> +				v4l_pix_format_touch(&p->fmt.pix);
> +			return ret;
> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
> +			ret = v4l_try_fmt_ext_pix(ops, file, fh, p);
> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
> +				v4l_pix_format_touch(&p->fmt.pix);
> +			return ret;
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> -		if (unlikely(!ops->vidioc_try_fmt_vid_cap_mplane))
> -			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> -					  bytesperline);
> -		return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
> +		if (ops->vidioc_try_fmt_vid_cap_mplane) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> +						  bytesperline);
> +			return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
> +			return v4l_try_fmt_ext_pix(ops, file, fh, p);
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>  		if (unlikely(!ops->vidioc_try_fmt_vid_overlay))
>  			break;
> @@ -1792,21 +2169,27 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>  		CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
>  		return ops->vidioc_try_fmt_sliced_vbi_cap(file, fh, arg);
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> -		if (unlikely(!ops->vidioc_try_fmt_vid_out))
> -			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix);
> -		ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
> -		/* just in case the driver zeroed it again */
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -		return ret;
> +		if (ops->vidioc_try_fmt_vid_out) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix);
> +			ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
> +			/* just in case the driver zeroed it again */
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +			return ret;
> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
> +			return v4l_try_fmt_ext_pix(ops, file, fh, p);
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> -		if (unlikely(!ops->vidioc_try_fmt_vid_out_mplane))
> -			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> -					  bytesperline);
> -		return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
> +		if (ops->vidioc_try_fmt_vid_out_mplane) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> +						  bytesperline);
> +			return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
> +			return v4l_try_fmt_ext_pix(ops, file, fh, p);
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>  		if (unlikely(!ops->vidioc_try_fmt_vid_out_overlay))
>  			break;
> @@ -1846,6 +2229,49 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>  	return -EINVAL;
>  }
>  
> +static int v4l_try_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
> +			       struct file *file, void *fh, void *arg)
> +{
> +	struct video_device *vfd = video_devdata(file);
> +	struct v4l2_ext_pix_format *ef = arg;
> +	struct v4l2_format f;
> +	int ret;
> +
> +	ret = check_fmt(file, ef->type);
> +	if (ret)
> +		return ret;
> +
> +	switch (ef->type) {
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +		if (ops->vidioc_try_ext_pix_fmt_vid_cap)
> +			return ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh,
> +								   ef);
> +		break;
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +		if (ops->vidioc_try_ext_pix_fmt_vid_out)
> +			return ops->vidioc_try_ext_pix_fmt_vid_out(file, fh,
> +								   ef);
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	ret = v4l2_ext_pix_format_to_format(ef, &f,
> +					    vfd->device_caps &
> +					    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> +					     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
> +					     V4L2_CAP_VIDEO_M2M_MPLANE),
> +					    false);
> +	if (ret)
> +		return ret;
> +
> +	ret = v4l_try_fmt(ops, file, fh, &f);
> +	if (ret)
> +		return ret;
> +
> +	return v4l2_format_to_ext_pix_format(&f, ef, true);
> +}
> +
>  static int v4l_streamon(const struct v4l2_ioctl_ops *ops,
>  				struct file *file, void *fh, void *arg)
>  {
> @@ -2765,7 +3191,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>  	IOCTL_INFO(VIDIOC_QUERYCAP, v4l_querycap, v4l_print_querycap, 0),
>  	IOCTL_INFO(VIDIOC_ENUM_FMT, v4l_enum_fmt, v4l_print_fmtdesc, 0),
>  	IOCTL_INFO(VIDIOC_G_FMT, v4l_g_fmt, v4l_print_format, 0),
> +	IOCTL_INFO(VIDIOC_G_EXT_PIX_FMT, v4l_g_ext_pix_fmt, v4l_print_ext_pix_format, 0),
>  	IOCTL_INFO(VIDIOC_S_FMT, v4l_s_fmt, v4l_print_format, INFO_FL_PRIO),
> +	IOCTL_INFO(VIDIOC_S_EXT_PIX_FMT, v4l_s_ext_pix_fmt, v4l_print_ext_pix_format, INFO_FL_PRIO),
>  	IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>  	IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
>  	IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
> @@ -2812,6 +3240,7 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>  	IOCTL_INFO(VIDIOC_S_JPEGCOMP, v4l_stub_s_jpegcomp, v4l_print_jpegcompression, INFO_FL_PRIO),
>  	IOCTL_INFO(VIDIOC_QUERYSTD, v4l_querystd, v4l_print_std, 0),
>  	IOCTL_INFO(VIDIOC_TRY_FMT, v4l_try_fmt, v4l_print_format, 0),
> +	IOCTL_INFO(VIDIOC_TRY_EXT_PIX_FMT, v4l_try_ext_pix_fmt, v4l_print_ext_pix_format, 0),
>  	IOCTL_INFO(VIDIOC_ENUMAUDIO, v4l_stub_enumaudio, v4l_print_audio, INFO_FL_CLEAR(v4l2_audio, index)),
>  	IOCTL_INFO(VIDIOC_ENUMAUDOUT, v4l_stub_enumaudout, v4l_print_audioout, INFO_FL_CLEAR(v4l2_audioout, index)),
>  	IOCTL_INFO(VIDIOC_G_PRIORITY, v4l_g_priority, v4l_print_u32, 0),
> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
> index 86878fba332b0..525ce86725260 100644
> --- a/include/media/v4l2-ioctl.h
> +++ b/include/media/v4l2-ioctl.h
> @@ -48,11 +48,17 @@ struct v4l2_fh;
>   * @vidioc_g_fmt_vid_cap: pointer to the function that implements
>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video capture
>   *	in single plane mode
> + * @vidioc_g_ext_pix_fmt_vid_cap: pointer to the function that implements
> + *	:ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
> + *	capture
>   * @vidioc_g_fmt_vid_overlay: pointer to the function that implements
>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>   * @vidioc_g_fmt_vid_out: pointer to the function that implements
>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video out
>   *	in single plane mode
> + * @vidioc_g_ext_pix_fmt_vid_out: pointer to the function that implements
> + *	:ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
> + *	out
>   * @vidioc_g_fmt_vid_out_overlay: pointer to the function that implements
>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
>   * @vidioc_g_fmt_vbi_cap: pointer to the function that implements
> @@ -82,11 +88,16 @@ struct v4l2_fh;
>   * @vidioc_s_fmt_vid_cap: pointer to the function that implements
>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video capture
>   *	in single plane mode
> + * @vidioc_s_ext_pix_fmt_vid_cap: pointer to the function that implements
> + *	:ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_s_ext_pix_fmt>` ioctl logic for video
> + *	capture
>   * @vidioc_s_fmt_vid_overlay: pointer to the function that implements
>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>   * @vidioc_s_fmt_vid_out: pointer to the function that implements
>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video out
>   *	in single plane mode
> + * @vidioc_s_ext_pix_fmt_vid_out: pointer to the function that implements
> + *	:ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
>   * @vidioc_s_fmt_vid_out_overlay: pointer to the function that implements
>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
>   * @vidioc_s_fmt_vbi_cap: pointer to the function that implements
> @@ -116,11 +127,16 @@ struct v4l2_fh;
>   * @vidioc_try_fmt_vid_cap: pointer to the function that implements
>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video capture
>   *	in single plane mode
> + * @vidioc_try_ext_pix_fmt_vid_cap: pointer to the function that implements
> + *	:ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_try_ext_pix_fmt>` ioctl logic for
> +	video capture
>   * @vidioc_try_fmt_vid_overlay: pointer to the function that implements
>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>   * @vidioc_try_fmt_vid_out: pointer to the function that implements
>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video out
>   *	in single plane mode
> + * @vidioc_try_ext_pix_fmt_vid_out: pointer to the function that implements
> + *	:ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
>   * @vidioc_try_fmt_vid_out_overlay: pointer to the function that implements
>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>   *	output
> @@ -319,10 +335,14 @@ struct v4l2_ioctl_ops {
>  	/* VIDIOC_G_FMT handlers */
>  	int (*vidioc_g_fmt_vid_cap)(struct file *file, void *fh,
>  				    struct v4l2_format *f);
> +	int (*vidioc_g_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
> +					    struct v4l2_ext_pix_format *f);
>  	int (*vidioc_g_fmt_vid_overlay)(struct file *file, void *fh,
>  					struct v4l2_format *f);
>  	int (*vidioc_g_fmt_vid_out)(struct file *file, void *fh,
>  				    struct v4l2_format *f);
> +	int (*vidioc_g_ext_pix_fmt_vid_out)(struct file *file, void *fh,
> +					    struct v4l2_ext_pix_format *f);
>  	int (*vidioc_g_fmt_vid_out_overlay)(struct file *file, void *fh,
>  					    struct v4l2_format *f);
>  	int (*vidioc_g_fmt_vbi_cap)(struct file *file, void *fh,
> @@ -349,10 +369,14 @@ struct v4l2_ioctl_ops {
>  	/* VIDIOC_S_FMT handlers */
>  	int (*vidioc_s_fmt_vid_cap)(struct file *file, void *fh,
>  				    struct v4l2_format *f);
> +	int (*vidioc_s_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
> +					    struct v4l2_ext_pix_format *f);
>  	int (*vidioc_s_fmt_vid_overlay)(struct file *file, void *fh,
>  					struct v4l2_format *f);
>  	int (*vidioc_s_fmt_vid_out)(struct file *file, void *fh,
>  				    struct v4l2_format *f);
> +	int (*vidioc_s_ext_pix_fmt_vid_out)(struct file *file, void *fh,
> +					    struct v4l2_ext_pix_format *f);
>  	int (*vidioc_s_fmt_vid_out_overlay)(struct file *file, void *fh,
>  					    struct v4l2_format *f);
>  	int (*vidioc_s_fmt_vbi_cap)(struct file *file, void *fh,
> @@ -379,10 +403,14 @@ struct v4l2_ioctl_ops {
>  	/* VIDIOC_TRY_FMT handlers */
>  	int (*vidioc_try_fmt_vid_cap)(struct file *file, void *fh,
>  				      struct v4l2_format *f);
> +	int (*vidioc_try_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
> +					      struct v4l2_ext_pix_format *f);
>  	int (*vidioc_try_fmt_vid_overlay)(struct file *file, void *fh,
>  					  struct v4l2_format *f);
>  	int (*vidioc_try_fmt_vid_out)(struct file *file, void *fh,
>  				      struct v4l2_format *f);
> +	int (*vidioc_try_ext_pix_fmt_vid_out)(struct file *file, void *fh,
> +					      struct v4l2_ext_pix_format *f);
>  	int (*vidioc_try_fmt_vid_out_overlay)(struct file *file, void *fh,
>  					     struct v4l2_format *f);
>  	int (*vidioc_try_fmt_vbi_cap)(struct file *file, void *fh,
> @@ -724,6 +752,12 @@ long int video_usercopy(struct file *file, unsigned int cmd,
>  long int video_ioctl2(struct file *file,
>  		      unsigned int cmd, unsigned long int arg);
>  
> +int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
> +				  struct v4l2_ext_pix_format *e, bool strict);
> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
> +				  struct v4l2_format *f,
> +				  bool mplane_cap, bool strict);
> +
>  /*
>   * The user space interpretation of the 'v4l2_event' differs
>   * based on the 'time_t' definition on 32-bit architectures, so
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index 303805438814f..fc04c81ce7713 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -2252,6 +2252,57 @@ struct v4l2_pix_format_mplane {
>  	__u8				reserved[7];
>  } __attribute__ ((packed));
>  
> +/**
> + * struct v4l2_plane_ext_pix_format - additional, per-plane format definition
> + * @sizeimage:		maximum size in bytes required for data, for which
> + *			this plane will be used
> + * @bytesperline:	distance in bytes between the leftmost pixels in two
> + *			adjacent lines
> + * @reserved:		extra space reserved for future fields, must be set to 0
> + */
> +struct v4l2_plane_ext_pix_format {
> +	__u32 sizeimage;
> +	__u32 bytesperline;
> +	__u32 reserved;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct v4l2_ext_pix_format - extended single/multiplanar format definition
> + * @type:		type of the data stream; V4L2_BUF_TYPE_VIDEO_CAPTURE or
> + *			V4L2_BUF_TYPE_VIDEO_OUTPUT
> + * @width:		image width in pixels
> + * @height:		image height in pixels
> + * @field:		enum v4l2_field; field order (for interlaced video)
> + * @pixelformat:	little endian four character code (fourcc)
> + * @modifier:		modifier applied to the format (used for tiled formats
> + *			and other kind of HW-specific formats, like compressed
> + *			formats)
> + * @colorspace:		enum v4l2_colorspace; supplemental to pixelformat
> + * @plane_fmt:		per-plane information
> + * @ycbcr_enc:		enum v4l2_ycbcr_encoding, Y'CbCr encoding
> + * @hsv_enc:		enum v4l2_hsv_encoding, HSV encoding
> + * @quantization:	enum v4l2_quantization, colorspace quantization
> + * @xfer_func:		enum v4l2_xfer_func, colorspace transfer function
> + * @reserved:		extra space reserved for future fields, must be set to 0
> + */
> +struct v4l2_ext_pix_format {
> +	__u32 type;
> +	__u32 width;
> +	__u32 height;
> +	__u32 field;
> +	__u32 pixelformat;
> +	__u64 modifier;
> +	__u32 colorspace;

This struct has holes and is not the same for 32 and 64 bit architectures.

Moving modifier to before pixelformat will help a lot.

> +	struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];
> +	union {
> +		__u8 ycbcr_enc;
> +		__u8 hsv_enc;
> +	};
> +	__u8 quantization;
> +	__u8 xfer_func;

I'd change u8 to u32 for these fields for easier alignment.

Regards,

	Hans

> +	__u32 reserved[4];
> +} __attribute__ ((packed));
> +
>  /**
>   * struct v4l2_sdr_format - SDR format definition
>   * @pixelformat:	little endian four character code (fourcc)
> @@ -2569,6 +2620,10 @@ struct v4l2_create_buffers {
>  
>  #define VIDIOC_QUERY_EXT_CTRL	_IOWR('V', 103, struct v4l2_query_ext_ctrl)
>  
> +#define VIDIOC_G_EXT_PIX_FMT	_IOWR('V', 104, struct v4l2_ext_pix_format)
> +#define VIDIOC_S_EXT_PIX_FMT	_IOWR('V', 105, struct v4l2_ext_pix_format)
> +#define VIDIOC_TRY_EXT_PIX_FMT	_IOWR('V', 106, struct v4l2_ext_pix_format)
> +
>  /* Reminder: when adding new ioctls please add support for them to
>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
>  
> 


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v4 2/6] media: v4l2: Add extended buffer operations
  2020-07-17 11:54 ` [PATCH v4 2/6] media: v4l2: Add extended buffer operations Helen Koike
@ 2020-07-21 10:48   ` Hans Verkuil
  2020-07-21 14:36     ` Helen Koike
  2020-07-21 11:26   ` Stanimir Varbanov
  1 sibling, 1 reply; 29+ messages in thread
From: Hans Verkuil @ 2020-07-21 10:48 UTC (permalink / raw)
  To: Helen Koike, mchehab, hans.verkuil, laurent.pinchart,
	sakari.ailus, linux-media
  Cc: Boris Brezillon, tfiga, hiroh, nicolas, Brian.Starkey, kernel,
	narmstrong, linux-kernel, frkoenig, mjourdan, stanimir.varbanov

On 17/07/2020 13:54, Helen Koike wrote:
> From: Hans Verkuil <hans.verkuil@cisco.com>
> 
> Those extended buffer ops have several purpose:
> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
>    the number of ns elapsed since 1970
> 2/ Unify single/multiplanar handling
> 3/ Add a new start offset field to each v4l2 plane buffer info struct
>    to support the case where a single buffer object is storing all
>    planes data, each one being placed at a different offset
> 
> New hooks are created in v4l2_ioctl_ops so that drivers can start using
> these new objects.
> 
> The core takes care of converting new ioctls requests to old ones
> if the driver does not support the new hooks, and vice versa.
> 
> Note that the timecode field is gone, since there doesn't seem to be
> in-kernel users. We can be added back in the reserved area if needed or
> use the Request API to collect more metadata information from the
> frame.
> 
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> ---
> Changes in v4:
> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
> I think we can add this later, so I removed it from this RFC to simplify it.
> - Remove num_planes field from struct v4l2_ext_buffer
> - Add flags field to struct v4l2_ext_create_buffers
> - Reformulate struct v4l2_ext_plane
> - Fix some bugs caught by v4l2-compliance
> - Rebased on top of media/master (post 5.8-rc1)
> 
> Changes in v3:
> - Rebased on top of media/master (post 5.4-rc1)
> 
> Changes in v2:
> - Add reserved space to v4l2_ext_buffer so that new fields can be added
>   later on
> ---
>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
>  drivers/media/v4l2-core/v4l2-ioctl.c | 349 +++++++++++++++++++++++++--
>  include/media/v4l2-ioctl.h           |  26 ++
>  include/uapi/linux/videodev2.h       |  89 +++++++
>  4 files changed, 471 insertions(+), 22 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
> index e1829906bc086..cb21ee8eb075c 100644
> --- a/drivers/media/v4l2-core/v4l2-dev.c
> +++ b/drivers/media/v4l2-core/v4l2-dev.c
> @@ -720,15 +720,34 @@ static void determine_valid_ioctls(struct video_device *vdev)
>  		SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_sdr_out);
>  	}
>  
> +	if (is_vid || is_tch) {
> +		/* ioctls valid for video and touch */
> +		if (ops->vidioc_querybuf || ops->vidioc_ext_querybuf)
> +			set_bit(_IOC_NR(VIDIOC_EXT_QUERYBUF), valid_ioctls);
> +		if (ops->vidioc_qbuf || ops->vidioc_ext_qbuf)
> +			set_bit(_IOC_NR(VIDIOC_EXT_QBUF), valid_ioctls);
> +		if (ops->vidioc_dqbuf || ops->vidioc_ext_dqbuf)
> +			set_bit(_IOC_NR(VIDIOC_EXT_DQBUF), valid_ioctls);
> +		if (ops->vidioc_create_bufs || ops->vidioc_ext_create_bufs)
> +			set_bit(_IOC_NR(VIDIOC_EXT_CREATE_BUFS), valid_ioctls);
> +		if (ops->vidioc_prepare_buf || ops->vidioc_ext_prepare_buf)
> +			set_bit(_IOC_NR(VIDIOC_EXT_PREPARE_BUF), valid_ioctls);
> +	}
> +
>  	if (is_vid || is_vbi || is_sdr || is_tch || is_meta) {
>  		/* ioctls valid for video, vbi, sdr, touch and metadata */
>  		SET_VALID_IOCTL(ops, VIDIOC_REQBUFS, vidioc_reqbufs);
> -		SET_VALID_IOCTL(ops, VIDIOC_QUERYBUF, vidioc_querybuf);
> -		SET_VALID_IOCTL(ops, VIDIOC_QBUF, vidioc_qbuf);
>  		SET_VALID_IOCTL(ops, VIDIOC_EXPBUF, vidioc_expbuf);
> -		SET_VALID_IOCTL(ops, VIDIOC_DQBUF, vidioc_dqbuf);
> -		SET_VALID_IOCTL(ops, VIDIOC_CREATE_BUFS, vidioc_create_bufs);
> -		SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, vidioc_prepare_buf);
> +		if (ops->vidioc_querybuf || ops->vidioc_ext_querybuf)
> +			set_bit(_IOC_NR(VIDIOC_QUERYBUF), valid_ioctls);
> +		if (ops->vidioc_qbuf || ops->vidioc_ext_qbuf)
> +			set_bit(_IOC_NR(VIDIOC_QBUF), valid_ioctls);
> +		if (ops->vidioc_dqbuf || ops->vidioc_ext_dqbuf)
> +			set_bit(_IOC_NR(VIDIOC_DQBUF), valid_ioctls);
> +		if (ops->vidioc_create_bufs || ops->vidioc_ext_create_bufs)
> +			set_bit(_IOC_NR(VIDIOC_CREATE_BUFS), valid_ioctls);
> +		if (ops->vidioc_prepare_buf || ops->vidioc_ext_prepare_buf)
> +			set_bit(_IOC_NR(VIDIOC_PREPARE_BUF), valid_ioctls);
>  		SET_VALID_IOCTL(ops, VIDIOC_STREAMON, vidioc_streamon);
>  		SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, vidioc_streamoff);
>  	}
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 3b77433f6c32b..5ddd57939c49c 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -527,6 +527,26 @@ static void v4l_print_buffer(const void *arg, bool write_only)
>  			tc->type, tc->flags, tc->frames, *(__u32 *)tc->userbits);
>  }
>  
> +static void v4l_print_ext_buffer(const void *arg, bool write_only)
> +{
> +	const struct v4l2_ext_buffer *e = arg;
> +	const struct v4l2_ext_plane *plane;
> +	unsigned int i;
> +
> +	pr_cont("%lld index=%d, type=%s, flags=0x%08x, field=%s, sequence=%d, memory=%s\n",
> +		e->timestamp, e->index, prt_names(e->type, v4l2_type_names),
> +		e->flags, prt_names(e->field, v4l2_field_names),
> +		e->sequence, prt_names(e->memory, v4l2_memory_names));
> +
> +	for (i = 0; i < VIDEO_MAX_PLANES &&
> +		    e->planes[i].buffer_length; i++) {
> +		plane = &e->planes[i];
> +		pr_debug("plane %d: buffer_length=%d, plane_length=%d offset=0x%08x\n",
> +			 i, plane->buffer_length, plane->plane_length,
> +			 plane->offset);
> +	}
> +}
> +
>  static void v4l_print_exportbuffer(const void *arg, bool write_only)
>  {
>  	const struct v4l2_exportbuffer *p = arg;
> @@ -546,6 +566,15 @@ static void v4l_print_create_buffers(const void *arg, bool write_only)
>  	v4l_print_format(&p->format, write_only);
>  }
>  
> +static void v4l_print_ext_create_buffers(const void *arg, bool write_only)
> +{
> +	const struct v4l2_ext_create_buffers *p = arg;
> +
> +	pr_cont("index=%d, count=%d, memory=%s, ", p->index, p->count,
> +		prt_names(p->memory, v4l2_memory_names));
> +	v4l_print_ext_pix_format(&p->format, write_only);
> +}
> +
>  static void v4l_print_streamparm(const void *arg, bool write_only)
>  {
>  	const struct v4l2_streamparm *p = arg;
> @@ -1214,6 +1243,139 @@ int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
>  }
>  EXPORT_SYMBOL_GPL(v4l2_format_to_ext_pix_format);
>  
> +/*
> + * If mplane_cap is true, b->m.planes should have a valid pointer of a
> + * struct v4l2_plane array, and b->length with its size
> + */
> +int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
> +			      struct v4l2_buffer *b, bool mplane_cap)
> +{
> +	unsigned int planes_array_size = b->length;
> +	struct v4l2_plane *planes = b->m.planes;
> +	u64 nsecs;
> +
> +	if (!mplane_cap && e->planes[1].buffer_length != 0)
> +		return -EINVAL;
> +
> +	memset(b, 0, sizeof(*b));
> +
> +	b->index = e->index;
> +	b->flags = e->flags;
> +	b->field = e->field;
> +	b->sequence = e->sequence;
> +	b->memory = e->memory;
> +	b->request_fd = e->request_fd;
> +	b->timestamp.tv_sec = div64_u64_rem(e->timestamp, NSEC_PER_SEC, &nsecs);
> +	b->timestamp.tv_usec = (u32)nsecs / NSEC_PER_USEC;
> +
> +	if (mplane_cap) {
> +		unsigned int i;
> +
> +		if (!planes || !planes_array_size)
> +			return -EINVAL;
> +
> +		b->m.planes = planes;
> +
> +		if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +			b->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
> +		else
> +			b->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
> +
> +		for (i = 0; i < VIDEO_MAX_PLANES && i < planes_array_size &&
> +			    e->planes[i].buffer_length; i++) {
> +
> +			if (b->memory != V4L2_MEMORY_MMAP && e->planes[i].offset)
> +				return -EINVAL;
> +
> +			memset(&b->m.planes[i], 0, sizeof(b->m.planes[i]));
> +
> +			if (b->memory == V4L2_MEMORY_MMAP)
> +				b->m.planes[i].m.mem_offset = e->planes[i].offset;
> +			else if (b->memory == V4L2_MEMORY_DMABUF)
> +				b->m.planes[i].m.fd = e->planes[i].m.dmabuf_fd;
> +			else
> +				b->m.planes[i].m.userptr = e->planes[i].m.userptr;
> +
> +			b->m.planes[i].bytesused = e->planes[i].plane_length;
> +			b->m.planes[i].length = e->planes[i].buffer_length;
> +		}
> +		/* In multi-planar, length contain the number of planes */
> +		b->length = i;
> +	} else {
> +		b->type = e->type;
> +		b->bytesused = e->planes[0].plane_length;
> +		b->length = e->planes[0].buffer_length;
> +
> +		if (b->memory != V4L2_MEMORY_MMAP && e->planes[0].offset)
> +			return -EINVAL;
> +
> +		if (b->memory == V4L2_MEMORY_MMAP)
> +			b->m.offset = e->planes[0].offset;
> +		else if (b->memory == V4L2_MEMORY_DMABUF)
> +			b->m.fd = e->planes[0].m.dmabuf_fd;
> +		else
> +			b->m.userptr = e->planes[0].m.userptr;
> +	}
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_ext_buffer_to_buffer);
> +
> +int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
> +			      struct v4l2_ext_buffer *e)
> +{
> +	memset(e, 0, sizeof(*e));
> +
> +	e->index = b->index;
> +	e->flags = b->flags;
> +	e->field = b->field;
> +	e->sequence = b->sequence;
> +	e->memory = b->memory;
> +	e->request_fd = b->request_fd;
> +	e->timestamp = b->timestamp.tv_sec * NSEC_PER_SEC +
> +		b->timestamp.tv_usec * NSEC_PER_USEC;
> +	if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
> +		unsigned int i;
> +
> +		if (!b->m.planes)
> +			return -EINVAL;
> +
> +		if (b->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
> +			e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +		else
> +			e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
> +
> +		/* In multi-planar, length contain the number of planes */
> +		for (i = 0; i < b->length; i++) {
> +			if (b->memory == V4L2_MEMORY_MMAP)
> +				e->planes[i].offset = b->m.planes[i].m.mem_offset;
> +			else if (b->memory == V4L2_MEMORY_DMABUF)
> +				e->planes[i].m.dmabuf_fd = b->m.planes[i].m.fd;
> +			else
> +				e->planes[i].m.userptr = b->m.planes[i].m.userptr;
> +
> +			e->planes[i].buffer_length = b->m.planes[i].length;
> +			e->planes[i].plane_length = b->m.planes[i].bytesused;
> +			if (b->m.planes[i].data_offset)
> +				pr_warn("Ignoring data_offset value %d\n",
> +					b->m.planes[i].data_offset);
> +		}
> +	} else {
> +		e->type = b->type;
> +		e->planes[0].plane_length = b->bytesused;
> +		e->planes[0].buffer_length = b->length;
> +		if (b->memory == V4L2_MEMORY_MMAP)
> +			e->planes[0].offset = b->m.offset;
> +		else if (b->memory == V4L2_MEMORY_DMABUF)
> +			e->planes[0].m.dmabuf_fd = b->m.fd;
> +		else
> +			e->planes[0].m.userptr = b->m.userptr;
> +	}
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_buffer_to_ext_buffer);
> +
>  static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
>  				struct file *file, void *fh, void *arg)
>  {
> @@ -2467,31 +2629,112 @@ static int v4l_reqbufs(const struct v4l2_ioctl_ops *ops,
>  	return ops->vidioc_reqbufs(file, fh, p);
>  }
>  
> +static int v4l_do_buf_op(int (*op)(struct file *, void *,
> +				   struct v4l2_buffer *),
> +			 int (*ext_op)(struct file *, void *,
> +				       struct v4l2_ext_buffer *),
> +			 struct file *file, void *fh, struct v4l2_buffer *b)
> +{
> +	struct v4l2_ext_buffer e;
> +	int ret;
> +
> +	ret = check_fmt(file, b->type);
> +	if (ret)
> +		return ret;
> +
> +	if (op)
> +		return op(file, fh, b);
> +
> +	ret = v4l2_buffer_to_ext_buffer(b, &e);
> +	if (ret)
> +		return ret;
> +
> +	ret = ext_op(file, fh, &e);
> +	if (ret)
> +		return ret;
> +
> +	v4l2_ext_buffer_to_buffer(&e, b, V4L2_TYPE_IS_MULTIPLANAR(b->type));
> +	return 0;
> +}
> +
> +static int v4l_do_ext_buf_op(int (*op)(struct file *, void *,
> +				       struct v4l2_buffer *),
> +			     int (*ext_op)(struct file *, void *,
> +					   struct v4l2_ext_buffer *),
> +			     struct file *file, void *fh,
> +			     struct v4l2_ext_buffer *e)
> +{
> +	struct video_device *vdev = video_devdata(file);
> +	struct v4l2_plane planes[VIDEO_MAX_PLANES];
> +	struct v4l2_buffer b;
> +	bool mplane_cap;
> +	int ret;
> +
> +	ret = check_fmt(file, e->type);
> +	if (ret)
> +		return ret;
> +
> +	if (ext_op)
> +		return ext_op(file, fh, e);
> +
> +	mplane_cap = !!(vdev->device_caps &
> +			(V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> +			 V4L2_CAP_VIDEO_OUTPUT_MPLANE |
> +			 V4L2_CAP_VIDEO_M2M_MPLANE));
> +	b.m.planes = planes;
> +	b.length = VIDEO_MAX_PLANES;
> +	ret = v4l2_ext_buffer_to_buffer(e, &b, mplane_cap);
> +	if (ret)
> +		return ret;
> +
> +	ret = op(file, fh, &b);
> +	if (ret)
> +		return ret;
> +
> +	v4l2_buffer_to_ext_buffer(&b, e);
> +	return 0;
> +}
> +
>  static int v4l_querybuf(const struct v4l2_ioctl_ops *ops,
> -				struct file *file, void *fh, void *arg)
> +			struct file *file, void *fh, void *arg)
>  {
> -	struct v4l2_buffer *p = arg;
> -	int ret = check_fmt(file, p->type);
> +	return v4l_do_buf_op(ops->vidioc_querybuf, ops->vidioc_ext_querybuf,
> +			     file, fh, arg);
> +}
>  
> -	return ret ? ret : ops->vidioc_querybuf(file, fh, p);
> +static int v4l_ext_querybuf(const struct v4l2_ioctl_ops *ops,
> +			    struct file *file, void *fh, void *arg)
> +{
> +	return v4l_do_ext_buf_op(ops->vidioc_querybuf,
> +				 ops->vidioc_ext_querybuf, file, fh, arg);
>  }
>  
>  static int v4l_qbuf(const struct v4l2_ioctl_ops *ops,
> -				struct file *file, void *fh, void *arg)
> +		    struct file *file, void *fh, void *arg)
>  {
> -	struct v4l2_buffer *p = arg;
> -	int ret = check_fmt(file, p->type);
> +	return v4l_do_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
> +			     file, fh, arg);
> +}
>  
> -	return ret ? ret : ops->vidioc_qbuf(file, fh, p);
> +static int v4l_ext_qbuf(const struct v4l2_ioctl_ops *ops,
> +			struct file *file, void *fh, void *arg)
> +{
> +	return v4l_do_ext_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
> +				 file, fh, arg);
>  }
>  
>  static int v4l_dqbuf(const struct v4l2_ioctl_ops *ops,
> -				struct file *file, void *fh, void *arg)
> +		     struct file *file, void *fh, void *arg)
>  {
> -	struct v4l2_buffer *p = arg;
> -	int ret = check_fmt(file, p->type);
> +	return v4l_do_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
> +			     file, fh, arg);
> +}
>  
> -	return ret ? ret : ops->vidioc_dqbuf(file, fh, p);
> +static int v4l_ext_dqbuf(const struct v4l2_ioctl_ops *ops,
> +			 struct file *file, void *fh, void *arg)
> +{
> +	return v4l_do_ext_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
> +				 file, fh, arg);
>  }
>  
>  static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
> @@ -2507,7 +2750,27 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
>  
>  	v4l_sanitize_format(&create->format);
>  
> -	ret = ops->vidioc_create_bufs(file, fh, create);
> +	if (ops->vidioc_create_bufs) {
> +		ret = ops->vidioc_create_bufs(file, fh, create);
> +	} else {
> +		struct v4l2_ext_create_buffers ecreate = {
> +			.count = create->count,
> +			.memory = create->memory,
> +		};
> +
> +		ret = v4l2_format_to_ext_pix_format(&create->format,
> +						    &ecreate.format, true);
> +		if (ret)
> +			return ret;
> +
> +		ret = ops->vidioc_ext_create_bufs(file, fh, &ecreate);
> +		if (ret)
> +			return ret;
> +
> +		create->index = ecreate.index;
> +		create->count = ecreate.count;
> +		create->capabilities = ecreate.capabilities;
> +	}
>  
>  	if (create->format.type == V4L2_BUF_TYPE_VIDEO_CAPTURE ||
>  	    create->format.type == V4L2_BUF_TYPE_VIDEO_OUTPUT)
> @@ -2516,13 +2779,60 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
>  	return ret;
>  }
>  
> +static int v4l_ext_create_bufs(const struct v4l2_ioctl_ops *ops,
> +			       struct file *file, void *fh, void *arg)
> +{
> +	struct v4l2_ext_create_buffers *ecreate = arg;
> +	struct video_device *vdev = video_devdata(file);
> +	struct v4l2_create_buffers create = {
> +		.count = ecreate->count,
> +		.memory = ecreate->memory,
> +		.flags = ecreate->flags,
> +	};
> +	bool mplane_cap;
> +	int ret;
> +
> +	ret = check_fmt(file, ecreate->format.type);
> +	if (ret)
> +		return ret;
> +
> +	if (ops->vidioc_ext_create_bufs)
> +		return ops->vidioc_ext_create_bufs(file, fh, ecreate);
> +
> +	mplane_cap = !!(vdev->device_caps &
> +			(V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> +			 V4L2_CAP_VIDEO_OUTPUT_MPLANE |
> +			 V4L2_CAP_VIDEO_M2M_MPLANE));
> +	ret = v4l2_ext_pix_format_to_format(&ecreate->format,
> +					    &create.format, mplane_cap, true);
> +	if (ret)
> +		return ret;
> +
> +	ret = v4l_create_bufs(ops, file, fh, &create);
> +	if (ret)
> +		return ret;
> +
> +	ecreate->index = create.index;
> +	ecreate->count = create.count;
> +	ecreate->capabilities = create.capabilities;
> +
> +	return 0;
> +}
> +
>  static int v4l_prepare_buf(const struct v4l2_ioctl_ops *ops,
> -				struct file *file, void *fh, void *arg)
> +			   struct file *file, void *fh, void *arg)
>  {
> -	struct v4l2_buffer *b = arg;
> -	int ret = check_fmt(file, b->type);
> +	return v4l_do_buf_op(ops->vidioc_prepare_buf,
> +			     ops->vidioc_ext_prepare_buf,
> +			     file, fh, arg);
> +}
>  
> -	return ret ? ret : ops->vidioc_prepare_buf(file, fh, b);
> +static int v4l_ext_prepare_buf(const struct v4l2_ioctl_ops *ops,
> +			       struct file *file, void *fh, void *arg)
> +{
> +	return v4l_do_ext_buf_op(ops->vidioc_prepare_buf,
> +				 ops->vidioc_ext_prepare_buf,
> +				 file, fh, arg);
>  }
>  
>  static int v4l_g_parm(const struct v4l2_ioctl_ops *ops,
> @@ -3196,12 +3506,15 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>  	IOCTL_INFO(VIDIOC_S_EXT_PIX_FMT, v4l_s_ext_pix_fmt, v4l_print_ext_pix_format, INFO_FL_PRIO),
>  	IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>  	IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
> +	IOCTL_INFO(VIDIOC_EXT_QUERYBUF, v4l_ext_querybuf, v4l_print_ext_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_ext_buffer, planes)),
>  	IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
>  	IOCTL_INFO(VIDIOC_S_FBUF, v4l_stub_s_fbuf, v4l_print_framebuffer, INFO_FL_PRIO),
>  	IOCTL_INFO(VIDIOC_OVERLAY, v4l_overlay, v4l_print_u32, INFO_FL_PRIO),
>  	IOCTL_INFO(VIDIOC_QBUF, v4l_qbuf, v4l_print_buffer, INFO_FL_QUEUE),
>  	IOCTL_INFO(VIDIOC_EXPBUF, v4l_stub_expbuf, v4l_print_exportbuffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_exportbuffer, flags)),
> +	IOCTL_INFO(VIDIOC_EXT_QBUF, v4l_ext_qbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),
>  	IOCTL_INFO(VIDIOC_DQBUF, v4l_dqbuf, v4l_print_buffer, INFO_FL_QUEUE),
> +	IOCTL_INFO(VIDIOC_EXT_DQBUF, v4l_ext_dqbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),
>  	IOCTL_INFO(VIDIOC_STREAMON, v4l_streamon, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
>  	IOCTL_INFO(VIDIOC_STREAMOFF, v4l_streamoff, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
>  	IOCTL_INFO(VIDIOC_G_PARM, v4l_g_parm, v4l_print_streamparm, INFO_FL_CLEAR(v4l2_streamparm, type)),
> @@ -3266,7 +3579,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>  	IOCTL_INFO(VIDIOC_SUBSCRIBE_EVENT, v4l_subscribe_event, v4l_print_event_subscription, 0),
>  	IOCTL_INFO(VIDIOC_UNSUBSCRIBE_EVENT, v4l_unsubscribe_event, v4l_print_event_subscription, 0),
>  	IOCTL_INFO(VIDIOC_CREATE_BUFS, v4l_create_bufs, v4l_print_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
> +	IOCTL_INFO(VIDIOC_EXT_CREATE_BUFS, v4l_ext_create_bufs, v4l_print_ext_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>  	IOCTL_INFO(VIDIOC_PREPARE_BUF, v4l_prepare_buf, v4l_print_buffer, INFO_FL_QUEUE),
> +	IOCTL_INFO(VIDIOC_EXT_PREPARE_BUF, v4l_ext_prepare_buf, v4l_print_ext_buffer, INFO_FL_QUEUE),
>  	IOCTL_INFO(VIDIOC_ENUM_DV_TIMINGS, v4l_stub_enum_dv_timings, v4l_print_enum_dv_timings, INFO_FL_CLEAR(v4l2_enum_dv_timings, pad)),
>  	IOCTL_INFO(VIDIOC_QUERY_DV_TIMINGS, v4l_stub_query_dv_timings, v4l_print_dv_timings, INFO_FL_ALWAYS_COPY),
>  	IOCTL_INFO(VIDIOC_DV_TIMINGS_CAP, v4l_stub_dv_timings_cap, v4l_print_dv_timings_cap, INFO_FL_CLEAR(v4l2_dv_timings_cap, pad)),
> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
> index 525ce86725260..524caedebab3d 100644
> --- a/include/media/v4l2-ioctl.h
> +++ b/include/media/v4l2-ioctl.h
> @@ -169,16 +169,26 @@ struct v4l2_fh;
>   *	:ref:`VIDIOC_REQBUFS <vidioc_reqbufs>` ioctl
>   * @vidioc_querybuf: pointer to the function that implements
>   *	:ref:`VIDIOC_QUERYBUF <vidioc_querybuf>` ioctl
> + * @vidioc_ext_querybuf: pointer to the function that implements
> + *	:ref:`VIDIOC_EXT_QUERYBUF <vidioc_ext_querybuf>` ioctl
>   * @vidioc_qbuf: pointer to the function that implements
>   *	:ref:`VIDIOC_QBUF <vidioc_qbuf>` ioctl
> + * @vidioc_ext_qbuf: pointer to the function that implements
> + *	:ref:`VIDIOC_EXT_QBUF <vidioc_ext_qbuf>` ioctl
>   * @vidioc_expbuf: pointer to the function that implements
>   *	:ref:`VIDIOC_EXPBUF <vidioc_expbuf>` ioctl
>   * @vidioc_dqbuf: pointer to the function that implements
>   *	:ref:`VIDIOC_DQBUF <vidioc_qbuf>` ioctl
> + * @vidioc_ext_dqbuf: pointer to the function that implements
> + *	:ref:`VIDIOC_EXT_DQBUF <vidioc_ext_qbuf>` ioctl
>   * @vidioc_create_bufs: pointer to the function that implements
>   *	:ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl
> + * @vidioc_ext_create_bufs: pointer to the function that implements
> + *	:ref:`VIDIOC_EXT_CREATE_BUFS <vidioc_ext_create_bufs>` ioctl
>   * @vidioc_prepare_buf: pointer to the function that implements
>   *	:ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl
> + * @vidioc_ext_prepare_buf: pointer to the function that implements
> + *	:ref:`VIDIOC_EXT_PREPARE_BUF <vidioc_ext_prepare_buf>` ioctl
>   * @vidioc_overlay: pointer to the function that implements
>   *	:ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl
>   * @vidioc_g_fbuf: pointer to the function that implements
> @@ -439,17 +449,27 @@ struct v4l2_ioctl_ops {
>  			      struct v4l2_requestbuffers *b);
>  	int (*vidioc_querybuf)(struct file *file, void *fh,
>  			       struct v4l2_buffer *b);
> +	int (*vidioc_ext_querybuf)(struct file *file, void *fh,
> +				   struct v4l2_ext_buffer *b);
>  	int (*vidioc_qbuf)(struct file *file, void *fh,
>  			   struct v4l2_buffer *b);
> +	int (*vidioc_ext_qbuf)(struct file *file, void *fh,
> +			       struct v4l2_ext_buffer *b);
>  	int (*vidioc_expbuf)(struct file *file, void *fh,
>  			     struct v4l2_exportbuffer *e);
>  	int (*vidioc_dqbuf)(struct file *file, void *fh,
>  			    struct v4l2_buffer *b);
> +	int (*vidioc_ext_dqbuf)(struct file *file, void *fh,
> +				struct v4l2_ext_buffer *b);
>  
>  	int (*vidioc_create_bufs)(struct file *file, void *fh,
>  				  struct v4l2_create_buffers *b);
> +	int (*vidioc_ext_create_bufs)(struct file *file, void *fh,
> +				      struct v4l2_ext_create_buffers *b);
>  	int (*vidioc_prepare_buf)(struct file *file, void *fh,
>  				  struct v4l2_buffer *b);
> +	int (*vidioc_ext_prepare_buf)(struct file *file, void *fh,
> +				      struct v4l2_ext_buffer *b);
>  
>  	int (*vidioc_overlay)(struct file *file, void *fh, unsigned int i);
>  	int (*vidioc_g_fbuf)(struct file *file, void *fh,
> @@ -758,6 +778,12 @@ int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
>  				  struct v4l2_format *f,
>  				  bool mplane_cap, bool strict);
>  
> +int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
> +			      struct v4l2_buffer *b,
> +			      bool mplane_cap);
> +int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
> +			      struct v4l2_ext_buffer *e);
> +
>  /*
>   * The user space interpretation of the 'v4l2_event' differs
>   * based on the 'time_t' definition on 32-bit architectures, so
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index fc04c81ce7713..f4906adddc280 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -994,6 +994,37 @@ struct v4l2_plane {
>  	__u32			reserved[11];
>  };
>  
> +/**
> + * struct v4l2_ext_plane - extended plane buffer info
> + * @buffer_length:	size of the entire buffer in bytes, should fit
> + *			@offset + @plane_length
> + * @plane_length:	size of the plane in bytes.
> + * @userptr:		when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
> + *			to this plane.
> + * @dmabuf_fd:		when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
> + *			associated with this plane.
> + * @offset:		offset in the memory buffer where the plane starts. If
> + *			V4L2_MEMORY_MMAP is used, then it can be a "cookie" that
> + *			should be passed to mmap() called on the video node.

I wouldn't overload this field. Just keep a mem_offset field in the union m.

The offset field itself is still valid, even for MMAPed planes.

> + * @reserved:		extra space reserved for future fields, must be set to 0.
> + *
> + *
> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
> + * can have one plane for Y, and another for interleaved CbCr components.
> + * Each plane can reside in a separate memory buffer, or even in
> + * a completely separate memory node (e.g. in embedded devices).
> + */
> +struct v4l2_ext_plane {
> +	__u32 buffer_length;
> +	__u32 plane_length;
> +	union {
> +		__u64 userptr;
> +		__s32 dmabuf_fd;
> +	} m;
> +	__u32 offset;
> +	__u32 reserved[4];
> +};
> +
>  /**
>   * struct v4l2_buffer - video buffer info
>   * @index:	id number of the buffer
> @@ -1055,6 +1086,36 @@ struct v4l2_buffer {
>  	};
>  };
>  
> +/**
> + * struct v4l2_ext_buffer - extended video buffer info
> + * @index:	id number of the buffer
> + * @type:	V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
> + * @flags:	buffer informational flags
> + * @field:	enum v4l2_field; field order of the image in the buffer
> + * @timestamp:	frame timestamp
> + * @sequence:	sequence count of this frame
> + * @memory:	enum v4l2_memory; the method, in which the actual video data is
> + *		passed
> + * @planes:	per-plane buffer information
> + * @request_fd:	fd of the request that this buffer should use
> + * @reserved:	extra space reserved for future fields, must be set to 0
> + *
> + * Contains data exchanged by application and driver using one of the Streaming
> + * I/O methods.
> + */
> +struct v4l2_ext_buffer {
> +	__u32 index;
> +	__u32 type;
> +	__u32 flags;

Make this a u64: we have quite a few flags already.

> +	__u32 field;
> +	__u64 timestamp;

We need a second timestamp here, i.e. the time at which the buffer was finalized
and returned to userspace. This is needed to let userspace make a timeline between
events and buffers. Each v4l2_event has a timestamp but it can't be used to
see if a buffer was returned before or after that event. It's something that's
been annoying me for some time now.

A '__u64 seq_timestamp' seems like a decent name.

Regards,

	Hans

> +	__u32 sequence;
> +	__u32 memory;
> +	__u32 request_fd;
> +	struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
> +	__u32 reserved[4];
> +};
> +
>  #ifndef __KERNEL__
>  /**
>   * v4l2_timeval_to_ns - Convert timeval to nanoseconds
> @@ -2520,6 +2581,29 @@ struct v4l2_create_buffers {
>  	__u32			reserved[6];
>  };
>  
> +/**
> + * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
> + * @index:	on return, index of the first created buffer
> + * @count:	entry: number of requested buffers,
> + *		return: number of created buffers
> + * @memory:	enum v4l2_memory; buffer memory type
> + * @capabilities: capabilities of this buffer type.
> + * @format:	frame format, for which buffers are requested
> + * @flags:	additional buffer management attributes (ignored unless the
> + *		queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
> + *		and configured for MMAP streaming I/O).
> + * @reserved:	extra space reserved for future fields, must be set to 0
> + */
> +struct v4l2_ext_create_buffers {
> +	__u32				index;
> +	__u32				count;
> +	__u32				memory;
> +	struct v4l2_ext_pix_format	format;
> +	__u32				capabilities;
> +	__u32				flags;
> +	__u32 reserved[4];
> +};
> +
>  /*
>   *	I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
>   *
> @@ -2623,6 +2707,11 @@ struct v4l2_create_buffers {
>  #define VIDIOC_G_EXT_PIX_FMT	_IOWR('V', 104, struct v4l2_ext_pix_format)
>  #define VIDIOC_S_EXT_PIX_FMT	_IOWR('V', 105, struct v4l2_ext_pix_format)
>  #define VIDIOC_TRY_EXT_PIX_FMT	_IOWR('V', 106, struct v4l2_ext_pix_format)
> +#define VIDIOC_EXT_CREATE_BUFS	_IOWR('V', 107, struct v4l2_ext_create_buffers)
> +#define VIDIOC_EXT_QUERYBUF	_IOWR('V', 108, struct v4l2_ext_buffer)
> +#define VIDIOC_EXT_QBUF		_IOWR('V', 109, struct v4l2_ext_buffer)
> +#define VIDIOC_EXT_DQBUF	_IOWR('V', 110, struct v4l2_ext_buffer)
> +#define VIDIOC_EXT_PREPARE_BUF	_IOWR('V', 111, struct v4l2_ext_buffer)
>  
>  /* Reminder: when adding new ioctls please add support for them to
>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
> 


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v4 0/6] media: v4l2: Add extended fmt and buffer ioctls
  2020-07-17 11:54 [PATCH v4 0/6] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
                   ` (7 preceding siblings ...)
  2020-07-21 10:24 ` Hans Verkuil
@ 2020-07-21 11:15 ` Boris Brezillon
  2020-07-21 13:56   ` Helen Koike
  8 siblings, 1 reply; 29+ messages in thread
From: Boris Brezillon @ 2020-07-21 11:15 UTC (permalink / raw)
  To: Helen Koike, Brian.Starkey
  Cc: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus,
	linux-media, tfiga, hiroh, nicolas, kernel, narmstrong,
	linux-kernel, frkoenig, mjourdan, stanimir.varbanov

Hello Helen,

Just a few drive-by comments.

On Fri, 17 Jul 2020 08:54:29 -0300
Helen Koike <helen.koike@collabora.com> wrote:

> Hi,
> 
> I'm sorry for taking too long to submit v4.
> 
> It is not perfect, not all v4l2-compliance tests passes, but I'd like a review,
> specially on the API and potential problems, so I can focus on improving implementation
> and maybe drop the RFC tag for next version.
> 
> Follow below what changed in v4 and some items I'd like to discuss:
> 
> 
> * Ioctl to replace v4l2_pix_format
> ---------------------------------------------------------------------------------
> During last media summit, we agreed to create ioctls that replace the v4l2_pix_format
> struct and leave the other structs in the v4l2_format union alone.
> Thus I refactored the code to receive struct v4l2_ext_pix_format, and I renamed the
> ioctls, so now we have:
> 
> int ioctl(int fd, VIDIOC_G_EXT_FMT, struct v4l2_ext_pix_format *argp);

Maybe use the EXT_PIX_FMT suffix here since the struct is really only
about pixel formats.

> int ioctl(int fd, VIDIOC_S_EXT_FMT, struct v4l2_ext_pix_format *argp);
> int ioctl(int fd, VIDIOC_TRY_EXT_FMT, struct v4l2_ext_pix_format *argp);
> 
> The only valid types are V4L2_BUF_TYPE_VIDEO_CAPTURE and V4L2_BUF_TYPE_VIDEO_OUTPUT,
> all the other types are invalid with this API.
> 

[...]

> 
> 
> Boris Brezillon (5):
>   media: v4l2: Extend pixel formats to unify single/multi-planar
>     handling (and more)
>   media: videobuf2: Expose helpers to implement the _ext_fmt and
>     _ext_buf hooks
>   media: mediabus: Add helpers to convert a ext_pix format to/from a
>     mbus_fmt
>   media: vivid: Convert the capture and output drivers to
>     EXT_FMT/EXT_BUF
>   media: vimc: Implement the ext_fmt and ext_buf hooks

I think you should take ownership of these patches. The end result is
likely to be completely different from what I initially posted, and
you're the one doing the hard work here.

> 
> Hans Verkuil (1):
>   media: v4l2: Add extended buffer operations
> 
>  .../media/common/videobuf2/videobuf2-core.c   |   2 +
>  .../media/common/videobuf2/videobuf2-v4l2.c   | 549 +++++-----
>  .../media/test-drivers/vimc/vimc-capture.c    |  61 +-
>  drivers/media/test-drivers/vimc/vimc-common.c |   6 +-
>  drivers/media/test-drivers/vimc/vimc-common.h |   2 +-
>  drivers/media/test-drivers/vivid/vivid-core.c |  70 +-
>  .../test-drivers/vivid/vivid-touch-cap.c      |  26 +-
>  .../test-drivers/vivid/vivid-touch-cap.h      |   3 +-
>  .../media/test-drivers/vivid/vivid-vid-cap.c  | 169 +---
>  .../media/test-drivers/vivid/vivid-vid-cap.h  |  15 +-
>  .../media/test-drivers/vivid/vivid-vid-out.c  | 193 ++--
>  .../media/test-drivers/vivid/vivid-vid-out.h  |  15 +-
>  drivers/media/v4l2-core/v4l2-dev.c            |  50 +-
>  drivers/media/v4l2-core/v4l2-ioctl.c          | 934 ++++++++++++++++--
>  include/media/v4l2-ioctl.h                    |  60 ++
>  include/media/v4l2-mediabus.h                 |  42 +
>  include/media/videobuf2-core.h                |   6 +-
>  include/media/videobuf2-v4l2.h                |  21 +-
>  include/uapi/linux/videodev2.h                | 144 +++
>  19 files changed, 1650 insertions(+), 718 deletions(-)
> 


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v4 2/6] media: v4l2: Add extended buffer operations
  2020-07-17 11:54 ` [PATCH v4 2/6] media: v4l2: Add extended buffer operations Helen Koike
  2020-07-21 10:48   ` Hans Verkuil
@ 2020-07-21 11:26   ` Stanimir Varbanov
  2020-07-21 13:54     ` Helen Koike
  1 sibling, 1 reply; 29+ messages in thread
From: Stanimir Varbanov @ 2020-07-21 11:26 UTC (permalink / raw)
  To: Helen Koike, mchehab, hans.verkuil, laurent.pinchart,
	sakari.ailus, linux-media
  Cc: Boris Brezillon, tfiga, hiroh, nicolas, Brian.Starkey, kernel,
	narmstrong, linux-kernel, frkoenig, mjourdan



On 7/17/20 2:54 PM, Helen Koike wrote:
> From: Hans Verkuil <hans.verkuil@cisco.com>
> 
> Those extended buffer ops have several purpose:
> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
>    the number of ns elapsed since 1970
> 2/ Unify single/multiplanar handling
> 3/ Add a new start offset field to each v4l2 plane buffer info struct
>    to support the case where a single buffer object is storing all
>    planes data, each one being placed at a different offset
> 
> New hooks are created in v4l2_ioctl_ops so that drivers can start using
> these new objects.
> 
> The core takes care of converting new ioctls requests to old ones
> if the driver does not support the new hooks, and vice versa.
> 
> Note that the timecode field is gone, since there doesn't seem to be
> in-kernel users. We can be added back in the reserved area if needed or
> use the Request API to collect more metadata information from the
> frame.
> 
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> ---
> Changes in v4:
> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
> I think we can add this later, so I removed it from this RFC to simplify it.
> - Remove num_planes field from struct v4l2_ext_buffer
> - Add flags field to struct v4l2_ext_create_buffers
> - Reformulate struct v4l2_ext_plane
> - Fix some bugs caught by v4l2-compliance
> - Rebased on top of media/master (post 5.8-rc1)
> 
> Changes in v3:
> - Rebased on top of media/master (post 5.4-rc1)
> 
> Changes in v2:
> - Add reserved space to v4l2_ext_buffer so that new fields can be added
>   later on
> ---
>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
>  drivers/media/v4l2-core/v4l2-ioctl.c | 349 +++++++++++++++++++++++++--
>  include/media/v4l2-ioctl.h           |  26 ++
>  include/uapi/linux/videodev2.h       |  89 +++++++
>  4 files changed, 471 insertions(+), 22 deletions(-)
> 

<cut>

> +/**
> + * struct v4l2_ext_plane - extended plane buffer info
> + * @buffer_length:	size of the entire buffer in bytes, should fit
> + *			@offset + @plane_length
> + * @plane_length:	size of the plane in bytes.
> + * @userptr:		when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
> + *			to this plane.
> + * @dmabuf_fd:		when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
> + *			associated with this plane.
> + * @offset:		offset in the memory buffer where the plane starts. If
> + *			V4L2_MEMORY_MMAP is used, then it can be a "cookie" that
> + *			should be passed to mmap() called on the video node.
> + * @reserved:		extra space reserved for future fields, must be set to 0.
> + *
> + *
> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
> + * can have one plane for Y, and another for interleaved CbCr components.
> + * Each plane can reside in a separate memory buffer, or even in
> + * a completely separate memory node (e.g. in embedded devices).
> + */
> +struct v4l2_ext_plane {
> +	__u32 buffer_length;
> +	__u32 plane_length;
> +	union {
> +		__u64 userptr;
> +		__s32 dmabuf_fd;
> +	} m;
> +	__u32 offset;
> +	__u32 reserved[4];
> +};
> +
>  /**
>   * struct v4l2_buffer - video buffer info
>   * @index:	id number of the buffer
> @@ -1055,6 +1086,36 @@ struct v4l2_buffer {
>  	};
>  };
>  
> +/**
> + * struct v4l2_ext_buffer - extended video buffer info
> + * @index:	id number of the buffer
> + * @type:	V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
> + * @flags:	buffer informational flags
> + * @field:	enum v4l2_field; field order of the image in the buffer
> + * @timestamp:	frame timestamp
> + * @sequence:	sequence count of this frame
> + * @memory:	enum v4l2_memory; the method, in which the actual video data is
> + *		passed
> + * @planes:	per-plane buffer information
> + * @request_fd:	fd of the request that this buffer should use
> + * @reserved:	extra space reserved for future fields, must be set to 0
> + *
> + * Contains data exchanged by application and driver using one of the Streaming
> + * I/O methods.
> + */
> +struct v4l2_ext_buffer {
> +	__u32 index;
> +	__u32 type;
> +	__u32 flags;
> +	__u32 field;
> +	__u64 timestamp;
> +	__u32 sequence;
> +	__u32 memory;
> +	__u32 request_fd;

This should be __s32, at least for consistency with dmabuf_fd?

> +	struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
> +	__u32 reserved[4];

I think we have to reserve more words here for future extensions.

I'd like also to propose to add here __s32 metadata_fd. The idea behind
this is to have a way to pass per-frame metadata dmabuf buffers for
synchronous type of metadata where the metadata is coming at the same
time with data buffers. What would be the format of the metadata buffer
is TBD.

One option for metadata buffer format could be:

header {
	num_ctrls
	array_of_ctrls [0..N]
		ctrl_id
		ctrl_size
		ctrl_offset
}

data {
	cid0	//offset of cid0 in dmabuf buffer
	cid1
	cidN
}

This will make easy to get concrete ctrl id without a need to parse the
whole metadata buffer. Also using dmabuf we don't need to copy data
between userspace <-> kernelspace (just cache syncs through
begin/end_cpu_access).

The open question is who will validate the metadata buffer when it comes
from userspace. The obvious answer is v4l2-core but looking into DRM
subsytem they give more freedom to the drivers, and just provide generic
helpers which are not mandatory.

I guess this will be a voice in the wilderness but I wanted to know your
opinion.

> +};
> +
>  #ifndef __KERNEL__
>  /**
>   * v4l2_timeval_to_ns - Convert timeval to nanoseconds
> @@ -2520,6 +2581,29 @@ struct v4l2_create_buffers {
>  	__u32			reserved[6];
>  };
>  
> +/**
> + * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
> + * @index:	on return, index of the first created buffer
> + * @count:	entry: number of requested buffers,
> + *		return: number of created buffers
> + * @memory:	enum v4l2_memory; buffer memory type
> + * @capabilities: capabilities of this buffer type.
> + * @format:	frame format, for which buffers are requested
> + * @flags:	additional buffer management attributes (ignored unless the
> + *		queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
> + *		and configured for MMAP streaming I/O).
> + * @reserved:	extra space reserved for future fields, must be set to 0
> + */
> +struct v4l2_ext_create_buffers {
> +	__u32				index;
> +	__u32				count;
> +	__u32				memory;
> +	struct v4l2_ext_pix_format	format;
> +	__u32				capabilities;
> +	__u32				flags;
> +	__u32 reserved[4];
> +};
> +
>  /*
>   *	I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
>   *
> @@ -2623,6 +2707,11 @@ struct v4l2_create_buffers {
>  #define VIDIOC_G_EXT_PIX_FMT	_IOWR('V', 104, struct v4l2_ext_pix_format)
>  #define VIDIOC_S_EXT_PIX_FMT	_IOWR('V', 105, struct v4l2_ext_pix_format)
>  #define VIDIOC_TRY_EXT_PIX_FMT	_IOWR('V', 106, struct v4l2_ext_pix_format)
> +#define VIDIOC_EXT_CREATE_BUFS	_IOWR('V', 107, struct v4l2_ext_create_buffers)
> +#define VIDIOC_EXT_QUERYBUF	_IOWR('V', 108, struct v4l2_ext_buffer)
> +#define VIDIOC_EXT_QBUF		_IOWR('V', 109, struct v4l2_ext_buffer)
> +#define VIDIOC_EXT_DQBUF	_IOWR('V', 110, struct v4l2_ext_buffer)
> +#define VIDIOC_EXT_PREPARE_BUF	_IOWR('V', 111, struct v4l2_ext_buffer)
>  
>  /* Reminder: when adding new ioctls please add support for them to
>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
> 

-- 
regards,
Stan

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v4 2/6] media: v4l2: Add extended buffer operations
  2020-07-21 11:26   ` Stanimir Varbanov
@ 2020-07-21 13:54     ` Helen Koike
  2020-07-21 14:30       ` Stanimir Varbanov
  0 siblings, 1 reply; 29+ messages in thread
From: Helen Koike @ 2020-07-21 13:54 UTC (permalink / raw)
  To: Stanimir Varbanov, mchehab, hans.verkuil, laurent.pinchart,
	sakari.ailus, linux-media
  Cc: Boris Brezillon, tfiga, hiroh, nicolas, Brian.Starkey, kernel,
	narmstrong, linux-kernel, frkoenig, mjourdan

Hi,

On 7/21/20 8:26 AM, Stanimir Varbanov wrote:
> 
> 
> On 7/17/20 2:54 PM, Helen Koike wrote:
>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>
>> Those extended buffer ops have several purpose:
>> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
>>    the number of ns elapsed since 1970
>> 2/ Unify single/multiplanar handling
>> 3/ Add a new start offset field to each v4l2 plane buffer info struct
>>    to support the case where a single buffer object is storing all
>>    planes data, each one being placed at a different offset
>>
>> New hooks are created in v4l2_ioctl_ops so that drivers can start using
>> these new objects.
>>
>> The core takes care of converting new ioctls requests to old ones
>> if the driver does not support the new hooks, and vice versa.
>>
>> Note that the timecode field is gone, since there doesn't seem to be
>> in-kernel users. We can be added back in the reserved area if needed or
>> use the Request API to collect more metadata information from the
>> frame.
>>
>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>> ---
>> Changes in v4:
>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
>> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
>> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
>> I think we can add this later, so I removed it from this RFC to simplify it.
>> - Remove num_planes field from struct v4l2_ext_buffer
>> - Add flags field to struct v4l2_ext_create_buffers
>> - Reformulate struct v4l2_ext_plane
>> - Fix some bugs caught by v4l2-compliance
>> - Rebased on top of media/master (post 5.8-rc1)
>>
>> Changes in v3:
>> - Rebased on top of media/master (post 5.4-rc1)
>>
>> Changes in v2:
>> - Add reserved space to v4l2_ext_buffer so that new fields can be added
>>   later on
>> ---
>>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
>>  drivers/media/v4l2-core/v4l2-ioctl.c | 349 +++++++++++++++++++++++++--
>>  include/media/v4l2-ioctl.h           |  26 ++
>>  include/uapi/linux/videodev2.h       |  89 +++++++
>>  4 files changed, 471 insertions(+), 22 deletions(-)
>>
> 
> <cut>
> 
>> +/**
>> + * struct v4l2_ext_plane - extended plane buffer info
>> + * @buffer_length:	size of the entire buffer in bytes, should fit
>> + *			@offset + @plane_length
>> + * @plane_length:	size of the plane in bytes.
>> + * @userptr:		when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
>> + *			to this plane.
>> + * @dmabuf_fd:		when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
>> + *			associated with this plane.
>> + * @offset:		offset in the memory buffer where the plane starts. If
>> + *			V4L2_MEMORY_MMAP is used, then it can be a "cookie" that
>> + *			should be passed to mmap() called on the video node.
>> + * @reserved:		extra space reserved for future fields, must be set to 0.
>> + *
>> + *
>> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
>> + * can have one plane for Y, and another for interleaved CbCr components.
>> + * Each plane can reside in a separate memory buffer, or even in
>> + * a completely separate memory node (e.g. in embedded devices).
>> + */
>> +struct v4l2_ext_plane {
>> +	__u32 buffer_length;
>> +	__u32 plane_length;
>> +	union {
>> +		__u64 userptr;
>> +		__s32 dmabuf_fd;
>> +	} m;
>> +	__u32 offset;
>> +	__u32 reserved[4];
>> +};
>> +
>>  /**
>>   * struct v4l2_buffer - video buffer info
>>   * @index:	id number of the buffer
>> @@ -1055,6 +1086,36 @@ struct v4l2_buffer {
>>  	};
>>  };
>>  
>> +/**
>> + * struct v4l2_ext_buffer - extended video buffer info
>> + * @index:	id number of the buffer
>> + * @type:	V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
>> + * @flags:	buffer informational flags
>> + * @field:	enum v4l2_field; field order of the image in the buffer
>> + * @timestamp:	frame timestamp
>> + * @sequence:	sequence count of this frame
>> + * @memory:	enum v4l2_memory; the method, in which the actual video data is
>> + *		passed
>> + * @planes:	per-plane buffer information
>> + * @request_fd:	fd of the request that this buffer should use
>> + * @reserved:	extra space reserved for future fields, must be set to 0
>> + *
>> + * Contains data exchanged by application and driver using one of the Streaming
>> + * I/O methods.
>> + */
>> +struct v4l2_ext_buffer {
>> +	__u32 index;
>> +	__u32 type;
>> +	__u32 flags;
>> +	__u32 field;
>> +	__u64 timestamp;
>> +	__u32 sequence;
>> +	__u32 memory;
>> +	__u32 request_fd;
> 
> This should be __s32, at least for consistency with dmabuf_fd?

I see that in struct v4l2_buffer, we have __s32, I don't mind changing it
to keep the consistency, I just don't see where this value can be a negative
number.

> 
>> +	struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
>> +	__u32 reserved[4];
> 
> I think we have to reserve more words here for future extensions.
> 
> I'd like also to propose to add here __s32 metadata_fd. The idea behind
> this is to have a way to pass per-frame metadata dmabuf buffers for
> synchronous type of metadata where the metadata is coming at the same
> time with data buffers. What would be the format of the metadata buffer
> is TBD.
> 
> One option for metadata buffer format could be:
> 
> header {
> 	num_ctrls
> 	array_of_ctrls [0..N]
> 		ctrl_id
> 		ctrl_size
> 		ctrl_offset
> }
> 
> data {
> 	cid0	//offset of cid0 in dmabuf buffer
> 	cid1
> 	cidN
> }

Would it be better if, instead of adding a medatata_fd inside struct v4l2_ext_buffer,
we create a new ioctl that gets this structs for the controls and sync them using the
Request API ?

I'd like to avoid too much metadata in the buffer object.

Regards,
Helen

> 
> This will make easy to get concrete ctrl id without a need to parse the
> whole metadata buffer. Also using dmabuf we don't need to copy data
> between userspace <-> kernelspace (just cache syncs through
> begin/end_cpu_access).
> 
> The open question is who will validate the metadata buffer when it comes
> from userspace. The obvious answer is v4l2-core but looking into DRM
> subsytem they give more freedom to the drivers, and just provide generic
> helpers which are not mandatory.
> 
> I guess this will be a voice in the wilderness but I wanted to know your
> opinion.
> 
>> +};
>> +
>>  #ifndef __KERNEL__
>>  /**
>>   * v4l2_timeval_to_ns - Convert timeval to nanoseconds
>> @@ -2520,6 +2581,29 @@ struct v4l2_create_buffers {
>>  	__u32			reserved[6];
>>  };
>>  
>> +/**
>> + * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
>> + * @index:	on return, index of the first created buffer
>> + * @count:	entry: number of requested buffers,
>> + *		return: number of created buffers
>> + * @memory:	enum v4l2_memory; buffer memory type
>> + * @capabilities: capabilities of this buffer type.
>> + * @format:	frame format, for which buffers are requested
>> + * @flags:	additional buffer management attributes (ignored unless the
>> + *		queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
>> + *		and configured for MMAP streaming I/O).
>> + * @reserved:	extra space reserved for future fields, must be set to 0
>> + */
>> +struct v4l2_ext_create_buffers {
>> +	__u32				index;
>> +	__u32				count;
>> +	__u32				memory;
>> +	struct v4l2_ext_pix_format	format;
>> +	__u32				capabilities;
>> +	__u32				flags;
>> +	__u32 reserved[4];
>> +};
>> +
>>  /*
>>   *	I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
>>   *
>> @@ -2623,6 +2707,11 @@ struct v4l2_create_buffers {
>>  #define VIDIOC_G_EXT_PIX_FMT	_IOWR('V', 104, struct v4l2_ext_pix_format)
>>  #define VIDIOC_S_EXT_PIX_FMT	_IOWR('V', 105, struct v4l2_ext_pix_format)
>>  #define VIDIOC_TRY_EXT_PIX_FMT	_IOWR('V', 106, struct v4l2_ext_pix_format)
>> +#define VIDIOC_EXT_CREATE_BUFS	_IOWR('V', 107, struct v4l2_ext_create_buffers)
>> +#define VIDIOC_EXT_QUERYBUF	_IOWR('V', 108, struct v4l2_ext_buffer)
>> +#define VIDIOC_EXT_QBUF		_IOWR('V', 109, struct v4l2_ext_buffer)
>> +#define VIDIOC_EXT_DQBUF	_IOWR('V', 110, struct v4l2_ext_buffer)
>> +#define VIDIOC_EXT_PREPARE_BUF	_IOWR('V', 111, struct v4l2_ext_buffer)
>>  
>>  /* Reminder: when adding new ioctls please add support for them to
>>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
>>
> 

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v4 0/6] media: v4l2: Add extended fmt and buffer ioctls
  2020-07-21 11:15 ` Boris Brezillon
@ 2020-07-21 13:56   ` Helen Koike
  0 siblings, 0 replies; 29+ messages in thread
From: Helen Koike @ 2020-07-21 13:56 UTC (permalink / raw)
  To: Boris Brezillon, Brian.Starkey
  Cc: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus,
	linux-media, tfiga, hiroh, nicolas, kernel, narmstrong,
	linux-kernel, frkoenig, mjourdan, stanimir.varbanov

Hi Boris,

On 7/21/20 8:15 AM, Boris Brezillon wrote:
> Hello Helen,
> 
> Just a few drive-by comments.
> 
> On Fri, 17 Jul 2020 08:54:29 -0300
> Helen Koike <helen.koike@collabora.com> wrote:
> 
>> Hi,
>>
>> I'm sorry for taking too long to submit v4.
>>
>> It is not perfect, not all v4l2-compliance tests passes, but I'd like a review,
>> specially on the API and potential problems, so I can focus on improving implementation
>> and maybe drop the RFC tag for next version.
>>
>> Follow below what changed in v4 and some items I'd like to discuss:
>>
>>
>> * Ioctl to replace v4l2_pix_format
>> ---------------------------------------------------------------------------------
>> During last media summit, we agreed to create ioctls that replace the v4l2_pix_format
>> struct and leave the other structs in the v4l2_format union alone.
>> Thus I refactored the code to receive struct v4l2_ext_pix_format, and I renamed the
>> ioctls, so now we have:
>>
>> int ioctl(int fd, VIDIOC_G_EXT_FMT, struct v4l2_ext_pix_format *argp);
> 
> Maybe use the EXT_PIX_FMT suffix here since the struct is really only
> about pixel formats.

Sorry, this is a copy&paste error, I'm already using this suffix in the code, except for the ioctls
that handle buffers (since they get v4l2_ext_buffer struct).

Regards,
Helen

> 
>> int ioctl(int fd, VIDIOC_S_EXT_FMT, struct v4l2_ext_pix_format *argp);
>> int ioctl(int fd, VIDIOC_TRY_EXT_FMT, struct v4l2_ext_pix_format *argp);
>>
>> The only valid types are V4L2_BUF_TYPE_VIDEO_CAPTURE and V4L2_BUF_TYPE_VIDEO_OUTPUT,
>> all the other types are invalid with this API.
>>
> 
> [...]
> 
>>
>>
>> Boris Brezillon (5):
>>   media: v4l2: Extend pixel formats to unify single/multi-planar
>>     handling (and more)
>>   media: videobuf2: Expose helpers to implement the _ext_fmt and
>>     _ext_buf hooks
>>   media: mediabus: Add helpers to convert a ext_pix format to/from a
>>     mbus_fmt
>>   media: vivid: Convert the capture and output drivers to
>>     EXT_FMT/EXT_BUF
>>   media: vimc: Implement the ext_fmt and ext_buf hooks
> 
> I think you should take ownership of these patches. The end result is
> likely to be completely different from what I initially posted, and
> you're the one doing the hard work here.
> 
>>
>> Hans Verkuil (1):
>>   media: v4l2: Add extended buffer operations
>>
>>  .../media/common/videobuf2/videobuf2-core.c   |   2 +
>>  .../media/common/videobuf2/videobuf2-v4l2.c   | 549 +++++-----
>>  .../media/test-drivers/vimc/vimc-capture.c    |  61 +-
>>  drivers/media/test-drivers/vimc/vimc-common.c |   6 +-
>>  drivers/media/test-drivers/vimc/vimc-common.h |   2 +-
>>  drivers/media/test-drivers/vivid/vivid-core.c |  70 +-
>>  .../test-drivers/vivid/vivid-touch-cap.c      |  26 +-
>>  .../test-drivers/vivid/vivid-touch-cap.h      |   3 +-
>>  .../media/test-drivers/vivid/vivid-vid-cap.c  | 169 +---
>>  .../media/test-drivers/vivid/vivid-vid-cap.h  |  15 +-
>>  .../media/test-drivers/vivid/vivid-vid-out.c  | 193 ++--
>>  .../media/test-drivers/vivid/vivid-vid-out.h  |  15 +-
>>  drivers/media/v4l2-core/v4l2-dev.c            |  50 +-
>>  drivers/media/v4l2-core/v4l2-ioctl.c          | 934 ++++++++++++++++--
>>  include/media/v4l2-ioctl.h                    |  60 ++
>>  include/media/v4l2-mediabus.h                 |  42 +
>>  include/media/videobuf2-core.h                |   6 +-
>>  include/media/videobuf2-v4l2.h                |  21 +-
>>  include/uapi/linux/videodev2.h                | 144 +++
>>  19 files changed, 1650 insertions(+), 718 deletions(-)
>>
> 

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v4 0/6] media: v4l2: Add extended fmt and buffer ioctls
  2020-07-21 10:24 ` Hans Verkuil
@ 2020-07-21 14:23   ` Helen Koike
  2020-07-21 14:34     ` Hans Verkuil
  0 siblings, 1 reply; 29+ messages in thread
From: Helen Koike @ 2020-07-21 14:23 UTC (permalink / raw)
  To: Hans Verkuil, mchehab, hans.verkuil, laurent.pinchart,
	sakari.ailus, linux-media
  Cc: tfiga, hiroh, nicolas, Brian.Starkey, kernel, boris.brezillon,
	narmstrong, linux-kernel, frkoenig, mjourdan, stanimir.varbanov

Hi,

On 7/21/20 7:24 AM, Hans Verkuil wrote:
> On 17/07/2020 13:54, Helen Koike wrote:
>> Hi,
>>
>> I'm sorry for taking too long to submit v4.
>>
>> It is not perfect, not all v4l2-compliance tests passes, but I'd like a review,
>> specially on the API and potential problems, so I can focus on improving implementation
>> and maybe drop the RFC tag for next version.
>>
>> Follow below what changed in v4 and some items I'd like to discuss:
>>
>>
>> * Ioctl to replace v4l2_pix_format
>> ---------------------------------------------------------------------------------
>> During last media summit, we agreed to create ioctls that replace the v4l2_pix_format
>> struct and leave the other structs in the v4l2_format union alone.
>> Thus I refactored the code to receive struct v4l2_ext_pix_format, and I renamed the
>> ioctls, so now we have:
>>
>> int ioctl(int fd, VIDIOC_G_EXT_FMT, struct v4l2_ext_pix_format *argp);
>> int ioctl(int fd, VIDIOC_S_EXT_FMT, struct v4l2_ext_pix_format *argp);
>> int ioctl(int fd, VIDIOC_TRY_EXT_FMT, struct v4l2_ext_pix_format *argp);
>>
>> The only valid types are V4L2_BUF_TYPE_VIDEO_CAPTURE and V4L2_BUF_TYPE_VIDEO_OUTPUT,
>> all the other types are invalid with this API.
>>
>>
>> * Modifiers
>> ---------------------------------------------------------------------------------
>> I understand that unifying DRM and V4L2 pixel formats is not possible, but I'd like
>> to unify the modifiers [1].
>>
>> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/drm/drm_fourcc.h#n290
>>
>> Should we use the DRM modifiers directly in the V4L2 API?
> 
> For now, yes. Most of the modifier work is done in DRM, it is only fairly recent
> that the media subsystem starts to have a need for it. So for now just use the drm
> header and prefixes.

ack

> 
>> Or should we move this header to a common place and change the prefix? (which requires
>> us to sync with DRM community).
>> Or should we create a v4l2 header, defining V4L2_ prefixed macros mapping to DRM_
>> macros?
>>
>> For now, patch 1/6 includes drm/drm_fourcc.h and it is using DRM_FORMAT_MOD_*
>>
>> As discussed before, It would be nice to have documentation describing DRM fourcc
>> equivalents (I'm not sure if someone started this already), listing the number of
>> planes per format.
>>
>> We should also document which pixelformats are valid for the EXT_API, since multiplanar
>> and tile versions like V4L2_PIX_FMT_NV12MT_16X16 (which seems equivalent to
>> DRM_FORMAT_MOD_SAMSUNG_16_16_TILE, and could have a more generic name) should be
>> replaced by a modifier.
>>
>> Regarding flags [2] field in struct v4l2_pix_format_mplane [3]:
>> The only defined flag is V4L2_PIX_FMT_FLAG_PREMUL_ALPHA, and it is only used by vsp1 driver.
>> Which I believe could be replaced by a modifier, to avoid another field that changes
>> pixel formats, so I removed it from the EXT API (we can always add it back later with
>> the reserved fields).
> 
> The colorspace series that Dafna is working on will add a V4L2_PIX_FMT_FLAG_SET_CSC
> flag, so this flags field will be needed.

This was because the CSC fields were defined in the API as read only (filled by the driver),
what if those fields in struct v4l2_ext_pix_format allows user to change the CSC fields,
and it will just fill the right one if it is not supported (similar to how other fields works
already).
Please, let me know if I'm missing something.

> 
>>
>> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/pixfmt-reserved.html#format-flags
>> [3] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/pixfmt-v4l2-mplane.html?highlight=v4l2_pix_format_mplane#c.v4l2_pix_format_mplane
>>
>> We also discussed to add a new ENUM_FMT_EXT ioctl to return all pixelformats + modifiers
>> combinations. I still didn't add it in this version, but I don't think it affects
>> what is in this RFC and it can be added later.
>>
>>
>> * Buffers/Plane offset
>> ---------------------------------------------------------------------------------
>>
>> My understanding is that inside a memory buffer we can have multiple planes in random
>> offsets.
>> I was comparing with the DRM API [4], where it can have the same dmabuf for multiple
>> planes in different offsets, and I started to think we could simplify our API, so
>> I took the liberty to do some more changes, please review struct v4l2_ext_plane in
>> this RFC.
>>
>> I removed the data_offset, since it is unused (See Laurent's RFC repurposing this
>> field [5]). And comparing to the DRM API, it seems to me we only need a single offset
>> field.
>>
>> We could also check about overlapping planes in a memory buffer, but this is complicated
>> if we use the same memory buffer with different v4l2_ext_buffer objects. We can also leave
>> to the driver to check situations that may cause HW errors.
>>
>> [4] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/drm/drm_mode.h#n489
>> [5] https://patchwork.linuxtv.org/patch/29177/
>>
>>
>> * Multistream Channels
>> ---------------------------------------------------------------------------------
>> During last media summit, we discussed about adding a channel number to the API to
>> support multistreams. i.e, to have multiple queues through a single video node.
>>
>> Use cases:
>>
>>     - Blitters: can take multiple streams as input, which would require multiple OUTPUT queues.
>>
>>     As Nicolas was explaining me:
>>     "The blitters comes with a lot of variation between hardware. Most blitters at
>>     least support 3 frames buffer. 2 inputs and one output. The second input is usually
>>     optional, as the output buffer data is not always overwritten (e.g. SRC_OVER
>>     blend or 1 input). Some of them have additional solid color or pattern that can
>>     be used too. Advanced blitters will have composition feature, and may support more
>>     input buffers to reduce the added latency that would be normally done through cascading
>>     the operations. Note that each input can have different size and different cropping
>>     region. Many blitters can scale and render to a sub-region of the CAPTURE buffer."
>>
>>     - Multis-calers: can produce multiple streams, which would require multiple CAPTURE queues.
>>
>>     As Nicolas was explaining me:
>>     "This type of HW (or soft IP) is commonly found on HW used to produce internet
>>     streams for fragmented and scalable protocols (HLS, DASH).  Basically they are
>>     used to transform one stream into multiple sized streams prior from being encoded."
>>
>> Modeling as channels allows the API to have synchronized Start/Stop between queues,
>> and also avoid the complexity of using the Media API in a topology with multiple video
>> nodes, which complicates userspace.
>>
>> This requires adding a new channel id in ioctls for formats (G_FMT/S_FMT/TRY_FMT), and
>> also for buffers (QBUF/DBUF).
>> We also need a mechanism to enumerate channels and their properties.
>> Since we don't have a clear view how this would work, for now I'm leaving reserved bits
>> in the structs, so we can add them later.
>>
>>
>> * Timecode
>> ---------------------------------------------------------------------------------
>> During last media summit, we discussed to return the v4l2_timecode field to the API,
>> since Nicolas mentioned that, even if it is not used by any upstreamed driver, it
>> is used by out-of-tree drivers.
>>
>> I've been discussing with Nicolas about this, and we can avoid adding too many metadata
>> to the buffer struct by using the Read-Only Request API [6] for retrieving more information
>> when required, similar to HDR.
>>
>> The RO Request API has the ability to read a control using a request that has already
>> completed, the control value lives as long as the request object. If it's not read
>> (or if there was no request), the data is simply ignored/discard.
>>
>> Since no upstream driver uses the timecode field, there are no conversions that need
>> to be done.
> 
> That's a reasonable solution.
> 
>>
>> [6] https://patchwork.kernel.org/cover/11635927/
>>
>>
>> * Other changes (and some questions) in this version:
>> ---------------------------------------------------------------------------------
>> - Added reserved fields to struct
>>
>> - The only difference between previously proposed VIDIOC_EXT_EXPBUF and VIDIOC_EXPBUF,
>> was that with VIDIOC_EXT_EXPBUF we can export multiple planes at once. I think we
>> can add this later, so I removed it from this RFC to simplify it.
>>
>> - v4l2_buffer [7] has a memory field (enum v4l2_memory [8]). We kept this field in
>> struct v4l2_ext_buffer, buf I was wondering if this shouldn't be in struct v4l2_ext_plane
>> instead.
> 
> This pops up every so often. The only use-case I can think of is when you return both
> video planes and metadata planes where the metadata might be MMAP and the video planes
> DMABUF. But it would add quite a bit of complexity, I suspect.

We could move it to struct v4l2_ext_plane to not limit the API, but for now, only allowing
a single memory type for all planes. So we don't need to extend the struct later if we see
a need for this. What do you think?

> 
>>
>> [7] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/buffer.html?highlight=v4l2_buffer#c.v4l2_buffer
>> [8] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/buffer.html?highlight=v4l2_memory#c.v4l2_memory
>>
>> - In struct v4l2_ext_pix_format, we have:
>>
>>         struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];
>>
>> The number of planes can be deducted from plane_fmt[i].sizeimage != 0, so I removed
>> the num_planes field. Please let me know if we can't use sizeimage for this.
>> In DRM, we know the number of planes from drm_mode_fb_cmd2 by the number of handle
>> args passed which are not 0.
>> This also avoids num_planes to be bigger then VIDEO_MAX_PLANES.
> 
> I have no objection to this. You do probably need to add a note about there not
> being holes, e.g. plane_fmt[0].sizeimage is != 0, so is plane_fmt[2].sizeimage,
> but plane_fmt[1].sizeimage == 0. That's likely something you don't want.

ack


Regards,
Helen

> 
> Regards,
> 
> 	Hans
> 
>>
>> - Added flags field to struct v4l2_ext_create_buffers
>>
>>
>> * Fixed bugs here and there
>> ---------------------------------------------------------------------------------
>> I fixed some bugs found with v4l2-compliance (not all of them yet),
>> through script v4l-utils/contrib/test/test-media.
>>
>> I adapted what Boris did for v4l-utils in previous version to this version:
>> https://gitlab.collabora.com/koike/v4l-utils/-/tree/ext-api/wip
>>
>> Boris' questions regarding DMABUF in last version still holds [9].
>>
>> [9] https://patchwork.linuxtv.org/project/linux-media/cover/20191008091119.7294-1-boris.brezillon@collabora.com/
>>
>>
>> Please, let me know your feedback,
>> Helen
>>
>>
>> Boris Brezillon (5):
>>   media: v4l2: Extend pixel formats to unify single/multi-planar
>>     handling (and more)
>>   media: videobuf2: Expose helpers to implement the _ext_fmt and
>>     _ext_buf hooks
>>   media: mediabus: Add helpers to convert a ext_pix format to/from a
>>     mbus_fmt
>>   media: vivid: Convert the capture and output drivers to
>>     EXT_FMT/EXT_BUF
>>   media: vimc: Implement the ext_fmt and ext_buf hooks
>>
>> Hans Verkuil (1):
>>   media: v4l2: Add extended buffer operations
>>
>>  .../media/common/videobuf2/videobuf2-core.c   |   2 +
>>  .../media/common/videobuf2/videobuf2-v4l2.c   | 549 +++++-----
>>  .../media/test-drivers/vimc/vimc-capture.c    |  61 +-
>>  drivers/media/test-drivers/vimc/vimc-common.c |   6 +-
>>  drivers/media/test-drivers/vimc/vimc-common.h |   2 +-
>>  drivers/media/test-drivers/vivid/vivid-core.c |  70 +-
>>  .../test-drivers/vivid/vivid-touch-cap.c      |  26 +-
>>  .../test-drivers/vivid/vivid-touch-cap.h      |   3 +-
>>  .../media/test-drivers/vivid/vivid-vid-cap.c  | 169 +---
>>  .../media/test-drivers/vivid/vivid-vid-cap.h  |  15 +-
>>  .../media/test-drivers/vivid/vivid-vid-out.c  | 193 ++--
>>  .../media/test-drivers/vivid/vivid-vid-out.h  |  15 +-
>>  drivers/media/v4l2-core/v4l2-dev.c            |  50 +-
>>  drivers/media/v4l2-core/v4l2-ioctl.c          | 934 ++++++++++++++++--
>>  include/media/v4l2-ioctl.h                    |  60 ++
>>  include/media/v4l2-mediabus.h                 |  42 +
>>  include/media/videobuf2-core.h                |   6 +-
>>  include/media/videobuf2-v4l2.h                |  21 +-
>>  include/uapi/linux/videodev2.h                | 144 +++
>>  19 files changed, 1650 insertions(+), 718 deletions(-)
>>
> 

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v4 2/6] media: v4l2: Add extended buffer operations
  2020-07-21 13:54     ` Helen Koike
@ 2020-07-21 14:30       ` Stanimir Varbanov
  2020-07-21 14:40         ` Helen Koike
  0 siblings, 1 reply; 29+ messages in thread
From: Stanimir Varbanov @ 2020-07-21 14:30 UTC (permalink / raw)
  To: Helen Koike, mchehab, hans.verkuil, laurent.pinchart,
	sakari.ailus, linux-media
  Cc: Boris Brezillon, tfiga, hiroh, nicolas, Brian.Starkey, kernel,
	narmstrong, linux-kernel, frkoenig, mjourdan

Hi Helen,

On 7/21/20 4:54 PM, Helen Koike wrote:
> Hi,
> 
> On 7/21/20 8:26 AM, Stanimir Varbanov wrote:
>>
>>
>> On 7/17/20 2:54 PM, Helen Koike wrote:
>>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>>
>>> Those extended buffer ops have several purpose:
>>> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
>>>    the number of ns elapsed since 1970
>>> 2/ Unify single/multiplanar handling
>>> 3/ Add a new start offset field to each v4l2 plane buffer info struct
>>>    to support the case where a single buffer object is storing all
>>>    planes data, each one being placed at a different offset
>>>
>>> New hooks are created in v4l2_ioctl_ops so that drivers can start using
>>> these new objects.
>>>
>>> The core takes care of converting new ioctls requests to old ones
>>> if the driver does not support the new hooks, and vice versa.
>>>
>>> Note that the timecode field is gone, since there doesn't seem to be
>>> in-kernel users. We can be added back in the reserved area if needed or
>>> use the Request API to collect more metadata information from the
>>> frame.
>>>
>>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>> ---
>>> Changes in v4:
>>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
>>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
>>> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
>>> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
>>> I think we can add this later, so I removed it from this RFC to simplify it.
>>> - Remove num_planes field from struct v4l2_ext_buffer
>>> - Add flags field to struct v4l2_ext_create_buffers
>>> - Reformulate struct v4l2_ext_plane
>>> - Fix some bugs caught by v4l2-compliance
>>> - Rebased on top of media/master (post 5.8-rc1)
>>>
>>> Changes in v3:
>>> - Rebased on top of media/master (post 5.4-rc1)
>>>
>>> Changes in v2:
>>> - Add reserved space to v4l2_ext_buffer so that new fields can be added
>>>   later on
>>> ---
>>>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
>>>  drivers/media/v4l2-core/v4l2-ioctl.c | 349 +++++++++++++++++++++++++--
>>>  include/media/v4l2-ioctl.h           |  26 ++
>>>  include/uapi/linux/videodev2.h       |  89 +++++++
>>>  4 files changed, 471 insertions(+), 22 deletions(-)
>>>
>>
>> <cut>
>>
>>> +/**
>>> + * struct v4l2_ext_plane - extended plane buffer info
>>> + * @buffer_length:	size of the entire buffer in bytes, should fit
>>> + *			@offset + @plane_length
>>> + * @plane_length:	size of the plane in bytes.
>>> + * @userptr:		when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
>>> + *			to this plane.
>>> + * @dmabuf_fd:		when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
>>> + *			associated with this plane.
>>> + * @offset:		offset in the memory buffer where the plane starts. If
>>> + *			V4L2_MEMORY_MMAP is used, then it can be a "cookie" that
>>> + *			should be passed to mmap() called on the video node.
>>> + * @reserved:		extra space reserved for future fields, must be set to 0.
>>> + *
>>> + *
>>> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
>>> + * can have one plane for Y, and another for interleaved CbCr components.
>>> + * Each plane can reside in a separate memory buffer, or even in
>>> + * a completely separate memory node (e.g. in embedded devices).
>>> + */
>>> +struct v4l2_ext_plane {
>>> +	__u32 buffer_length;
>>> +	__u32 plane_length;
>>> +	union {
>>> +		__u64 userptr;
>>> +		__s32 dmabuf_fd;
>>> +	} m;
>>> +	__u32 offset;
>>> +	__u32 reserved[4];
>>> +};
>>> +
>>>  /**
>>>   * struct v4l2_buffer - video buffer info
>>>   * @index:	id number of the buffer
>>> @@ -1055,6 +1086,36 @@ struct v4l2_buffer {
>>>  	};
>>>  };
>>>  
>>> +/**
>>> + * struct v4l2_ext_buffer - extended video buffer info
>>> + * @index:	id number of the buffer
>>> + * @type:	V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
>>> + * @flags:	buffer informational flags
>>> + * @field:	enum v4l2_field; field order of the image in the buffer
>>> + * @timestamp:	frame timestamp
>>> + * @sequence:	sequence count of this frame
>>> + * @memory:	enum v4l2_memory; the method, in which the actual video data is
>>> + *		passed
>>> + * @planes:	per-plane buffer information
>>> + * @request_fd:	fd of the request that this buffer should use
>>> + * @reserved:	extra space reserved for future fields, must be set to 0
>>> + *
>>> + * Contains data exchanged by application and driver using one of the Streaming
>>> + * I/O methods.
>>> + */
>>> +struct v4l2_ext_buffer {
>>> +	__u32 index;
>>> +	__u32 type;
>>> +	__u32 flags;
>>> +	__u32 field;
>>> +	__u64 timestamp;
>>> +	__u32 sequence;
>>> +	__u32 memory;
>>> +	__u32 request_fd;
>>
>> This should be __s32, at least for consistency with dmabuf_fd?
> 
> I see that in struct v4l2_buffer, we have __s32, I don't mind changing it
> to keep the consistency, I just don't see where this value can be a negative
> number.

here
https://elixir.bootlin.com/linux/v5.8-rc4/source/drivers/media/common/videobuf2/videobuf2-v4l2.c#L134

> 
>>
>>> +	struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
>>> +	__u32 reserved[4];
>>
>> I think we have to reserve more words here for future extensions.
>>
>> I'd like also to propose to add here __s32 metadata_fd. The idea behind
>> this is to have a way to pass per-frame metadata dmabuf buffers for
>> synchronous type of metadata where the metadata is coming at the same
>> time with data buffers. What would be the format of the metadata buffer
>> is TBD.
>>
>> One option for metadata buffer format could be:
>>
>> header {
>> 	num_ctrls
>> 	array_of_ctrls [0..N]
>> 		ctrl_id
>> 		ctrl_size
>> 		ctrl_offset
>> }
>>
>> data {
>> 	cid0	//offset of cid0 in dmabuf buffer
>> 	cid1
>> 	cidN
>> }
> 
> Would it be better if, instead of adding a medatata_fd inside struct v4l2_ext_buffer,
> we create a new ioctl that gets this structs for the controls and sync them using the
> Request API ?

no, this solution has performance drawbacks when the metadata is big,
think of 64K.

> 
> I'd like to avoid too much metadata in the buffer object.
> 
> Regards,
> Helen
> 
>>
>> This will make easy to get concrete ctrl id without a need to parse the
>> whole metadata buffer. Also using dmabuf we don't need to copy data
>> between userspace <-> kernelspace (just cache syncs through
>> begin/end_cpu_access).
>>
>> The open question is who will validate the metadata buffer when it comes
>> from userspace. The obvious answer is v4l2-core but looking into DRM
>> subsytem they give more freedom to the drivers, and just provide generic
>> helpers which are not mandatory.
>>
>> I guess this will be a voice in the wilderness but I wanted to know your
>> opinion.
>>
>>> +};
>>> +
>>>  #ifndef __KERNEL__
>>>  /**
>>>   * v4l2_timeval_to_ns - Convert timeval to nanoseconds
>>> @@ -2520,6 +2581,29 @@ struct v4l2_create_buffers {
>>>  	__u32			reserved[6];
>>>  };
>>>  
>>> +/**
>>> + * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
>>> + * @index:	on return, index of the first created buffer
>>> + * @count:	entry: number of requested buffers,
>>> + *		return: number of created buffers
>>> + * @memory:	enum v4l2_memory; buffer memory type
>>> + * @capabilities: capabilities of this buffer type.
>>> + * @format:	frame format, for which buffers are requested
>>> + * @flags:	additional buffer management attributes (ignored unless the
>>> + *		queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
>>> + *		and configured for MMAP streaming I/O).
>>> + * @reserved:	extra space reserved for future fields, must be set to 0
>>> + */
>>> +struct v4l2_ext_create_buffers {
>>> +	__u32				index;
>>> +	__u32				count;
>>> +	__u32				memory;
>>> +	struct v4l2_ext_pix_format	format;
>>> +	__u32				capabilities;
>>> +	__u32				flags;
>>> +	__u32 reserved[4];
>>> +};
>>> +
>>>  /*
>>>   *	I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
>>>   *
>>> @@ -2623,6 +2707,11 @@ struct v4l2_create_buffers {
>>>  #define VIDIOC_G_EXT_PIX_FMT	_IOWR('V', 104, struct v4l2_ext_pix_format)
>>>  #define VIDIOC_S_EXT_PIX_FMT	_IOWR('V', 105, struct v4l2_ext_pix_format)
>>>  #define VIDIOC_TRY_EXT_PIX_FMT	_IOWR('V', 106, struct v4l2_ext_pix_format)
>>> +#define VIDIOC_EXT_CREATE_BUFS	_IOWR('V', 107, struct v4l2_ext_create_buffers)
>>> +#define VIDIOC_EXT_QUERYBUF	_IOWR('V', 108, struct v4l2_ext_buffer)
>>> +#define VIDIOC_EXT_QBUF		_IOWR('V', 109, struct v4l2_ext_buffer)
>>> +#define VIDIOC_EXT_DQBUF	_IOWR('V', 110, struct v4l2_ext_buffer)
>>> +#define VIDIOC_EXT_PREPARE_BUF	_IOWR('V', 111, struct v4l2_ext_buffer)
>>>  
>>>  /* Reminder: when adding new ioctls please add support for them to
>>>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
>>>
>>

-- 
regards,
Stan

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v4 0/6] media: v4l2: Add extended fmt and buffer ioctls
  2020-07-21 14:23   ` Helen Koike
@ 2020-07-21 14:34     ` Hans Verkuil
  0 siblings, 0 replies; 29+ messages in thread
From: Hans Verkuil @ 2020-07-21 14:34 UTC (permalink / raw)
  To: Helen Koike, mchehab, hans.verkuil, laurent.pinchart,
	sakari.ailus, linux-media
  Cc: tfiga, hiroh, nicolas, Brian.Starkey, kernel, boris.brezillon,
	narmstrong, linux-kernel, frkoenig, mjourdan, stanimir.varbanov

On 21/07/2020 16:23, Helen Koike wrote:
> Hi,
> 
> On 7/21/20 7:24 AM, Hans Verkuil wrote:
>> On 17/07/2020 13:54, Helen Koike wrote:
>>> Hi,
>>>
>>> I'm sorry for taking too long to submit v4.
>>>
>>> It is not perfect, not all v4l2-compliance tests passes, but I'd like a review,
>>> specially on the API and potential problems, so I can focus on improving implementation
>>> and maybe drop the RFC tag for next version.
>>>
>>> Follow below what changed in v4 and some items I'd like to discuss:
>>>
>>>
>>> * Ioctl to replace v4l2_pix_format
>>> ---------------------------------------------------------------------------------
>>> During last media summit, we agreed to create ioctls that replace the v4l2_pix_format
>>> struct and leave the other structs in the v4l2_format union alone.
>>> Thus I refactored the code to receive struct v4l2_ext_pix_format, and I renamed the
>>> ioctls, so now we have:
>>>
>>> int ioctl(int fd, VIDIOC_G_EXT_FMT, struct v4l2_ext_pix_format *argp);
>>> int ioctl(int fd, VIDIOC_S_EXT_FMT, struct v4l2_ext_pix_format *argp);
>>> int ioctl(int fd, VIDIOC_TRY_EXT_FMT, struct v4l2_ext_pix_format *argp);
>>>
>>> The only valid types are V4L2_BUF_TYPE_VIDEO_CAPTURE and V4L2_BUF_TYPE_VIDEO_OUTPUT,
>>> all the other types are invalid with this API.
>>>
>>>
>>> * Modifiers
>>> ---------------------------------------------------------------------------------
>>> I understand that unifying DRM and V4L2 pixel formats is not possible, but I'd like
>>> to unify the modifiers [1].
>>>
>>> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/drm/drm_fourcc.h#n290
>>>
>>> Should we use the DRM modifiers directly in the V4L2 API?
>>
>> For now, yes. Most of the modifier work is done in DRM, it is only fairly recent
>> that the media subsystem starts to have a need for it. So for now just use the drm
>> header and prefixes.
> 
> ack
> 
>>
>>> Or should we move this header to a common place and change the prefix? (which requires
>>> us to sync with DRM community).
>>> Or should we create a v4l2 header, defining V4L2_ prefixed macros mapping to DRM_
>>> macros?
>>>
>>> For now, patch 1/6 includes drm/drm_fourcc.h and it is using DRM_FORMAT_MOD_*
>>>
>>> As discussed before, It would be nice to have documentation describing DRM fourcc
>>> equivalents (I'm not sure if someone started this already), listing the number of
>>> planes per format.
>>>
>>> We should also document which pixelformats are valid for the EXT_API, since multiplanar
>>> and tile versions like V4L2_PIX_FMT_NV12MT_16X16 (which seems equivalent to
>>> DRM_FORMAT_MOD_SAMSUNG_16_16_TILE, and could have a more generic name) should be
>>> replaced by a modifier.
>>>
>>> Regarding flags [2] field in struct v4l2_pix_format_mplane [3]:
>>> The only defined flag is V4L2_PIX_FMT_FLAG_PREMUL_ALPHA, and it is only used by vsp1 driver.
>>> Which I believe could be replaced by a modifier, to avoid another field that changes
>>> pixel formats, so I removed it from the EXT API (we can always add it back later with
>>> the reserved fields).
>>
>> The colorspace series that Dafna is working on will add a V4L2_PIX_FMT_FLAG_SET_CSC
>> flag, so this flags field will be needed.
> 
> This was because the CSC fields were defined in the API as read only (filled by the driver),
> what if those fields in struct v4l2_ext_pix_format allows user to change the CSC fields,
> and it will just fill the right one if it is not supported (similar to how other fields works
> already).
> Please, let me know if I'm missing something.

Ah, that's true, I forgot about that.

> 
>>
>>>
>>> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/pixfmt-reserved.html#format-flags
>>> [3] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/pixfmt-v4l2-mplane.html?highlight=v4l2_pix_format_mplane#c.v4l2_pix_format_mplane
>>>
>>> We also discussed to add a new ENUM_FMT_EXT ioctl to return all pixelformats + modifiers
>>> combinations. I still didn't add it in this version, but I don't think it affects
>>> what is in this RFC and it can be added later.
>>>
>>>
>>> * Buffers/Plane offset
>>> ---------------------------------------------------------------------------------
>>>
>>> My understanding is that inside a memory buffer we can have multiple planes in random
>>> offsets.
>>> I was comparing with the DRM API [4], where it can have the same dmabuf for multiple
>>> planes in different offsets, and I started to think we could simplify our API, so
>>> I took the liberty to do some more changes, please review struct v4l2_ext_plane in
>>> this RFC.
>>>
>>> I removed the data_offset, since it is unused (See Laurent's RFC repurposing this
>>> field [5]). And comparing to the DRM API, it seems to me we only need a single offset
>>> field.
>>>
>>> We could also check about overlapping planes in a memory buffer, but this is complicated
>>> if we use the same memory buffer with different v4l2_ext_buffer objects. We can also leave
>>> to the driver to check situations that may cause HW errors.
>>>
>>> [4] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/drm/drm_mode.h#n489
>>> [5] https://patchwork.linuxtv.org/patch/29177/
>>>
>>>
>>> * Multistream Channels
>>> ---------------------------------------------------------------------------------
>>> During last media summit, we discussed about adding a channel number to the API to
>>> support multistreams. i.e, to have multiple queues through a single video node.
>>>
>>> Use cases:
>>>
>>>     - Blitters: can take multiple streams as input, which would require multiple OUTPUT queues.
>>>
>>>     As Nicolas was explaining me:
>>>     "The blitters comes with a lot of variation between hardware. Most blitters at
>>>     least support 3 frames buffer. 2 inputs and one output. The second input is usually
>>>     optional, as the output buffer data is not always overwritten (e.g. SRC_OVER
>>>     blend or 1 input). Some of them have additional solid color or pattern that can
>>>     be used too. Advanced blitters will have composition feature, and may support more
>>>     input buffers to reduce the added latency that would be normally done through cascading
>>>     the operations. Note that each input can have different size and different cropping
>>>     region. Many blitters can scale and render to a sub-region of the CAPTURE buffer."
>>>
>>>     - Multis-calers: can produce multiple streams, which would require multiple CAPTURE queues.
>>>
>>>     As Nicolas was explaining me:
>>>     "This type of HW (or soft IP) is commonly found on HW used to produce internet
>>>     streams for fragmented and scalable protocols (HLS, DASH).  Basically they are
>>>     used to transform one stream into multiple sized streams prior from being encoded."
>>>
>>> Modeling as channels allows the API to have synchronized Start/Stop between queues,
>>> and also avoid the complexity of using the Media API in a topology with multiple video
>>> nodes, which complicates userspace.
>>>
>>> This requires adding a new channel id in ioctls for formats (G_FMT/S_FMT/TRY_FMT), and
>>> also for buffers (QBUF/DBUF).
>>> We also need a mechanism to enumerate channels and their properties.
>>> Since we don't have a clear view how this would work, for now I'm leaving reserved bits
>>> in the structs, so we can add them later.
>>>
>>>
>>> * Timecode
>>> ---------------------------------------------------------------------------------
>>> During last media summit, we discussed to return the v4l2_timecode field to the API,
>>> since Nicolas mentioned that, even if it is not used by any upstreamed driver, it
>>> is used by out-of-tree drivers.
>>>
>>> I've been discussing with Nicolas about this, and we can avoid adding too many metadata
>>> to the buffer struct by using the Read-Only Request API [6] for retrieving more information
>>> when required, similar to HDR.
>>>
>>> The RO Request API has the ability to read a control using a request that has already
>>> completed, the control value lives as long as the request object. If it's not read
>>> (or if there was no request), the data is simply ignored/discard.
>>>
>>> Since no upstream driver uses the timecode field, there are no conversions that need
>>> to be done.
>>
>> That's a reasonable solution.
>>
>>>
>>> [6] https://patchwork.kernel.org/cover/11635927/
>>>
>>>
>>> * Other changes (and some questions) in this version:
>>> ---------------------------------------------------------------------------------
>>> - Added reserved fields to struct
>>>
>>> - The only difference between previously proposed VIDIOC_EXT_EXPBUF and VIDIOC_EXPBUF,
>>> was that with VIDIOC_EXT_EXPBUF we can export multiple planes at once. I think we
>>> can add this later, so I removed it from this RFC to simplify it.
>>>
>>> - v4l2_buffer [7] has a memory field (enum v4l2_memory [8]). We kept this field in
>>> struct v4l2_ext_buffer, buf I was wondering if this shouldn't be in struct v4l2_ext_plane
>>> instead.
>>
>> This pops up every so often. The only use-case I can think of is when you return both
>> video planes and metadata planes where the metadata might be MMAP and the video planes
>> DMABUF. But it would add quite a bit of complexity, I suspect.
> 
> We could move it to struct v4l2_ext_plane to not limit the API, but for now, only allowing
> a single memory type for all planes. So we don't need to extend the struct later if we see
> a need for this. What do you think?

Go with that for now, yes.

Regards,

	Hans

> 
>>
>>>
>>> [7] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/buffer.html?highlight=v4l2_buffer#c.v4l2_buffer
>>> [8] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/buffer.html?highlight=v4l2_memory#c.v4l2_memory
>>>
>>> - In struct v4l2_ext_pix_format, we have:
>>>
>>>         struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];
>>>
>>> The number of planes can be deducted from plane_fmt[i].sizeimage != 0, so I removed
>>> the num_planes field. Please let me know if we can't use sizeimage for this.
>>> In DRM, we know the number of planes from drm_mode_fb_cmd2 by the number of handle
>>> args passed which are not 0.
>>> This also avoids num_planes to be bigger then VIDEO_MAX_PLANES.
>>
>> I have no objection to this. You do probably need to add a note about there not
>> being holes, e.g. plane_fmt[0].sizeimage is != 0, so is plane_fmt[2].sizeimage,
>> but plane_fmt[1].sizeimage == 0. That's likely something you don't want.
> 
> ack
> 
> 
> Regards,
> Helen
> 
>>
>> Regards,
>>
>> 	Hans
>>
>>>
>>> - Added flags field to struct v4l2_ext_create_buffers
>>>
>>>
>>> * Fixed bugs here and there
>>> ---------------------------------------------------------------------------------
>>> I fixed some bugs found with v4l2-compliance (not all of them yet),
>>> through script v4l-utils/contrib/test/test-media.
>>>
>>> I adapted what Boris did for v4l-utils in previous version to this version:
>>> https://gitlab.collabora.com/koike/v4l-utils/-/tree/ext-api/wip
>>>
>>> Boris' questions regarding DMABUF in last version still holds [9].
>>>
>>> [9] https://patchwork.linuxtv.org/project/linux-media/cover/20191008091119.7294-1-boris.brezillon@collabora.com/
>>>
>>>
>>> Please, let me know your feedback,
>>> Helen
>>>
>>>
>>> Boris Brezillon (5):
>>>   media: v4l2: Extend pixel formats to unify single/multi-planar
>>>     handling (and more)
>>>   media: videobuf2: Expose helpers to implement the _ext_fmt and
>>>     _ext_buf hooks
>>>   media: mediabus: Add helpers to convert a ext_pix format to/from a
>>>     mbus_fmt
>>>   media: vivid: Convert the capture and output drivers to
>>>     EXT_FMT/EXT_BUF
>>>   media: vimc: Implement the ext_fmt and ext_buf hooks
>>>
>>> Hans Verkuil (1):
>>>   media: v4l2: Add extended buffer operations
>>>
>>>  .../media/common/videobuf2/videobuf2-core.c   |   2 +
>>>  .../media/common/videobuf2/videobuf2-v4l2.c   | 549 +++++-----
>>>  .../media/test-drivers/vimc/vimc-capture.c    |  61 +-
>>>  drivers/media/test-drivers/vimc/vimc-common.c |   6 +-
>>>  drivers/media/test-drivers/vimc/vimc-common.h |   2 +-
>>>  drivers/media/test-drivers/vivid/vivid-core.c |  70 +-
>>>  .../test-drivers/vivid/vivid-touch-cap.c      |  26 +-
>>>  .../test-drivers/vivid/vivid-touch-cap.h      |   3 +-
>>>  .../media/test-drivers/vivid/vivid-vid-cap.c  | 169 +---
>>>  .../media/test-drivers/vivid/vivid-vid-cap.h  |  15 +-
>>>  .../media/test-drivers/vivid/vivid-vid-out.c  | 193 ++--
>>>  .../media/test-drivers/vivid/vivid-vid-out.h  |  15 +-
>>>  drivers/media/v4l2-core/v4l2-dev.c            |  50 +-
>>>  drivers/media/v4l2-core/v4l2-ioctl.c          | 934 ++++++++++++++++--
>>>  include/media/v4l2-ioctl.h                    |  60 ++
>>>  include/media/v4l2-mediabus.h                 |  42 +
>>>  include/media/videobuf2-core.h                |   6 +-
>>>  include/media/videobuf2-v4l2.h                |  21 +-
>>>  include/uapi/linux/videodev2.h                | 144 +++
>>>  19 files changed, 1650 insertions(+), 718 deletions(-)
>>>
>>


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v4 2/6] media: v4l2: Add extended buffer operations
  2020-07-21 10:48   ` Hans Verkuil
@ 2020-07-21 14:36     ` Helen Koike
  0 siblings, 0 replies; 29+ messages in thread
From: Helen Koike @ 2020-07-21 14:36 UTC (permalink / raw)
  To: Hans Verkuil, mchehab, hans.verkuil, laurent.pinchart,
	sakari.ailus, linux-media
  Cc: Boris Brezillon, tfiga, hiroh, nicolas, Brian.Starkey, kernel,
	narmstrong, linux-kernel, frkoenig, mjourdan, stanimir.varbanov

Hello,

On 7/21/20 7:48 AM, Hans Verkuil wrote:
> On 17/07/2020 13:54, Helen Koike wrote:
>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>
>> Those extended buffer ops have several purpose:
>> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
>>    the number of ns elapsed since 1970
>> 2/ Unify single/multiplanar handling
>> 3/ Add a new start offset field to each v4l2 plane buffer info struct
>>    to support the case where a single buffer object is storing all
>>    planes data, each one being placed at a different offset
>>
>> New hooks are created in v4l2_ioctl_ops so that drivers can start using
>> these new objects.
>>
>> The core takes care of converting new ioctls requests to old ones
>> if the driver does not support the new hooks, and vice versa.
>>
>> Note that the timecode field is gone, since there doesn't seem to be
>> in-kernel users. We can be added back in the reserved area if needed or
>> use the Request API to collect more metadata information from the
>> frame.
>>
>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>> ---
>> Changes in v4:
>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
>> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
>> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
>> I think we can add this later, so I removed it from this RFC to simplify it.
>> - Remove num_planes field from struct v4l2_ext_buffer
>> - Add flags field to struct v4l2_ext_create_buffers
>> - Reformulate struct v4l2_ext_plane
>> - Fix some bugs caught by v4l2-compliance
>> - Rebased on top of media/master (post 5.8-rc1)
>>
>> Changes in v3:
>> - Rebased on top of media/master (post 5.4-rc1)
>>
>> Changes in v2:
>> - Add reserved space to v4l2_ext_buffer so that new fields can be added
>>   later on
>> ---
>>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
>>  drivers/media/v4l2-core/v4l2-ioctl.c | 349 +++++++++++++++++++++++++--
>>  include/media/v4l2-ioctl.h           |  26 ++
>>  include/uapi/linux/videodev2.h       |  89 +++++++
>>  4 files changed, 471 insertions(+), 22 deletions(-)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
>> index e1829906bc086..cb21ee8eb075c 100644
>> --- a/drivers/media/v4l2-core/v4l2-dev.c
>> +++ b/drivers/media/v4l2-core/v4l2-dev.c
>> @@ -720,15 +720,34 @@ static void determine_valid_ioctls(struct video_device *vdev)
>>  		SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_sdr_out);
>>  	}
>>  
>> +	if (is_vid || is_tch) {
>> +		/* ioctls valid for video and touch */
>> +		if (ops->vidioc_querybuf || ops->vidioc_ext_querybuf)
>> +			set_bit(_IOC_NR(VIDIOC_EXT_QUERYBUF), valid_ioctls);
>> +		if (ops->vidioc_qbuf || ops->vidioc_ext_qbuf)
>> +			set_bit(_IOC_NR(VIDIOC_EXT_QBUF), valid_ioctls);
>> +		if (ops->vidioc_dqbuf || ops->vidioc_ext_dqbuf)
>> +			set_bit(_IOC_NR(VIDIOC_EXT_DQBUF), valid_ioctls);
>> +		if (ops->vidioc_create_bufs || ops->vidioc_ext_create_bufs)
>> +			set_bit(_IOC_NR(VIDIOC_EXT_CREATE_BUFS), valid_ioctls);
>> +		if (ops->vidioc_prepare_buf || ops->vidioc_ext_prepare_buf)
>> +			set_bit(_IOC_NR(VIDIOC_EXT_PREPARE_BUF), valid_ioctls);
>> +	}
>> +
>>  	if (is_vid || is_vbi || is_sdr || is_tch || is_meta) {
>>  		/* ioctls valid for video, vbi, sdr, touch and metadata */
>>  		SET_VALID_IOCTL(ops, VIDIOC_REQBUFS, vidioc_reqbufs);
>> -		SET_VALID_IOCTL(ops, VIDIOC_QUERYBUF, vidioc_querybuf);
>> -		SET_VALID_IOCTL(ops, VIDIOC_QBUF, vidioc_qbuf);
>>  		SET_VALID_IOCTL(ops, VIDIOC_EXPBUF, vidioc_expbuf);
>> -		SET_VALID_IOCTL(ops, VIDIOC_DQBUF, vidioc_dqbuf);
>> -		SET_VALID_IOCTL(ops, VIDIOC_CREATE_BUFS, vidioc_create_bufs);
>> -		SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, vidioc_prepare_buf);
>> +		if (ops->vidioc_querybuf || ops->vidioc_ext_querybuf)
>> +			set_bit(_IOC_NR(VIDIOC_QUERYBUF), valid_ioctls);
>> +		if (ops->vidioc_qbuf || ops->vidioc_ext_qbuf)
>> +			set_bit(_IOC_NR(VIDIOC_QBUF), valid_ioctls);
>> +		if (ops->vidioc_dqbuf || ops->vidioc_ext_dqbuf)
>> +			set_bit(_IOC_NR(VIDIOC_DQBUF), valid_ioctls);
>> +		if (ops->vidioc_create_bufs || ops->vidioc_ext_create_bufs)
>> +			set_bit(_IOC_NR(VIDIOC_CREATE_BUFS), valid_ioctls);
>> +		if (ops->vidioc_prepare_buf || ops->vidioc_ext_prepare_buf)
>> +			set_bit(_IOC_NR(VIDIOC_PREPARE_BUF), valid_ioctls);
>>  		SET_VALID_IOCTL(ops, VIDIOC_STREAMON, vidioc_streamon);
>>  		SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, vidioc_streamoff);
>>  	}
>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
>> index 3b77433f6c32b..5ddd57939c49c 100644
>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>> @@ -527,6 +527,26 @@ static void v4l_print_buffer(const void *arg, bool write_only)
>>  			tc->type, tc->flags, tc->frames, *(__u32 *)tc->userbits);
>>  }
>>  
>> +static void v4l_print_ext_buffer(const void *arg, bool write_only)
>> +{
>> +	const struct v4l2_ext_buffer *e = arg;
>> +	const struct v4l2_ext_plane *plane;
>> +	unsigned int i;
>> +
>> +	pr_cont("%lld index=%d, type=%s, flags=0x%08x, field=%s, sequence=%d, memory=%s\n",
>> +		e->timestamp, e->index, prt_names(e->type, v4l2_type_names),
>> +		e->flags, prt_names(e->field, v4l2_field_names),
>> +		e->sequence, prt_names(e->memory, v4l2_memory_names));
>> +
>> +	for (i = 0; i < VIDEO_MAX_PLANES &&
>> +		    e->planes[i].buffer_length; i++) {
>> +		plane = &e->planes[i];
>> +		pr_debug("plane %d: buffer_length=%d, plane_length=%d offset=0x%08x\n",
>> +			 i, plane->buffer_length, plane->plane_length,
>> +			 plane->offset);
>> +	}
>> +}
>> +
>>  static void v4l_print_exportbuffer(const void *arg, bool write_only)
>>  {
>>  	const struct v4l2_exportbuffer *p = arg;
>> @@ -546,6 +566,15 @@ static void v4l_print_create_buffers(const void *arg, bool write_only)
>>  	v4l_print_format(&p->format, write_only);
>>  }
>>  
>> +static void v4l_print_ext_create_buffers(const void *arg, bool write_only)
>> +{
>> +	const struct v4l2_ext_create_buffers *p = arg;
>> +
>> +	pr_cont("index=%d, count=%d, memory=%s, ", p->index, p->count,
>> +		prt_names(p->memory, v4l2_memory_names));
>> +	v4l_print_ext_pix_format(&p->format, write_only);
>> +}
>> +
>>  static void v4l_print_streamparm(const void *arg, bool write_only)
>>  {
>>  	const struct v4l2_streamparm *p = arg;
>> @@ -1214,6 +1243,139 @@ int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
>>  }
>>  EXPORT_SYMBOL_GPL(v4l2_format_to_ext_pix_format);
>>  
>> +/*
>> + * If mplane_cap is true, b->m.planes should have a valid pointer of a
>> + * struct v4l2_plane array, and b->length with its size
>> + */
>> +int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
>> +			      struct v4l2_buffer *b, bool mplane_cap)
>> +{
>> +	unsigned int planes_array_size = b->length;
>> +	struct v4l2_plane *planes = b->m.planes;
>> +	u64 nsecs;
>> +
>> +	if (!mplane_cap && e->planes[1].buffer_length != 0)
>> +		return -EINVAL;
>> +
>> +	memset(b, 0, sizeof(*b));
>> +
>> +	b->index = e->index;
>> +	b->flags = e->flags;
>> +	b->field = e->field;
>> +	b->sequence = e->sequence;
>> +	b->memory = e->memory;
>> +	b->request_fd = e->request_fd;
>> +	b->timestamp.tv_sec = div64_u64_rem(e->timestamp, NSEC_PER_SEC, &nsecs);
>> +	b->timestamp.tv_usec = (u32)nsecs / NSEC_PER_USEC;
>> +
>> +	if (mplane_cap) {
>> +		unsigned int i;
>> +
>> +		if (!planes || !planes_array_size)
>> +			return -EINVAL;
>> +
>> +		b->m.planes = planes;
>> +
>> +		if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
>> +			b->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
>> +		else
>> +			b->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
>> +
>> +		for (i = 0; i < VIDEO_MAX_PLANES && i < planes_array_size &&
>> +			    e->planes[i].buffer_length; i++) {
>> +
>> +			if (b->memory != V4L2_MEMORY_MMAP && e->planes[i].offset)
>> +				return -EINVAL;
>> +
>> +			memset(&b->m.planes[i], 0, sizeof(b->m.planes[i]));
>> +
>> +			if (b->memory == V4L2_MEMORY_MMAP)
>> +				b->m.planes[i].m.mem_offset = e->planes[i].offset;
>> +			else if (b->memory == V4L2_MEMORY_DMABUF)
>> +				b->m.planes[i].m.fd = e->planes[i].m.dmabuf_fd;
>> +			else
>> +				b->m.planes[i].m.userptr = e->planes[i].m.userptr;
>> +
>> +			b->m.planes[i].bytesused = e->planes[i].plane_length;
>> +			b->m.planes[i].length = e->planes[i].buffer_length;
>> +		}
>> +		/* In multi-planar, length contain the number of planes */
>> +		b->length = i;
>> +	} else {
>> +		b->type = e->type;
>> +		b->bytesused = e->planes[0].plane_length;
>> +		b->length = e->planes[0].buffer_length;
>> +
>> +		if (b->memory != V4L2_MEMORY_MMAP && e->planes[0].offset)
>> +			return -EINVAL;
>> +
>> +		if (b->memory == V4L2_MEMORY_MMAP)
>> +			b->m.offset = e->planes[0].offset;
>> +		else if (b->memory == V4L2_MEMORY_DMABUF)
>> +			b->m.fd = e->planes[0].m.dmabuf_fd;
>> +		else
>> +			b->m.userptr = e->planes[0].m.userptr;
>> +	}
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_ext_buffer_to_buffer);
>> +
>> +int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
>> +			      struct v4l2_ext_buffer *e)
>> +{
>> +	memset(e, 0, sizeof(*e));
>> +
>> +	e->index = b->index;
>> +	e->flags = b->flags;
>> +	e->field = b->field;
>> +	e->sequence = b->sequence;
>> +	e->memory = b->memory;
>> +	e->request_fd = b->request_fd;
>> +	e->timestamp = b->timestamp.tv_sec * NSEC_PER_SEC +
>> +		b->timestamp.tv_usec * NSEC_PER_USEC;
>> +	if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
>> +		unsigned int i;
>> +
>> +		if (!b->m.planes)
>> +			return -EINVAL;
>> +
>> +		if (b->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
>> +			e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
>> +		else
>> +			e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
>> +
>> +		/* In multi-planar, length contain the number of planes */
>> +		for (i = 0; i < b->length; i++) {
>> +			if (b->memory == V4L2_MEMORY_MMAP)
>> +				e->planes[i].offset = b->m.planes[i].m.mem_offset;
>> +			else if (b->memory == V4L2_MEMORY_DMABUF)
>> +				e->planes[i].m.dmabuf_fd = b->m.planes[i].m.fd;
>> +			else
>> +				e->planes[i].m.userptr = b->m.planes[i].m.userptr;
>> +
>> +			e->planes[i].buffer_length = b->m.planes[i].length;
>> +			e->planes[i].plane_length = b->m.planes[i].bytesused;
>> +			if (b->m.planes[i].data_offset)
>> +				pr_warn("Ignoring data_offset value %d\n",
>> +					b->m.planes[i].data_offset);
>> +		}
>> +	} else {
>> +		e->type = b->type;
>> +		e->planes[0].plane_length = b->bytesused;
>> +		e->planes[0].buffer_length = b->length;
>> +		if (b->memory == V4L2_MEMORY_MMAP)
>> +			e->planes[0].offset = b->m.offset;
>> +		else if (b->memory == V4L2_MEMORY_DMABUF)
>> +			e->planes[0].m.dmabuf_fd = b->m.fd;
>> +		else
>> +			e->planes[0].m.userptr = b->m.userptr;
>> +	}
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_buffer_to_ext_buffer);
>> +
>>  static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
>>  				struct file *file, void *fh, void *arg)
>>  {
>> @@ -2467,31 +2629,112 @@ static int v4l_reqbufs(const struct v4l2_ioctl_ops *ops,
>>  	return ops->vidioc_reqbufs(file, fh, p);
>>  }
>>  
>> +static int v4l_do_buf_op(int (*op)(struct file *, void *,
>> +				   struct v4l2_buffer *),
>> +			 int (*ext_op)(struct file *, void *,
>> +				       struct v4l2_ext_buffer *),
>> +			 struct file *file, void *fh, struct v4l2_buffer *b)
>> +{
>> +	struct v4l2_ext_buffer e;
>> +	int ret;
>> +
>> +	ret = check_fmt(file, b->type);
>> +	if (ret)
>> +		return ret;
>> +
>> +	if (op)
>> +		return op(file, fh, b);
>> +
>> +	ret = v4l2_buffer_to_ext_buffer(b, &e);
>> +	if (ret)
>> +		return ret;
>> +
>> +	ret = ext_op(file, fh, &e);
>> +	if (ret)
>> +		return ret;
>> +
>> +	v4l2_ext_buffer_to_buffer(&e, b, V4L2_TYPE_IS_MULTIPLANAR(b->type));
>> +	return 0;
>> +}
>> +
>> +static int v4l_do_ext_buf_op(int (*op)(struct file *, void *,
>> +				       struct v4l2_buffer *),
>> +			     int (*ext_op)(struct file *, void *,
>> +					   struct v4l2_ext_buffer *),
>> +			     struct file *file, void *fh,
>> +			     struct v4l2_ext_buffer *e)
>> +{
>> +	struct video_device *vdev = video_devdata(file);
>> +	struct v4l2_plane planes[VIDEO_MAX_PLANES];
>> +	struct v4l2_buffer b;
>> +	bool mplane_cap;
>> +	int ret;
>> +
>> +	ret = check_fmt(file, e->type);
>> +	if (ret)
>> +		return ret;
>> +
>> +	if (ext_op)
>> +		return ext_op(file, fh, e);
>> +
>> +	mplane_cap = !!(vdev->device_caps &
>> +			(V4L2_CAP_VIDEO_CAPTURE_MPLANE |
>> +			 V4L2_CAP_VIDEO_OUTPUT_MPLANE |
>> +			 V4L2_CAP_VIDEO_M2M_MPLANE));
>> +	b.m.planes = planes;
>> +	b.length = VIDEO_MAX_PLANES;
>> +	ret = v4l2_ext_buffer_to_buffer(e, &b, mplane_cap);
>> +	if (ret)
>> +		return ret;
>> +
>> +	ret = op(file, fh, &b);
>> +	if (ret)
>> +		return ret;
>> +
>> +	v4l2_buffer_to_ext_buffer(&b, e);
>> +	return 0;
>> +}
>> +
>>  static int v4l_querybuf(const struct v4l2_ioctl_ops *ops,
>> -				struct file *file, void *fh, void *arg)
>> +			struct file *file, void *fh, void *arg)
>>  {
>> -	struct v4l2_buffer *p = arg;
>> -	int ret = check_fmt(file, p->type);
>> +	return v4l_do_buf_op(ops->vidioc_querybuf, ops->vidioc_ext_querybuf,
>> +			     file, fh, arg);
>> +}
>>  
>> -	return ret ? ret : ops->vidioc_querybuf(file, fh, p);
>> +static int v4l_ext_querybuf(const struct v4l2_ioctl_ops *ops,
>> +			    struct file *file, void *fh, void *arg)
>> +{
>> +	return v4l_do_ext_buf_op(ops->vidioc_querybuf,
>> +				 ops->vidioc_ext_querybuf, file, fh, arg);
>>  }
>>  
>>  static int v4l_qbuf(const struct v4l2_ioctl_ops *ops,
>> -				struct file *file, void *fh, void *arg)
>> +		    struct file *file, void *fh, void *arg)
>>  {
>> -	struct v4l2_buffer *p = arg;
>> -	int ret = check_fmt(file, p->type);
>> +	return v4l_do_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
>> +			     file, fh, arg);
>> +}
>>  
>> -	return ret ? ret : ops->vidioc_qbuf(file, fh, p);
>> +static int v4l_ext_qbuf(const struct v4l2_ioctl_ops *ops,
>> +			struct file *file, void *fh, void *arg)
>> +{
>> +	return v4l_do_ext_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
>> +				 file, fh, arg);
>>  }
>>  
>>  static int v4l_dqbuf(const struct v4l2_ioctl_ops *ops,
>> -				struct file *file, void *fh, void *arg)
>> +		     struct file *file, void *fh, void *arg)
>>  {
>> -	struct v4l2_buffer *p = arg;
>> -	int ret = check_fmt(file, p->type);
>> +	return v4l_do_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
>> +			     file, fh, arg);
>> +}
>>  
>> -	return ret ? ret : ops->vidioc_dqbuf(file, fh, p);
>> +static int v4l_ext_dqbuf(const struct v4l2_ioctl_ops *ops,
>> +			 struct file *file, void *fh, void *arg)
>> +{
>> +	return v4l_do_ext_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
>> +				 file, fh, arg);
>>  }
>>  
>>  static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
>> @@ -2507,7 +2750,27 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
>>  
>>  	v4l_sanitize_format(&create->format);
>>  
>> -	ret = ops->vidioc_create_bufs(file, fh, create);
>> +	if (ops->vidioc_create_bufs) {
>> +		ret = ops->vidioc_create_bufs(file, fh, create);
>> +	} else {
>> +		struct v4l2_ext_create_buffers ecreate = {
>> +			.count = create->count,
>> +			.memory = create->memory,
>> +		};
>> +
>> +		ret = v4l2_format_to_ext_pix_format(&create->format,
>> +						    &ecreate.format, true);
>> +		if (ret)
>> +			return ret;
>> +
>> +		ret = ops->vidioc_ext_create_bufs(file, fh, &ecreate);
>> +		if (ret)
>> +			return ret;
>> +
>> +		create->index = ecreate.index;
>> +		create->count = ecreate.count;
>> +		create->capabilities = ecreate.capabilities;
>> +	}
>>  
>>  	if (create->format.type == V4L2_BUF_TYPE_VIDEO_CAPTURE ||
>>  	    create->format.type == V4L2_BUF_TYPE_VIDEO_OUTPUT)
>> @@ -2516,13 +2779,60 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
>>  	return ret;
>>  }
>>  
>> +static int v4l_ext_create_bufs(const struct v4l2_ioctl_ops *ops,
>> +			       struct file *file, void *fh, void *arg)
>> +{
>> +	struct v4l2_ext_create_buffers *ecreate = arg;
>> +	struct video_device *vdev = video_devdata(file);
>> +	struct v4l2_create_buffers create = {
>> +		.count = ecreate->count,
>> +		.memory = ecreate->memory,
>> +		.flags = ecreate->flags,
>> +	};
>> +	bool mplane_cap;
>> +	int ret;
>> +
>> +	ret = check_fmt(file, ecreate->format.type);
>> +	if (ret)
>> +		return ret;
>> +
>> +	if (ops->vidioc_ext_create_bufs)
>> +		return ops->vidioc_ext_create_bufs(file, fh, ecreate);
>> +
>> +	mplane_cap = !!(vdev->device_caps &
>> +			(V4L2_CAP_VIDEO_CAPTURE_MPLANE |
>> +			 V4L2_CAP_VIDEO_OUTPUT_MPLANE |
>> +			 V4L2_CAP_VIDEO_M2M_MPLANE));
>> +	ret = v4l2_ext_pix_format_to_format(&ecreate->format,
>> +					    &create.format, mplane_cap, true);
>> +	if (ret)
>> +		return ret;
>> +
>> +	ret = v4l_create_bufs(ops, file, fh, &create);
>> +	if (ret)
>> +		return ret;
>> +
>> +	ecreate->index = create.index;
>> +	ecreate->count = create.count;
>> +	ecreate->capabilities = create.capabilities;
>> +
>> +	return 0;
>> +}
>> +
>>  static int v4l_prepare_buf(const struct v4l2_ioctl_ops *ops,
>> -				struct file *file, void *fh, void *arg)
>> +			   struct file *file, void *fh, void *arg)
>>  {
>> -	struct v4l2_buffer *b = arg;
>> -	int ret = check_fmt(file, b->type);
>> +	return v4l_do_buf_op(ops->vidioc_prepare_buf,
>> +			     ops->vidioc_ext_prepare_buf,
>> +			     file, fh, arg);
>> +}
>>  
>> -	return ret ? ret : ops->vidioc_prepare_buf(file, fh, b);
>> +static int v4l_ext_prepare_buf(const struct v4l2_ioctl_ops *ops,
>> +			       struct file *file, void *fh, void *arg)
>> +{
>> +	return v4l_do_ext_buf_op(ops->vidioc_prepare_buf,
>> +				 ops->vidioc_ext_prepare_buf,
>> +				 file, fh, arg);
>>  }
>>  
>>  static int v4l_g_parm(const struct v4l2_ioctl_ops *ops,
>> @@ -3196,12 +3506,15 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>>  	IOCTL_INFO(VIDIOC_S_EXT_PIX_FMT, v4l_s_ext_pix_fmt, v4l_print_ext_pix_format, INFO_FL_PRIO),
>>  	IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>>  	IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
>> +	IOCTL_INFO(VIDIOC_EXT_QUERYBUF, v4l_ext_querybuf, v4l_print_ext_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_ext_buffer, planes)),
>>  	IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
>>  	IOCTL_INFO(VIDIOC_S_FBUF, v4l_stub_s_fbuf, v4l_print_framebuffer, INFO_FL_PRIO),
>>  	IOCTL_INFO(VIDIOC_OVERLAY, v4l_overlay, v4l_print_u32, INFO_FL_PRIO),
>>  	IOCTL_INFO(VIDIOC_QBUF, v4l_qbuf, v4l_print_buffer, INFO_FL_QUEUE),
>>  	IOCTL_INFO(VIDIOC_EXPBUF, v4l_stub_expbuf, v4l_print_exportbuffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_exportbuffer, flags)),
>> +	IOCTL_INFO(VIDIOC_EXT_QBUF, v4l_ext_qbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),
>>  	IOCTL_INFO(VIDIOC_DQBUF, v4l_dqbuf, v4l_print_buffer, INFO_FL_QUEUE),
>> +	IOCTL_INFO(VIDIOC_EXT_DQBUF, v4l_ext_dqbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),
>>  	IOCTL_INFO(VIDIOC_STREAMON, v4l_streamon, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
>>  	IOCTL_INFO(VIDIOC_STREAMOFF, v4l_streamoff, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
>>  	IOCTL_INFO(VIDIOC_G_PARM, v4l_g_parm, v4l_print_streamparm, INFO_FL_CLEAR(v4l2_streamparm, type)),
>> @@ -3266,7 +3579,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>>  	IOCTL_INFO(VIDIOC_SUBSCRIBE_EVENT, v4l_subscribe_event, v4l_print_event_subscription, 0),
>>  	IOCTL_INFO(VIDIOC_UNSUBSCRIBE_EVENT, v4l_unsubscribe_event, v4l_print_event_subscription, 0),
>>  	IOCTL_INFO(VIDIOC_CREATE_BUFS, v4l_create_bufs, v4l_print_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>> +	IOCTL_INFO(VIDIOC_EXT_CREATE_BUFS, v4l_ext_create_bufs, v4l_print_ext_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>>  	IOCTL_INFO(VIDIOC_PREPARE_BUF, v4l_prepare_buf, v4l_print_buffer, INFO_FL_QUEUE),
>> +	IOCTL_INFO(VIDIOC_EXT_PREPARE_BUF, v4l_ext_prepare_buf, v4l_print_ext_buffer, INFO_FL_QUEUE),
>>  	IOCTL_INFO(VIDIOC_ENUM_DV_TIMINGS, v4l_stub_enum_dv_timings, v4l_print_enum_dv_timings, INFO_FL_CLEAR(v4l2_enum_dv_timings, pad)),
>>  	IOCTL_INFO(VIDIOC_QUERY_DV_TIMINGS, v4l_stub_query_dv_timings, v4l_print_dv_timings, INFO_FL_ALWAYS_COPY),
>>  	IOCTL_INFO(VIDIOC_DV_TIMINGS_CAP, v4l_stub_dv_timings_cap, v4l_print_dv_timings_cap, INFO_FL_CLEAR(v4l2_dv_timings_cap, pad)),
>> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
>> index 525ce86725260..524caedebab3d 100644
>> --- a/include/media/v4l2-ioctl.h
>> +++ b/include/media/v4l2-ioctl.h
>> @@ -169,16 +169,26 @@ struct v4l2_fh;
>>   *	:ref:`VIDIOC_REQBUFS <vidioc_reqbufs>` ioctl
>>   * @vidioc_querybuf: pointer to the function that implements
>>   *	:ref:`VIDIOC_QUERYBUF <vidioc_querybuf>` ioctl
>> + * @vidioc_ext_querybuf: pointer to the function that implements
>> + *	:ref:`VIDIOC_EXT_QUERYBUF <vidioc_ext_querybuf>` ioctl
>>   * @vidioc_qbuf: pointer to the function that implements
>>   *	:ref:`VIDIOC_QBUF <vidioc_qbuf>` ioctl
>> + * @vidioc_ext_qbuf: pointer to the function that implements
>> + *	:ref:`VIDIOC_EXT_QBUF <vidioc_ext_qbuf>` ioctl
>>   * @vidioc_expbuf: pointer to the function that implements
>>   *	:ref:`VIDIOC_EXPBUF <vidioc_expbuf>` ioctl
>>   * @vidioc_dqbuf: pointer to the function that implements
>>   *	:ref:`VIDIOC_DQBUF <vidioc_qbuf>` ioctl
>> + * @vidioc_ext_dqbuf: pointer to the function that implements
>> + *	:ref:`VIDIOC_EXT_DQBUF <vidioc_ext_qbuf>` ioctl
>>   * @vidioc_create_bufs: pointer to the function that implements
>>   *	:ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl
>> + * @vidioc_ext_create_bufs: pointer to the function that implements
>> + *	:ref:`VIDIOC_EXT_CREATE_BUFS <vidioc_ext_create_bufs>` ioctl
>>   * @vidioc_prepare_buf: pointer to the function that implements
>>   *	:ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl
>> + * @vidioc_ext_prepare_buf: pointer to the function that implements
>> + *	:ref:`VIDIOC_EXT_PREPARE_BUF <vidioc_ext_prepare_buf>` ioctl
>>   * @vidioc_overlay: pointer to the function that implements
>>   *	:ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl
>>   * @vidioc_g_fbuf: pointer to the function that implements
>> @@ -439,17 +449,27 @@ struct v4l2_ioctl_ops {
>>  			      struct v4l2_requestbuffers *b);
>>  	int (*vidioc_querybuf)(struct file *file, void *fh,
>>  			       struct v4l2_buffer *b);
>> +	int (*vidioc_ext_querybuf)(struct file *file, void *fh,
>> +				   struct v4l2_ext_buffer *b);
>>  	int (*vidioc_qbuf)(struct file *file, void *fh,
>>  			   struct v4l2_buffer *b);
>> +	int (*vidioc_ext_qbuf)(struct file *file, void *fh,
>> +			       struct v4l2_ext_buffer *b);
>>  	int (*vidioc_expbuf)(struct file *file, void *fh,
>>  			     struct v4l2_exportbuffer *e);
>>  	int (*vidioc_dqbuf)(struct file *file, void *fh,
>>  			    struct v4l2_buffer *b);
>> +	int (*vidioc_ext_dqbuf)(struct file *file, void *fh,
>> +				struct v4l2_ext_buffer *b);
>>  
>>  	int (*vidioc_create_bufs)(struct file *file, void *fh,
>>  				  struct v4l2_create_buffers *b);
>> +	int (*vidioc_ext_create_bufs)(struct file *file, void *fh,
>> +				      struct v4l2_ext_create_buffers *b);
>>  	int (*vidioc_prepare_buf)(struct file *file, void *fh,
>>  				  struct v4l2_buffer *b);
>> +	int (*vidioc_ext_prepare_buf)(struct file *file, void *fh,
>> +				      struct v4l2_ext_buffer *b);
>>  
>>  	int (*vidioc_overlay)(struct file *file, void *fh, unsigned int i);
>>  	int (*vidioc_g_fbuf)(struct file *file, void *fh,
>> @@ -758,6 +778,12 @@ int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
>>  				  struct v4l2_format *f,
>>  				  bool mplane_cap, bool strict);
>>  
>> +int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
>> +			      struct v4l2_buffer *b,
>> +			      bool mplane_cap);
>> +int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
>> +			      struct v4l2_ext_buffer *e);
>> +
>>  /*
>>   * The user space interpretation of the 'v4l2_event' differs
>>   * based on the 'time_t' definition on 32-bit architectures, so
>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>> index fc04c81ce7713..f4906adddc280 100644
>> --- a/include/uapi/linux/videodev2.h
>> +++ b/include/uapi/linux/videodev2.h
>> @@ -994,6 +994,37 @@ struct v4l2_plane {
>>  	__u32			reserved[11];
>>  };
>>  
>> +/**
>> + * struct v4l2_ext_plane - extended plane buffer info
>> + * @buffer_length:	size of the entire buffer in bytes, should fit
>> + *			@offset + @plane_length
>> + * @plane_length:	size of the plane in bytes.
>> + * @userptr:		when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
>> + *			to this plane.
>> + * @dmabuf_fd:		when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
>> + *			associated with this plane.
>> + * @offset:		offset in the memory buffer where the plane starts. If
>> + *			V4L2_MEMORY_MMAP is used, then it can be a "cookie" that
>> + *			should be passed to mmap() called on the video node.
> 
> I wouldn't overload this field. Just keep a mem_offset field in the union m.
> 
> The offset field itself is still valid, even for MMAPed planes.

ack

> 
>> + * @reserved:		extra space reserved for future fields, must be set to 0.
>> + *
>> + *
>> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
>> + * can have one plane for Y, and another for interleaved CbCr components.
>> + * Each plane can reside in a separate memory buffer, or even in
>> + * a completely separate memory node (e.g. in embedded devices).
>> + */
>> +struct v4l2_ext_plane {
>> +	__u32 buffer_length;
>> +	__u32 plane_length;
>> +	union {
>> +		__u64 userptr;
>> +		__s32 dmabuf_fd;
>> +	} m;
>> +	__u32 offset;
>> +	__u32 reserved[4];
>> +};
>> +
>>  /**
>>   * struct v4l2_buffer - video buffer info
>>   * @index:	id number of the buffer
>> @@ -1055,6 +1086,36 @@ struct v4l2_buffer {
>>  	};
>>  };
>>  
>> +/**
>> + * struct v4l2_ext_buffer - extended video buffer info
>> + * @index:	id number of the buffer
>> + * @type:	V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
>> + * @flags:	buffer informational flags
>> + * @field:	enum v4l2_field; field order of the image in the buffer
>> + * @timestamp:	frame timestamp
>> + * @sequence:	sequence count of this frame
>> + * @memory:	enum v4l2_memory; the method, in which the actual video data is
>> + *		passed
>> + * @planes:	per-plane buffer information
>> + * @request_fd:	fd of the request that this buffer should use
>> + * @reserved:	extra space reserved for future fields, must be set to 0
>> + *
>> + * Contains data exchanged by application and driver using one of the Streaming
>> + * I/O methods.
>> + */
>> +struct v4l2_ext_buffer {
>> +	__u32 index;
>> +	__u32 type;
>> +	__u32 flags;
> 
> Make this a u64: we have quite a few flags already.

ack

> 
>> +	__u32 field;
>> +	__u64 timestamp;
> 
> We need a second timestamp here, i.e. the time at which the buffer was finalized
> and returned to userspace. This is needed to let userspace make a timeline between
> events and buffers. Each v4l2_event has a timestamp but it can't be used to
> see if a buffer was returned before or after that event. It's something that's
> been annoying me for some time now.

I'll let Nicolas comment on this, but it seems synchronizing buffer and events
in userspace is a bit more complicated.
We could also add this later with the reserved fields when we have a clearer view
and a PoC with userspace.

Regards,
Helen

> 
> A '__u64 seq_timestamp' seems like a decent name.
> 
> Regards,
> 
> 	Hans
> 
>> +	__u32 sequence;
>> +	__u32 memory;
>> +	__u32 request_fd;
>> +	struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
>> +	__u32 reserved[4];
>> +};
>> +
>>  #ifndef __KERNEL__
>>  /**
>>   * v4l2_timeval_to_ns - Convert timeval to nanoseconds
>> @@ -2520,6 +2581,29 @@ struct v4l2_create_buffers {
>>  	__u32			reserved[6];
>>  };
>>  
>> +/**
>> + * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
>> + * @index:	on return, index of the first created buffer
>> + * @count:	entry: number of requested buffers,
>> + *		return: number of created buffers
>> + * @memory:	enum v4l2_memory; buffer memory type
>> + * @capabilities: capabilities of this buffer type.
>> + * @format:	frame format, for which buffers are requested
>> + * @flags:	additional buffer management attributes (ignored unless the
>> + *		queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
>> + *		and configured for MMAP streaming I/O).
>> + * @reserved:	extra space reserved for future fields, must be set to 0
>> + */
>> +struct v4l2_ext_create_buffers {
>> +	__u32				index;
>> +	__u32				count;
>> +	__u32				memory;
>> +	struct v4l2_ext_pix_format	format;
>> +	__u32				capabilities;
>> +	__u32				flags;
>> +	__u32 reserved[4];
>> +};
>> +
>>  /*
>>   *	I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
>>   *
>> @@ -2623,6 +2707,11 @@ struct v4l2_create_buffers {
>>  #define VIDIOC_G_EXT_PIX_FMT	_IOWR('V', 104, struct v4l2_ext_pix_format)
>>  #define VIDIOC_S_EXT_PIX_FMT	_IOWR('V', 105, struct v4l2_ext_pix_format)
>>  #define VIDIOC_TRY_EXT_PIX_FMT	_IOWR('V', 106, struct v4l2_ext_pix_format)
>> +#define VIDIOC_EXT_CREATE_BUFS	_IOWR('V', 107, struct v4l2_ext_create_buffers)
>> +#define VIDIOC_EXT_QUERYBUF	_IOWR('V', 108, struct v4l2_ext_buffer)
>> +#define VIDIOC_EXT_QBUF		_IOWR('V', 109, struct v4l2_ext_buffer)
>> +#define VIDIOC_EXT_DQBUF	_IOWR('V', 110, struct v4l2_ext_buffer)
>> +#define VIDIOC_EXT_PREPARE_BUF	_IOWR('V', 111, struct v4l2_ext_buffer)
>>  
>>  /* Reminder: when adding new ioctls please add support for them to
>>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
>>
> 

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v4 2/6] media: v4l2: Add extended buffer operations
  2020-07-21 14:30       ` Stanimir Varbanov
@ 2020-07-21 14:40         ` Helen Koike
  2020-07-24 13:16           ` Stanimir Varbanov
  0 siblings, 1 reply; 29+ messages in thread
From: Helen Koike @ 2020-07-21 14:40 UTC (permalink / raw)
  To: Stanimir Varbanov, mchehab, hans.verkuil, laurent.pinchart,
	sakari.ailus, linux-media
  Cc: Boris Brezillon, tfiga, hiroh, nicolas, Brian.Starkey, kernel,
	narmstrong, linux-kernel, frkoenig, mjourdan



On 7/21/20 11:30 AM, Stanimir Varbanov wrote:
> Hi Helen,
> 
> On 7/21/20 4:54 PM, Helen Koike wrote:
>> Hi,
>>
>> On 7/21/20 8:26 AM, Stanimir Varbanov wrote:
>>>
>>>
>>> On 7/17/20 2:54 PM, Helen Koike wrote:
>>>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>>>
>>>> Those extended buffer ops have several purpose:
>>>> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
>>>>    the number of ns elapsed since 1970
>>>> 2/ Unify single/multiplanar handling
>>>> 3/ Add a new start offset field to each v4l2 plane buffer info struct
>>>>    to support the case where a single buffer object is storing all
>>>>    planes data, each one being placed at a different offset
>>>>
>>>> New hooks are created in v4l2_ioctl_ops so that drivers can start using
>>>> these new objects.
>>>>
>>>> The core takes care of converting new ioctls requests to old ones
>>>> if the driver does not support the new hooks, and vice versa.
>>>>
>>>> Note that the timecode field is gone, since there doesn't seem to be
>>>> in-kernel users. We can be added back in the reserved area if needed or
>>>> use the Request API to collect more metadata information from the
>>>> frame.
>>>>
>>>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>>>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>>> ---
>>>> Changes in v4:
>>>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
>>>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
>>>> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
>>>> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
>>>> I think we can add this later, so I removed it from this RFC to simplify it.
>>>> - Remove num_planes field from struct v4l2_ext_buffer
>>>> - Add flags field to struct v4l2_ext_create_buffers
>>>> - Reformulate struct v4l2_ext_plane
>>>> - Fix some bugs caught by v4l2-compliance
>>>> - Rebased on top of media/master (post 5.8-rc1)
>>>>
>>>> Changes in v3:
>>>> - Rebased on top of media/master (post 5.4-rc1)
>>>>
>>>> Changes in v2:
>>>> - Add reserved space to v4l2_ext_buffer so that new fields can be added
>>>>   later on
>>>> ---
>>>>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
>>>>  drivers/media/v4l2-core/v4l2-ioctl.c | 349 +++++++++++++++++++++++++--
>>>>  include/media/v4l2-ioctl.h           |  26 ++
>>>>  include/uapi/linux/videodev2.h       |  89 +++++++
>>>>  4 files changed, 471 insertions(+), 22 deletions(-)
>>>>
>>>
>>> <cut>
>>>
>>>> +/**
>>>> + * struct v4l2_ext_plane - extended plane buffer info
>>>> + * @buffer_length:	size of the entire buffer in bytes, should fit
>>>> + *			@offset + @plane_length
>>>> + * @plane_length:	size of the plane in bytes.
>>>> + * @userptr:		when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
>>>> + *			to this plane.
>>>> + * @dmabuf_fd:		when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
>>>> + *			associated with this plane.
>>>> + * @offset:		offset in the memory buffer where the plane starts. If
>>>> + *			V4L2_MEMORY_MMAP is used, then it can be a "cookie" that
>>>> + *			should be passed to mmap() called on the video node.
>>>> + * @reserved:		extra space reserved for future fields, must be set to 0.
>>>> + *
>>>> + *
>>>> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
>>>> + * can have one plane for Y, and another for interleaved CbCr components.
>>>> + * Each plane can reside in a separate memory buffer, or even in
>>>> + * a completely separate memory node (e.g. in embedded devices).
>>>> + */
>>>> +struct v4l2_ext_plane {
>>>> +	__u32 buffer_length;
>>>> +	__u32 plane_length;
>>>> +	union {
>>>> +		__u64 userptr;
>>>> +		__s32 dmabuf_fd;
>>>> +	} m;
>>>> +	__u32 offset;
>>>> +	__u32 reserved[4];
>>>> +};
>>>> +
>>>>  /**
>>>>   * struct v4l2_buffer - video buffer info
>>>>   * @index:	id number of the buffer
>>>> @@ -1055,6 +1086,36 @@ struct v4l2_buffer {
>>>>  	};
>>>>  };
>>>>  
>>>> +/**
>>>> + * struct v4l2_ext_buffer - extended video buffer info
>>>> + * @index:	id number of the buffer
>>>> + * @type:	V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
>>>> + * @flags:	buffer informational flags
>>>> + * @field:	enum v4l2_field; field order of the image in the buffer
>>>> + * @timestamp:	frame timestamp
>>>> + * @sequence:	sequence count of this frame
>>>> + * @memory:	enum v4l2_memory; the method, in which the actual video data is
>>>> + *		passed
>>>> + * @planes:	per-plane buffer information
>>>> + * @request_fd:	fd of the request that this buffer should use
>>>> + * @reserved:	extra space reserved for future fields, must be set to 0
>>>> + *
>>>> + * Contains data exchanged by application and driver using one of the Streaming
>>>> + * I/O methods.
>>>> + */
>>>> +struct v4l2_ext_buffer {
>>>> +	__u32 index;
>>>> +	__u32 type;
>>>> +	__u32 flags;
>>>> +	__u32 field;
>>>> +	__u64 timestamp;
>>>> +	__u32 sequence;
>>>> +	__u32 memory;
>>>> +	__u32 request_fd;
>>>
>>> This should be __s32, at least for consistency with dmabuf_fd?
>>
>> I see that in struct v4l2_buffer, we have __s32, I don't mind changing it
>> to keep the consistency, I just don't see where this value can be a negative
>> number.
> 
> here
> https://elixir.bootlin.com/linux/v5.8-rc4/source/drivers/media/common/videobuf2/videobuf2-v4l2.c#L134

I saw that -1 is used to signal an invalid value, but I was just wondering when request_fd = 0 is valid.

> 
>>
>>>
>>>> +	struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
>>>> +	__u32 reserved[4];
>>>
>>> I think we have to reserve more words here for future extensions.
>>>
>>> I'd like also to propose to add here __s32 metadata_fd. The idea behind
>>> this is to have a way to pass per-frame metadata dmabuf buffers for
>>> synchronous type of metadata where the metadata is coming at the same
>>> time with data buffers. What would be the format of the metadata buffer
>>> is TBD.
>>>
>>> One option for metadata buffer format could be:
>>>
>>> header {
>>> 	num_ctrls
>>> 	array_of_ctrls [0..N]
>>> 		ctrl_id
>>> 		ctrl_size
>>> 		ctrl_offset
>>> }
>>>
>>> data {
>>> 	cid0	//offset of cid0 in dmabuf buffer
>>> 	cid1
>>> 	cidN
>>> }
>>
>> Would it be better if, instead of adding a medatata_fd inside struct v4l2_ext_buffer,
>> we create a new ioctl that gets this structs for the controls and sync them using the
>> Request API ?
> 
> no, this solution has performance drawbacks when the metadata is big,
> think of 64K.

Why? You could still use a dmabuf in this new ioctl, no?


Regards,
Helen

> 
>>
>> I'd like to avoid too much metadata in the buffer object.
>>
>> Regards,
>> Helen
>>
>>>
>>> This will make easy to get concrete ctrl id without a need to parse the
>>> whole metadata buffer. Also using dmabuf we don't need to copy data
>>> between userspace <-> kernelspace (just cache syncs through
>>> begin/end_cpu_access).
>>>
>>> The open question is who will validate the metadata buffer when it comes
>>> from userspace. The obvious answer is v4l2-core but looking into DRM
>>> subsytem they give more freedom to the drivers, and just provide generic
>>> helpers which are not mandatory.
>>>
>>> I guess this will be a voice in the wilderness but I wanted to know your
>>> opinion.
>>>
>>>> +};
>>>> +
>>>>  #ifndef __KERNEL__
>>>>  /**
>>>>   * v4l2_timeval_to_ns - Convert timeval to nanoseconds
>>>> @@ -2520,6 +2581,29 @@ struct v4l2_create_buffers {
>>>>  	__u32			reserved[6];
>>>>  };
>>>>  
>>>> +/**
>>>> + * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
>>>> + * @index:	on return, index of the first created buffer
>>>> + * @count:	entry: number of requested buffers,
>>>> + *		return: number of created buffers
>>>> + * @memory:	enum v4l2_memory; buffer memory type
>>>> + * @capabilities: capabilities of this buffer type.
>>>> + * @format:	frame format, for which buffers are requested
>>>> + * @flags:	additional buffer management attributes (ignored unless the
>>>> + *		queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
>>>> + *		and configured for MMAP streaming I/O).
>>>> + * @reserved:	extra space reserved for future fields, must be set to 0
>>>> + */
>>>> +struct v4l2_ext_create_buffers {
>>>> +	__u32				index;
>>>> +	__u32				count;
>>>> +	__u32				memory;
>>>> +	struct v4l2_ext_pix_format	format;
>>>> +	__u32				capabilities;
>>>> +	__u32				flags;
>>>> +	__u32 reserved[4];
>>>> +};
>>>> +
>>>>  /*
>>>>   *	I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
>>>>   *
>>>> @@ -2623,6 +2707,11 @@ struct v4l2_create_buffers {
>>>>  #define VIDIOC_G_EXT_PIX_FMT	_IOWR('V', 104, struct v4l2_ext_pix_format)
>>>>  #define VIDIOC_S_EXT_PIX_FMT	_IOWR('V', 105, struct v4l2_ext_pix_format)
>>>>  #define VIDIOC_TRY_EXT_PIX_FMT	_IOWR('V', 106, struct v4l2_ext_pix_format)
>>>> +#define VIDIOC_EXT_CREATE_BUFS	_IOWR('V', 107, struct v4l2_ext_create_buffers)
>>>> +#define VIDIOC_EXT_QUERYBUF	_IOWR('V', 108, struct v4l2_ext_buffer)
>>>> +#define VIDIOC_EXT_QBUF		_IOWR('V', 109, struct v4l2_ext_buffer)
>>>> +#define VIDIOC_EXT_DQBUF	_IOWR('V', 110, struct v4l2_ext_buffer)
>>>> +#define VIDIOC_EXT_PREPARE_BUF	_IOWR('V', 111, struct v4l2_ext_buffer)
>>>>  
>>>>  /* Reminder: when adding new ioctls please add support for them to
>>>>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
>>>>
>>>
> 

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v4 2/6] media: v4l2: Add extended buffer operations
  2020-07-21 14:40         ` Helen Koike
@ 2020-07-24 13:16           ` Stanimir Varbanov
  2020-07-27 12:01             ` Helen Koike
  2020-07-27 12:35             ` Tomasz Figa
  0 siblings, 2 replies; 29+ messages in thread
From: Stanimir Varbanov @ 2020-07-24 13:16 UTC (permalink / raw)
  To: Helen Koike, mchehab, hans.verkuil, laurent.pinchart,
	sakari.ailus, linux-media
  Cc: Boris Brezillon, tfiga, hiroh, nicolas, Brian.Starkey, kernel,
	narmstrong, linux-kernel, frkoenig, mjourdan



On 7/21/20 5:40 PM, Helen Koike wrote:
> 
> 
> On 7/21/20 11:30 AM, Stanimir Varbanov wrote:
>> Hi Helen,
>>
>> On 7/21/20 4:54 PM, Helen Koike wrote:
>>> Hi,
>>>
>>> On 7/21/20 8:26 AM, Stanimir Varbanov wrote:
>>>>
>>>>
>>>> On 7/17/20 2:54 PM, Helen Koike wrote:
>>>>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>>>>
>>>>> Those extended buffer ops have several purpose:
>>>>> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
>>>>>    the number of ns elapsed since 1970
>>>>> 2/ Unify single/multiplanar handling
>>>>> 3/ Add a new start offset field to each v4l2 plane buffer info struct
>>>>>    to support the case where a single buffer object is storing all
>>>>>    planes data, each one being placed at a different offset
>>>>>
>>>>> New hooks are created in v4l2_ioctl_ops so that drivers can start using
>>>>> these new objects.
>>>>>
>>>>> The core takes care of converting new ioctls requests to old ones
>>>>> if the driver does not support the new hooks, and vice versa.
>>>>>
>>>>> Note that the timecode field is gone, since there doesn't seem to be
>>>>> in-kernel users. We can be added back in the reserved area if needed or
>>>>> use the Request API to collect more metadata information from the
>>>>> frame.
>>>>>
>>>>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>>>>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
>>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>>>> ---
>>>>> Changes in v4:
>>>>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
>>>>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
>>>>> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
>>>>> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
>>>>> I think we can add this later, so I removed it from this RFC to simplify it.
>>>>> - Remove num_planes field from struct v4l2_ext_buffer
>>>>> - Add flags field to struct v4l2_ext_create_buffers
>>>>> - Reformulate struct v4l2_ext_plane
>>>>> - Fix some bugs caught by v4l2-compliance
>>>>> - Rebased on top of media/master (post 5.8-rc1)
>>>>>
>>>>> Changes in v3:
>>>>> - Rebased on top of media/master (post 5.4-rc1)
>>>>>
>>>>> Changes in v2:
>>>>> - Add reserved space to v4l2_ext_buffer so that new fields can be added
>>>>>   later on
>>>>> ---
>>>>>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
>>>>>  drivers/media/v4l2-core/v4l2-ioctl.c | 349 +++++++++++++++++++++++++--
>>>>>  include/media/v4l2-ioctl.h           |  26 ++
>>>>>  include/uapi/linux/videodev2.h       |  89 +++++++
>>>>>  4 files changed, 471 insertions(+), 22 deletions(-)
>>>>>
>>>>
>>>> <cut>
>>>>
>>>>> +/**
>>>>> + * struct v4l2_ext_plane - extended plane buffer info
>>>>> + * @buffer_length:	size of the entire buffer in bytes, should fit
>>>>> + *			@offset + @plane_length
>>>>> + * @plane_length:	size of the plane in bytes.
>>>>> + * @userptr:		when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
>>>>> + *			to this plane.
>>>>> + * @dmabuf_fd:		when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
>>>>> + *			associated with this plane.
>>>>> + * @offset:		offset in the memory buffer where the plane starts. If
>>>>> + *			V4L2_MEMORY_MMAP is used, then it can be a "cookie" that
>>>>> + *			should be passed to mmap() called on the video node.
>>>>> + * @reserved:		extra space reserved for future fields, must be set to 0.
>>>>> + *
>>>>> + *
>>>>> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
>>>>> + * can have one plane for Y, and another for interleaved CbCr components.
>>>>> + * Each plane can reside in a separate memory buffer, or even in
>>>>> + * a completely separate memory node (e.g. in embedded devices).
>>>>> + */
>>>>> +struct v4l2_ext_plane {
>>>>> +	__u32 buffer_length;
>>>>> +	__u32 plane_length;
>>>>> +	union {
>>>>> +		__u64 userptr;
>>>>> +		__s32 dmabuf_fd;
>>>>> +	} m;
>>>>> +	__u32 offset;
>>>>> +	__u32 reserved[4];
>>>>> +};
>>>>> +
>>>>>  /**
>>>>>   * struct v4l2_buffer - video buffer info
>>>>>   * @index:	id number of the buffer
>>>>> @@ -1055,6 +1086,36 @@ struct v4l2_buffer {
>>>>>  	};
>>>>>  };
>>>>>  
>>>>> +/**
>>>>> + * struct v4l2_ext_buffer - extended video buffer info
>>>>> + * @index:	id number of the buffer
>>>>> + * @type:	V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
>>>>> + * @flags:	buffer informational flags
>>>>> + * @field:	enum v4l2_field; field order of the image in the buffer
>>>>> + * @timestamp:	frame timestamp
>>>>> + * @sequence:	sequence count of this frame
>>>>> + * @memory:	enum v4l2_memory; the method, in which the actual video data is
>>>>> + *		passed
>>>>> + * @planes:	per-plane buffer information
>>>>> + * @request_fd:	fd of the request that this buffer should use
>>>>> + * @reserved:	extra space reserved for future fields, must be set to 0
>>>>> + *
>>>>> + * Contains data exchanged by application and driver using one of the Streaming
>>>>> + * I/O methods.
>>>>> + */
>>>>> +struct v4l2_ext_buffer {
>>>>> +	__u32 index;
>>>>> +	__u32 type;
>>>>> +	__u32 flags;
>>>>> +	__u32 field;
>>>>> +	__u64 timestamp;
>>>>> +	__u32 sequence;
>>>>> +	__u32 memory;
>>>>> +	__u32 request_fd;
>>>>
>>>> This should be __s32, at least for consistency with dmabuf_fd?
>>>
>>> I see that in struct v4l2_buffer, we have __s32, I don't mind changing it
>>> to keep the consistency, I just don't see where this value can be a negative
>>> number.
>>
>> here
>> https://elixir.bootlin.com/linux/v5.8-rc4/source/drivers/media/common/videobuf2/videobuf2-v4l2.c#L134
> 
> I saw that -1 is used to signal an invalid value, but I was just wondering when request_fd = 0 is valid.

The request_fd is valid system wide file descriptor and request_fd = 0
is STDIN_FILENO thus IMO it is valid as far as we call it file descriptor.

> 
>>
>>>
>>>>
>>>>> +	struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
>>>>> +	__u32 reserved[4];
>>>>
>>>> I think we have to reserve more words here for future extensions.
>>>>
>>>> I'd like also to propose to add here __s32 metadata_fd. The idea behind
>>>> this is to have a way to pass per-frame metadata dmabuf buffers for
>>>> synchronous type of metadata where the metadata is coming at the same
>>>> time with data buffers. What would be the format of the metadata buffer
>>>> is TBD.
>>>>
>>>> One option for metadata buffer format could be:
>>>>
>>>> header {
>>>> 	num_ctrls
>>>> 	array_of_ctrls [0..N]
>>>> 		ctrl_id
>>>> 		ctrl_size
>>>> 		ctrl_offset
>>>> }
>>>>
>>>> data {
>>>> 	cid0	//offset of cid0 in dmabuf buffer
>>>> 	cid1
>>>> 	cidN
>>>> }
>>>
>>> Would it be better if, instead of adding a medatata_fd inside struct v4l2_ext_buffer,
>>> we create a new ioctl that gets this structs for the controls and sync them using the
>>> Request API ?

New ioctl means new syscall. There are use-cases where encoding
framerate is 480 fps (and more in near future, for example 960fps) this
means 480 more syscalls per second. I don't think this is optimal and
scalable solution at all.

>>
>> no, this solution has performance drawbacks when the metadata is big,
>> think of 64K.
> 
> Why? You could still use a dmabuf in this new ioctl, no?
> 
> 
> Regards,
> Helen
> 
>>
>>>
>>> I'd like to avoid too much metadata in the buffer object.
>>>
>>> Regards,
>>> Helen
>>>
>>>>
>>>> This will make easy to get concrete ctrl id without a need to parse the
>>>> whole metadata buffer. Also using dmabuf we don't need to copy data
>>>> between userspace <-> kernelspace (just cache syncs through
>>>> begin/end_cpu_access).
>>>>
>>>> The open question is who will validate the metadata buffer when it comes
>>>> from userspace. The obvious answer is v4l2-core but looking into DRM
>>>> subsytem they give more freedom to the drivers, and just provide generic
>>>> helpers which are not mandatory.
>>>>
>>>> I guess this will be a voice in the wilderness but I wanted to know your
>>>> opinion.
>>>>
>>>>> +};
>>>>> +
>>>>>  #ifndef __KERNEL__
>>>>>  /**
>>>>>   * v4l2_timeval_to_ns - Convert timeval to nanoseconds
>>>>> @@ -2520,6 +2581,29 @@ struct v4l2_create_buffers {
>>>>>  	__u32			reserved[6];
>>>>>  };
>>>>>  
>>>>> +/**
>>>>> + * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
>>>>> + * @index:	on return, index of the first created buffer
>>>>> + * @count:	entry: number of requested buffers,
>>>>> + *		return: number of created buffers
>>>>> + * @memory:	enum v4l2_memory; buffer memory type
>>>>> + * @capabilities: capabilities of this buffer type.
>>>>> + * @format:	frame format, for which buffers are requested
>>>>> + * @flags:	additional buffer management attributes (ignored unless the
>>>>> + *		queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
>>>>> + *		and configured for MMAP streaming I/O).
>>>>> + * @reserved:	extra space reserved for future fields, must be set to 0
>>>>> + */
>>>>> +struct v4l2_ext_create_buffers {
>>>>> +	__u32				index;
>>>>> +	__u32				count;
>>>>> +	__u32				memory;
>>>>> +	struct v4l2_ext_pix_format	format;
>>>>> +	__u32				capabilities;
>>>>> +	__u32				flags;
>>>>> +	__u32 reserved[4];
>>>>> +};
>>>>> +
>>>>>  /*
>>>>>   *	I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
>>>>>   *
>>>>> @@ -2623,6 +2707,11 @@ struct v4l2_create_buffers {
>>>>>  #define VIDIOC_G_EXT_PIX_FMT	_IOWR('V', 104, struct v4l2_ext_pix_format)
>>>>>  #define VIDIOC_S_EXT_PIX_FMT	_IOWR('V', 105, struct v4l2_ext_pix_format)
>>>>>  #define VIDIOC_TRY_EXT_PIX_FMT	_IOWR('V', 106, struct v4l2_ext_pix_format)
>>>>> +#define VIDIOC_EXT_CREATE_BUFS	_IOWR('V', 107, struct v4l2_ext_create_buffers)
>>>>> +#define VIDIOC_EXT_QUERYBUF	_IOWR('V', 108, struct v4l2_ext_buffer)
>>>>> +#define VIDIOC_EXT_QBUF		_IOWR('V', 109, struct v4l2_ext_buffer)
>>>>> +#define VIDIOC_EXT_DQBUF	_IOWR('V', 110, struct v4l2_ext_buffer)
>>>>> +#define VIDIOC_EXT_PREPARE_BUF	_IOWR('V', 111, struct v4l2_ext_buffer)
>>>>>  
>>>>>  /* Reminder: when adding new ioctls please add support for them to
>>>>>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
>>>>>
>>>>
>>

-- 
regards,
Stan

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v4 2/6] media: v4l2: Add extended buffer operations
  2020-07-24 13:16           ` Stanimir Varbanov
@ 2020-07-27 12:01             ` Helen Koike
  2020-07-28 18:02               ` Stanimir Varbanov
  2020-07-27 12:35             ` Tomasz Figa
  1 sibling, 1 reply; 29+ messages in thread
From: Helen Koike @ 2020-07-27 12:01 UTC (permalink / raw)
  To: Stanimir Varbanov, mchehab, hans.verkuil, laurent.pinchart,
	sakari.ailus, linux-media
  Cc: Boris Brezillon, tfiga, hiroh, nicolas, Brian.Starkey, kernel,
	narmstrong, linux-kernel, frkoenig, mjourdan



On 7/24/20 10:16 AM, Stanimir Varbanov wrote:
> 
> 
> On 7/21/20 5:40 PM, Helen Koike wrote:
>>
>>
>> On 7/21/20 11:30 AM, Stanimir Varbanov wrote:
>>> Hi Helen,
>>>
>>> On 7/21/20 4:54 PM, Helen Koike wrote:
>>>> Hi,
>>>>
>>>> On 7/21/20 8:26 AM, Stanimir Varbanov wrote:
>>>>>
>>>>>
>>>>> On 7/17/20 2:54 PM, Helen Koike wrote:
>>>>>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>>>>>
>>>>>> Those extended buffer ops have several purpose:
>>>>>> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
>>>>>>    the number of ns elapsed since 1970
>>>>>> 2/ Unify single/multiplanar handling
>>>>>> 3/ Add a new start offset field to each v4l2 plane buffer info struct
>>>>>>    to support the case where a single buffer object is storing all
>>>>>>    planes data, each one being placed at a different offset
>>>>>>
>>>>>> New hooks are created in v4l2_ioctl_ops so that drivers can start using
>>>>>> these new objects.
>>>>>>
>>>>>> The core takes care of converting new ioctls requests to old ones
>>>>>> if the driver does not support the new hooks, and vice versa.
>>>>>>
>>>>>> Note that the timecode field is gone, since there doesn't seem to be
>>>>>> in-kernel users. We can be added back in the reserved area if needed or
>>>>>> use the Request API to collect more metadata information from the
>>>>>> frame.
>>>>>>
>>>>>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>>>>>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
>>>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>>>>> ---
>>>>>> Changes in v4:
>>>>>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
>>>>>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
>>>>>> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
>>>>>> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
>>>>>> I think we can add this later, so I removed it from this RFC to simplify it.
>>>>>> - Remove num_planes field from struct v4l2_ext_buffer
>>>>>> - Add flags field to struct v4l2_ext_create_buffers
>>>>>> - Reformulate struct v4l2_ext_plane
>>>>>> - Fix some bugs caught by v4l2-compliance
>>>>>> - Rebased on top of media/master (post 5.8-rc1)
>>>>>>
>>>>>> Changes in v3:
>>>>>> - Rebased on top of media/master (post 5.4-rc1)
>>>>>>
>>>>>> Changes in v2:
>>>>>> - Add reserved space to v4l2_ext_buffer so that new fields can be added
>>>>>>   later on
>>>>>> ---
>>>>>>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
>>>>>>  drivers/media/v4l2-core/v4l2-ioctl.c | 349 +++++++++++++++++++++++++--
>>>>>>  include/media/v4l2-ioctl.h           |  26 ++
>>>>>>  include/uapi/linux/videodev2.h       |  89 +++++++
>>>>>>  4 files changed, 471 insertions(+), 22 deletions(-)
>>>>>>
>>>>>
>>>>> <cut>
>>>>>
>>>>>> +/**
>>>>>> + * struct v4l2_ext_plane - extended plane buffer info
>>>>>> + * @buffer_length:	size of the entire buffer in bytes, should fit
>>>>>> + *			@offset + @plane_length
>>>>>> + * @plane_length:	size of the plane in bytes.
>>>>>> + * @userptr:		when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
>>>>>> + *			to this plane.
>>>>>> + * @dmabuf_fd:		when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
>>>>>> + *			associated with this plane.
>>>>>> + * @offset:		offset in the memory buffer where the plane starts. If
>>>>>> + *			V4L2_MEMORY_MMAP is used, then it can be a "cookie" that
>>>>>> + *			should be passed to mmap() called on the video node.
>>>>>> + * @reserved:		extra space reserved for future fields, must be set to 0.
>>>>>> + *
>>>>>> + *
>>>>>> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
>>>>>> + * can have one plane for Y, and another for interleaved CbCr components.
>>>>>> + * Each plane can reside in a separate memory buffer, or even in
>>>>>> + * a completely separate memory node (e.g. in embedded devices).
>>>>>> + */
>>>>>> +struct v4l2_ext_plane {
>>>>>> +	__u32 buffer_length;
>>>>>> +	__u32 plane_length;
>>>>>> +	union {
>>>>>> +		__u64 userptr;
>>>>>> +		__s32 dmabuf_fd;
>>>>>> +	} m;
>>>>>> +	__u32 offset;
>>>>>> +	__u32 reserved[4];
>>>>>> +};
>>>>>> +
>>>>>>  /**
>>>>>>   * struct v4l2_buffer - video buffer info
>>>>>>   * @index:	id number of the buffer
>>>>>> @@ -1055,6 +1086,36 @@ struct v4l2_buffer {
>>>>>>  	};
>>>>>>  };
>>>>>>  
>>>>>> +/**
>>>>>> + * struct v4l2_ext_buffer - extended video buffer info
>>>>>> + * @index:	id number of the buffer
>>>>>> + * @type:	V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
>>>>>> + * @flags:	buffer informational flags
>>>>>> + * @field:	enum v4l2_field; field order of the image in the buffer
>>>>>> + * @timestamp:	frame timestamp
>>>>>> + * @sequence:	sequence count of this frame
>>>>>> + * @memory:	enum v4l2_memory; the method, in which the actual video data is
>>>>>> + *		passed
>>>>>> + * @planes:	per-plane buffer information
>>>>>> + * @request_fd:	fd of the request that this buffer should use
>>>>>> + * @reserved:	extra space reserved for future fields, must be set to 0
>>>>>> + *
>>>>>> + * Contains data exchanged by application and driver using one of the Streaming
>>>>>> + * I/O methods.
>>>>>> + */
>>>>>> +struct v4l2_ext_buffer {
>>>>>> +	__u32 index;
>>>>>> +	__u32 type;
>>>>>> +	__u32 flags;
>>>>>> +	__u32 field;
>>>>>> +	__u64 timestamp;
>>>>>> +	__u32 sequence;
>>>>>> +	__u32 memory;
>>>>>> +	__u32 request_fd;
>>>>>
>>>>> This should be __s32, at least for consistency with dmabuf_fd?
>>>>
>>>> I see that in struct v4l2_buffer, we have __s32, I don't mind changing it
>>>> to keep the consistency, I just don't see where this value can be a negative
>>>> number.
>>>
>>> here
>>> https://elixir.bootlin.com/linux/v5.8-rc4/source/drivers/media/common/videobuf2/videobuf2-v4l2.c#L134
>>
>> I saw that -1 is used to signal an invalid value, but I was just wondering when request_fd = 0 is valid.
> 
> The request_fd is valid system wide file descriptor and request_fd = 0
> is STDIN_FILENO thus IMO it is valid as far as we call it file descriptor.

Ack

> 
>>
>>>
>>>>
>>>>>
>>>>>> +	struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
>>>>>> +	__u32 reserved[4];
>>>>>
>>>>> I think we have to reserve more words here for future extensions.
>>>>>
>>>>> I'd like also to propose to add here __s32 metadata_fd. The idea behind
>>>>> this is to have a way to pass per-frame metadata dmabuf buffers for
>>>>> synchronous type of metadata where the metadata is coming at the same
>>>>> time with data buffers. What would be the format of the metadata buffer
>>>>> is TBD.
>>>>>
>>>>> One option for metadata buffer format could be:
>>>>>
>>>>> header {
>>>>> 	num_ctrls
>>>>> 	array_of_ctrls [0..N]
>>>>> 		ctrl_id
>>>>> 		ctrl_size
>>>>> 		ctrl_offset
>>>>> }
>>>>>
>>>>> data {
>>>>> 	cid0	//offset of cid0 in dmabuf buffer
>>>>> 	cid1
>>>>> 	cidN
>>>>> }
>>>>
>>>> Would it be better if, instead of adding a medatata_fd inside struct v4l2_ext_buffer,
>>>> we create a new ioctl that gets this structs for the controls and sync them using the
>>>> Request API ?
> 
> New ioctl means new syscall. There are use-cases where encoding
> framerate is 480 fps (and more in near future, for example 960fps) this
> means 480 more syscalls per second. I don't think this is optimal and
> scalable solution at all.

I feel we have a more general problem then.

What I propose is to leave reserved fields for now, and we can discuss how to include
this new feature in the future with a different RFC when we have a better view of requirements,
what do you think?

Thanks
Helen

> 
>>>
>>> no, this solution has performance drawbacks when the metadata is big,
>>> think of 64K.
>>
>> Why? You could still use a dmabuf in this new ioctl, no?
>>
>>
>> Regards,
>> Helen
>>
>>>
>>>>
>>>> I'd like to avoid too much metadata in the buffer object.
>>>>
>>>> Regards,
>>>> Helen
>>>>
>>>>>
>>>>> This will make easy to get concrete ctrl id without a need to parse the
>>>>> whole metadata buffer. Also using dmabuf we don't need to copy data
>>>>> between userspace <-> kernelspace (just cache syncs through
>>>>> begin/end_cpu_access).
>>>>>
>>>>> The open question is who will validate the metadata buffer when it comes
>>>>> from userspace. The obvious answer is v4l2-core but looking into DRM
>>>>> subsytem they give more freedom to the drivers, and just provide generic
>>>>> helpers which are not mandatory.
>>>>>
>>>>> I guess this will be a voice in the wilderness but I wanted to know your
>>>>> opinion.
>>>>>
>>>>>> +};
>>>>>> +
>>>>>>  #ifndef __KERNEL__
>>>>>>  /**
>>>>>>   * v4l2_timeval_to_ns - Convert timeval to nanoseconds
>>>>>> @@ -2520,6 +2581,29 @@ struct v4l2_create_buffers {
>>>>>>  	__u32			reserved[6];
>>>>>>  };
>>>>>>  
>>>>>> +/**
>>>>>> + * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
>>>>>> + * @index:	on return, index of the first created buffer
>>>>>> + * @count:	entry: number of requested buffers,
>>>>>> + *		return: number of created buffers
>>>>>> + * @memory:	enum v4l2_memory; buffer memory type
>>>>>> + * @capabilities: capabilities of this buffer type.
>>>>>> + * @format:	frame format, for which buffers are requested
>>>>>> + * @flags:	additional buffer management attributes (ignored unless the
>>>>>> + *		queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
>>>>>> + *		and configured for MMAP streaming I/O).
>>>>>> + * @reserved:	extra space reserved for future fields, must be set to 0
>>>>>> + */
>>>>>> +struct v4l2_ext_create_buffers {
>>>>>> +	__u32				index;
>>>>>> +	__u32				count;
>>>>>> +	__u32				memory;
>>>>>> +	struct v4l2_ext_pix_format	format;
>>>>>> +	__u32				capabilities;
>>>>>> +	__u32				flags;
>>>>>> +	__u32 reserved[4];
>>>>>> +};
>>>>>> +
>>>>>>  /*
>>>>>>   *	I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
>>>>>>   *
>>>>>> @@ -2623,6 +2707,11 @@ struct v4l2_create_buffers {
>>>>>>  #define VIDIOC_G_EXT_PIX_FMT	_IOWR('V', 104, struct v4l2_ext_pix_format)
>>>>>>  #define VIDIOC_S_EXT_PIX_FMT	_IOWR('V', 105, struct v4l2_ext_pix_format)
>>>>>>  #define VIDIOC_TRY_EXT_PIX_FMT	_IOWR('V', 106, struct v4l2_ext_pix_format)
>>>>>> +#define VIDIOC_EXT_CREATE_BUFS	_IOWR('V', 107, struct v4l2_ext_create_buffers)
>>>>>> +#define VIDIOC_EXT_QUERYBUF	_IOWR('V', 108, struct v4l2_ext_buffer)
>>>>>> +#define VIDIOC_EXT_QBUF		_IOWR('V', 109, struct v4l2_ext_buffer)
>>>>>> +#define VIDIOC_EXT_DQBUF	_IOWR('V', 110, struct v4l2_ext_buffer)
>>>>>> +#define VIDIOC_EXT_PREPARE_BUF	_IOWR('V', 111, struct v4l2_ext_buffer)
>>>>>>  
>>>>>>  /* Reminder: when adding new ioctls please add support for them to
>>>>>>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
>>>>>>
>>>>>
>>>
> 

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v4 2/6] media: v4l2: Add extended buffer operations
  2020-07-24 13:16           ` Stanimir Varbanov
  2020-07-27 12:01             ` Helen Koike
@ 2020-07-27 12:35             ` Tomasz Figa
  2020-07-28 18:08               ` Stanimir Varbanov
  1 sibling, 1 reply; 29+ messages in thread
From: Tomasz Figa @ 2020-07-27 12:35 UTC (permalink / raw)
  To: Stanimir Varbanov
  Cc: Helen Koike, Mauro Carvalho Chehab, Hans Verkuil,
	Laurent Pinchart, Sakari Ailus, Linux Media Mailing List,
	Boris Brezillon, Hirokazu Honda, Nicolas Dufresne, Brian Starkey,
	kernel, Neil Armstrong, Linux Kernel Mailing List, frkoenig,
	Maxime Jourdan

Hi Stanimir,

On Fri, Jul 24, 2020 at 3:17 PM Stanimir Varbanov
<stanimir.varbanov@linaro.org> wrote:
>
>
>
> On 7/21/20 5:40 PM, Helen Koike wrote:
> >
> >
> > On 7/21/20 11:30 AM, Stanimir Varbanov wrote:
> >> Hi Helen,
> >>
> >> On 7/21/20 4:54 PM, Helen Koike wrote:
> >>> Hi,
> >>>
> >>> On 7/21/20 8:26 AM, Stanimir Varbanov wrote:
> >>>>
> >>>>
> >>>> On 7/17/20 2:54 PM, Helen Koike wrote:
> >>>>> From: Hans Verkuil <hans.verkuil@cisco.com>
> >>>>>
> >>>>> Those extended buffer ops have several purpose:
> >>>>> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
> >>>>>    the number of ns elapsed since 1970
> >>>>> 2/ Unify single/multiplanar handling
> >>>>> 3/ Add a new start offset field to each v4l2 plane buffer info struct
> >>>>>    to support the case where a single buffer object is storing all
> >>>>>    planes data, each one being placed at a different offset
> >>>>>
> >>>>> New hooks are created in v4l2_ioctl_ops so that drivers can start using
> >>>>> these new objects.
> >>>>>
> >>>>> The core takes care of converting new ioctls requests to old ones
> >>>>> if the driver does not support the new hooks, and vice versa.
> >>>>>
> >>>>> Note that the timecode field is gone, since there doesn't seem to be
> >>>>> in-kernel users. We can be added back in the reserved area if needed or
> >>>>> use the Request API to collect more metadata information from the
> >>>>> frame.
> >>>>>
> >>>>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> >>>>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
> >>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> >>>>> ---
> >>>>> Changes in v4:
> >>>>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
> >>>>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
> >>>>> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
> >>>>> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
> >>>>> I think we can add this later, so I removed it from this RFC to simplify it.
> >>>>> - Remove num_planes field from struct v4l2_ext_buffer
> >>>>> - Add flags field to struct v4l2_ext_create_buffers
> >>>>> - Reformulate struct v4l2_ext_plane
> >>>>> - Fix some bugs caught by v4l2-compliance
> >>>>> - Rebased on top of media/master (post 5.8-rc1)
> >>>>>
> >>>>> Changes in v3:
> >>>>> - Rebased on top of media/master (post 5.4-rc1)
> >>>>>
> >>>>> Changes in v2:
> >>>>> - Add reserved space to v4l2_ext_buffer so that new fields can be added
> >>>>>   later on
> >>>>> ---
> >>>>>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
> >>>>>  drivers/media/v4l2-core/v4l2-ioctl.c | 349 +++++++++++++++++++++++++--
> >>>>>  include/media/v4l2-ioctl.h           |  26 ++
> >>>>>  include/uapi/linux/videodev2.h       |  89 +++++++
> >>>>>  4 files changed, 471 insertions(+), 22 deletions(-)
> >>>>>
> >>>>
> >>>> <cut>
> >>>>
> >>>>> +/**
> >>>>> + * struct v4l2_ext_plane - extended plane buffer info
> >>>>> + * @buffer_length:       size of the entire buffer in bytes, should fit
> >>>>> + *                       @offset + @plane_length
> >>>>> + * @plane_length:        size of the plane in bytes.
> >>>>> + * @userptr:             when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
> >>>>> + *                       to this plane.
> >>>>> + * @dmabuf_fd:           when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
> >>>>> + *                       associated with this plane.
> >>>>> + * @offset:              offset in the memory buffer where the plane starts. If
> >>>>> + *                       V4L2_MEMORY_MMAP is used, then it can be a "cookie" that
> >>>>> + *                       should be passed to mmap() called on the video node.
> >>>>> + * @reserved:            extra space reserved for future fields, must be set to 0.
> >>>>> + *
> >>>>> + *
> >>>>> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
> >>>>> + * can have one plane for Y, and another for interleaved CbCr components.
> >>>>> + * Each plane can reside in a separate memory buffer, or even in
> >>>>> + * a completely separate memory node (e.g. in embedded devices).
> >>>>> + */
> >>>>> +struct v4l2_ext_plane {
> >>>>> + __u32 buffer_length;
> >>>>> + __u32 plane_length;
> >>>>> + union {
> >>>>> +         __u64 userptr;
> >>>>> +         __s32 dmabuf_fd;
> >>>>> + } m;
> >>>>> + __u32 offset;
> >>>>> + __u32 reserved[4];
> >>>>> +};
> >>>>> +
> >>>>>  /**
> >>>>>   * struct v4l2_buffer - video buffer info
> >>>>>   * @index:       id number of the buffer
> >>>>> @@ -1055,6 +1086,36 @@ struct v4l2_buffer {
> >>>>>   };
> >>>>>  };
> >>>>>
> >>>>> +/**
> >>>>> + * struct v4l2_ext_buffer - extended video buffer info
> >>>>> + * @index:       id number of the buffer
> >>>>> + * @type:        V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
> >>>>> + * @flags:       buffer informational flags
> >>>>> + * @field:       enum v4l2_field; field order of the image in the buffer
> >>>>> + * @timestamp:   frame timestamp
> >>>>> + * @sequence:    sequence count of this frame
> >>>>> + * @memory:      enum v4l2_memory; the method, in which the actual video data is
> >>>>> + *               passed
> >>>>> + * @planes:      per-plane buffer information
> >>>>> + * @request_fd:  fd of the request that this buffer should use
> >>>>> + * @reserved:    extra space reserved for future fields, must be set to 0
> >>>>> + *
> >>>>> + * Contains data exchanged by application and driver using one of the Streaming
> >>>>> + * I/O methods.
> >>>>> + */
> >>>>> +struct v4l2_ext_buffer {
> >>>>> + __u32 index;
> >>>>> + __u32 type;
> >>>>> + __u32 flags;
> >>>>> + __u32 field;
> >>>>> + __u64 timestamp;
> >>>>> + __u32 sequence;
> >>>>> + __u32 memory;
> >>>>> + __u32 request_fd;
> >>>>
> >>>> This should be __s32, at least for consistency with dmabuf_fd?
> >>>
> >>> I see that in struct v4l2_buffer, we have __s32, I don't mind changing it
> >>> to keep the consistency, I just don't see where this value can be a negative
> >>> number.
> >>
> >> here
> >> https://elixir.bootlin.com/linux/v5.8-rc4/source/drivers/media/common/videobuf2/videobuf2-v4l2.c#L134
> >
> > I saw that -1 is used to signal an invalid value, but I was just wondering when request_fd = 0 is valid.
>
> The request_fd is valid system wide file descriptor and request_fd = 0
> is STDIN_FILENO thus IMO it is valid as far as we call it file descriptor.
>
> >
> >>
> >>>
> >>>>
> >>>>> + struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
> >>>>> + __u32 reserved[4];
> >>>>
> >>>> I think we have to reserve more words here for future extensions.
> >>>>
> >>>> I'd like also to propose to add here __s32 metadata_fd. The idea behind
> >>>> this is to have a way to pass per-frame metadata dmabuf buffers for
> >>>> synchronous type of metadata where the metadata is coming at the same
> >>>> time with data buffers. What would be the format of the metadata buffer
> >>>> is TBD.
> >>>>
> >>>> One option for metadata buffer format could be:
> >>>>
> >>>> header {
> >>>>    num_ctrls
> >>>>    array_of_ctrls [0..N]
> >>>>            ctrl_id
> >>>>            ctrl_size
> >>>>            ctrl_offset
> >>>> }
> >>>>
> >>>> data {
> >>>>    cid0    //offset of cid0 in dmabuf buffer
> >>>>    cid1
> >>>>    cidN
> >>>> }
> >>>
> >>> Would it be better if, instead of adding a medatata_fd inside struct v4l2_ext_buffer,
> >>> we create a new ioctl that gets this structs for the controls and sync them using the
> >>> Request API ?
>
> New ioctl means new syscall. There are use-cases where encoding
> framerate is 480 fps (and more in near future, for example 960fps) this
> means 480 more syscalls per second. I don't think this is optimal and
> scalable solution at all.
>

Do you happen to have some data to confirm that it's indeed a problem?

Best regards,
Tomasz

> >>
> >> no, this solution has performance drawbacks when the metadata is big,
> >> think of 64K.
> >
> > Why? You could still use a dmabuf in this new ioctl, no?
> >
> >
> > Regards,
> > Helen
> >
> >>
> >>>
> >>> I'd like to avoid too much metadata in the buffer object.
> >>>
> >>> Regards,
> >>> Helen
> >>>
> >>>>
> >>>> This will make easy to get concrete ctrl id without a need to parse the
> >>>> whole metadata buffer. Also using dmabuf we don't need to copy data
> >>>> between userspace <-> kernelspace (just cache syncs through
> >>>> begin/end_cpu_access).
> >>>>
> >>>> The open question is who will validate the metadata buffer when it comes
> >>>> from userspace. The obvious answer is v4l2-core but looking into DRM
> >>>> subsytem they give more freedom to the drivers, and just provide generic
> >>>> helpers which are not mandatory.
> >>>>
> >>>> I guess this will be a voice in the wilderness but I wanted to know your
> >>>> opinion.
> >>>>
> >>>>> +};
> >>>>> +
> >>>>>  #ifndef __KERNEL__
> >>>>>  /**
> >>>>>   * v4l2_timeval_to_ns - Convert timeval to nanoseconds
> >>>>> @@ -2520,6 +2581,29 @@ struct v4l2_create_buffers {
> >>>>>   __u32                   reserved[6];
> >>>>>  };
> >>>>>
> >>>>> +/**
> >>>>> + * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
> >>>>> + * @index:       on return, index of the first created buffer
> >>>>> + * @count:       entry: number of requested buffers,
> >>>>> + *               return: number of created buffers
> >>>>> + * @memory:      enum v4l2_memory; buffer memory type
> >>>>> + * @capabilities: capabilities of this buffer type.
> >>>>> + * @format:      frame format, for which buffers are requested
> >>>>> + * @flags:       additional buffer management attributes (ignored unless the
> >>>>> + *               queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
> >>>>> + *               and configured for MMAP streaming I/O).
> >>>>> + * @reserved:    extra space reserved for future fields, must be set to 0
> >>>>> + */
> >>>>> +struct v4l2_ext_create_buffers {
> >>>>> + __u32                           index;
> >>>>> + __u32                           count;
> >>>>> + __u32                           memory;
> >>>>> + struct v4l2_ext_pix_format      format;
> >>>>> + __u32                           capabilities;
> >>>>> + __u32                           flags;
> >>>>> + __u32 reserved[4];
> >>>>> +};
> >>>>> +
> >>>>>  /*
> >>>>>   *       I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
> >>>>>   *
> >>>>> @@ -2623,6 +2707,11 @@ struct v4l2_create_buffers {
> >>>>>  #define VIDIOC_G_EXT_PIX_FMT     _IOWR('V', 104, struct v4l2_ext_pix_format)
> >>>>>  #define VIDIOC_S_EXT_PIX_FMT     _IOWR('V', 105, struct v4l2_ext_pix_format)
> >>>>>  #define VIDIOC_TRY_EXT_PIX_FMT   _IOWR('V', 106, struct v4l2_ext_pix_format)
> >>>>> +#define VIDIOC_EXT_CREATE_BUFS   _IOWR('V', 107, struct v4l2_ext_create_buffers)
> >>>>> +#define VIDIOC_EXT_QUERYBUF      _IOWR('V', 108, struct v4l2_ext_buffer)
> >>>>> +#define VIDIOC_EXT_QBUF          _IOWR('V', 109, struct v4l2_ext_buffer)
> >>>>> +#define VIDIOC_EXT_DQBUF _IOWR('V', 110, struct v4l2_ext_buffer)
> >>>>> +#define VIDIOC_EXT_PREPARE_BUF   _IOWR('V', 111, struct v4l2_ext_buffer)
> >>>>>
> >>>>>  /* Reminder: when adding new ioctls please add support for them to
> >>>>>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
> >>>>>
> >>>>
> >>
>
> --
> regards,
> Stan

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v4 1/6] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)
  2020-07-21 10:37   ` Hans Verkuil
@ 2020-07-28 15:18     ` Helen Koike
  2020-07-28 15:30       ` Hans Verkuil
  0 siblings, 1 reply; 29+ messages in thread
From: Helen Koike @ 2020-07-28 15:18 UTC (permalink / raw)
  To: Hans Verkuil, mchehab, hans.verkuil, laurent.pinchart,
	sakari.ailus, linux-media
  Cc: Boris Brezillon, tfiga, hiroh, nicolas, Brian.Starkey, kernel,
	narmstrong, linux-kernel, frkoenig, mjourdan, stanimir.varbanov

Hi Hans,

On 7/21/20 7:37 AM, Hans Verkuil wrote:
> On 17/07/2020 13:54, Helen Koike wrote:
>> From: Boris Brezillon <boris.brezillon@collabora.com>
>>
>> This is part of the multiplanar and singleplanar unification process.
>> v4l2_ext_pix_format is supposed to work for both cases.
>>
>> We also add the concept of modifiers already employed in DRM to expose
>> HW-specific formats (like tiled or compressed formats) and allow
>> exchanging this information with the DRM subsystem in a consistent way.
>>
>> Note that only V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] are accepted in
>> v4l2_ext_format, other types will be rejected if you use the {G,S,TRY}EXT_FMT
>> ioctls.
>>
>> New hooks have been added to v4l2_ioctl_ops to support those new ioctls
>> in drivers, but, in the meantime, the core takes care of converting
>> {S,G,TRY}_EXT_FMT requests into {S,G,TRY}_FMT so that old drivers can
>> still work if the userspace app/lib uses the new ioctls.
>> The conversion is also done the other around to allow userspace
>> apps/libs using {S,G,TRY}_FMT to work with drivers implementing the
>> _ext_ hooks.
>>
>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>> ---
>> Changes in v4:
>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
>> - Add reserved fields
>> - Removed num_planes from struct v4l2_ext_pix_format
>> - Removed flag field from struct v4l2_ext_pix_format, since the only
>>   defined value is V4L2_PIX_FMT_FLAG_PREMUL_ALPHA only used by vsp1,
>>   where we can use modifiers, or add it back later through the reserved
>>   bits.
>> - In v4l2_ext_format_to_format(), check if modifier is != MOD_LINEAR &&
>>   != MOD_INVALID
>> - Fix type assignment in v4l_g_fmt_ext_pix()
>> - Rebased on top of media/master (post 5.8-rc1)
>>
>> Changes in v3:
>> - Rebased on top of media/master (post 5.4-rc1)
>>
>> Changes in v2:
>> - Move the modifier in v4l2_ext_format (was formerly placed in
>>   v4l2_ext_plane)
>> - Fix a few bugs in the converters and add a strict parameter to
>>   allow conversion of uninitialized/mis-initialized objects
>> ---
>>  drivers/media/v4l2-core/v4l2-dev.c   |  21 +-
>>  drivers/media/v4l2-core/v4l2-ioctl.c | 585 +++++++++++++++++++++++----
>>  include/media/v4l2-ioctl.h           |  34 ++
>>  include/uapi/linux/videodev2.h       |  55 +++
>>  4 files changed, 614 insertions(+), 81 deletions(-)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
>> index a593ea0598b55..e1829906bc086 100644
>> --- a/drivers/media/v4l2-core/v4l2-dev.c
>> +++ b/drivers/media/v4l2-core/v4l2-dev.c
>> @@ -607,25 +607,37 @@ static void determine_valid_ioctls(struct video_device *vdev)
>>  			set_bit(_IOC_NR(VIDIOC_ENUM_FMT), valid_ioctls);
>>  		if ((is_rx && (ops->vidioc_g_fmt_vid_cap ||
>>  			       ops->vidioc_g_fmt_vid_cap_mplane ||
>> +			       ops->vidioc_g_ext_pix_fmt_vid_cap ||
>>  			       ops->vidioc_g_fmt_vid_overlay)) ||
>>  		    (is_tx && (ops->vidioc_g_fmt_vid_out ||
>>  			       ops->vidioc_g_fmt_vid_out_mplane ||
>> -			       ops->vidioc_g_fmt_vid_out_overlay)))
>> +			       ops->vidioc_g_ext_pix_fmt_vid_out ||
>> +			       ops->vidioc_g_fmt_vid_out_overlay))) {
>>  			 set_bit(_IOC_NR(VIDIOC_G_FMT), valid_ioctls);
>> +			 set_bit(_IOC_NR(VIDIOC_G_EXT_PIX_FMT), valid_ioctls);
>> +		}
>>  		if ((is_rx && (ops->vidioc_s_fmt_vid_cap ||
>>  			       ops->vidioc_s_fmt_vid_cap_mplane ||
>> +			       ops->vidioc_s_ext_pix_fmt_vid_cap ||
>>  			       ops->vidioc_s_fmt_vid_overlay)) ||
>>  		    (is_tx && (ops->vidioc_s_fmt_vid_out ||
>>  			       ops->vidioc_s_fmt_vid_out_mplane ||
>> -			       ops->vidioc_s_fmt_vid_out_overlay)))
>> +			       ops->vidioc_s_ext_pix_fmt_vid_out ||
>> +			       ops->vidioc_s_fmt_vid_out_overlay))) {
>>  			 set_bit(_IOC_NR(VIDIOC_S_FMT), valid_ioctls);
>> +			 set_bit(_IOC_NR(VIDIOC_S_EXT_PIX_FMT), valid_ioctls);
>> +		}
>>  		if ((is_rx && (ops->vidioc_try_fmt_vid_cap ||
>>  			       ops->vidioc_try_fmt_vid_cap_mplane ||
>> +			       ops->vidioc_try_ext_pix_fmt_vid_cap ||
>>  			       ops->vidioc_try_fmt_vid_overlay)) ||
>>  		    (is_tx && (ops->vidioc_try_fmt_vid_out ||
>>  			       ops->vidioc_try_fmt_vid_out_mplane ||
>> -			       ops->vidioc_try_fmt_vid_out_overlay)))
>> +			       ops->vidioc_try_ext_pix_fmt_vid_out ||
>> +			       ops->vidioc_try_fmt_vid_out_overlay))) {
>>  			 set_bit(_IOC_NR(VIDIOC_TRY_FMT), valid_ioctls);
>> +			 set_bit(_IOC_NR(VIDIOC_TRY_EXT_PIX_FMT), valid_ioctls);
>> +		}
>>  		SET_VALID_IOCTL(ops, VIDIOC_OVERLAY, vidioc_overlay);
>>  		SET_VALID_IOCTL(ops, VIDIOC_G_FBUF, vidioc_g_fbuf);
>>  		SET_VALID_IOCTL(ops, VIDIOC_S_FBUF, vidioc_s_fbuf);
>> @@ -682,8 +694,11 @@ static void determine_valid_ioctls(struct video_device *vdev)
>>  		/* touch specific ioctls */
>>  		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FMT, vidioc_enum_fmt_vid_cap);
>>  		SET_VALID_IOCTL(ops, VIDIOC_G_FMT, vidioc_g_fmt_vid_cap);
>> +		SET_VALID_IOCTL(ops, VIDIOC_G_EXT_PIX_FMT, vidioc_g_fmt_vid_cap);
>>  		SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_vid_cap);
>> +		SET_VALID_IOCTL(ops, VIDIOC_S_EXT_PIX_FMT, vidioc_s_fmt_vid_cap);
>>  		SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_vid_cap);
>> +		SET_VALID_IOCTL(ops, VIDIOC_TRY_EXT_PIX_FMT, vidioc_try_fmt_vid_cap);
>>  		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMESIZES, vidioc_enum_framesizes);
>>  		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMEINTERVALS, vidioc_enum_frameintervals);
>>  		SET_VALID_IOCTL(ops, VIDIOC_ENUMINPUT, vidioc_enum_input);
>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
>> index 02bfef0da76da..3b77433f6c32b 100644
>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>> @@ -17,6 +17,8 @@
>>  
>>  #include <linux/videodev2.h>
>>  
>> +#include <drm/drm_fourcc.h>
>> +
>>  #include <media/v4l2-common.h>
>>  #include <media/v4l2-ioctl.h>
>>  #include <media/v4l2-ctrls.h>
>> @@ -378,6 +380,27 @@ static void v4l_print_format(const void *arg, bool write_only)
>>  	}
>>  }
>>  
>> +static void v4l_print_ext_pix_format(const void *arg, bool write_only)
>> +{
>> +	const struct v4l2_ext_pix_format *pix = arg;
>> +	unsigned int i;
>> +
>> +	pr_cont("type=%s, width=%u, height=%u, format=%c%c%c%c, modifier %llx, field=%s, colorspace=%d, ycbcr_enc=%u, quantization=%u, xfer_func=%u\n",
>> +		prt_names(pix->type, v4l2_type_names),
>> +		pix->width, pix->height,
>> +		(pix->pixelformat & 0xff),
>> +		(pix->pixelformat >>  8) & 0xff,
>> +		(pix->pixelformat >> 16) & 0xff,
>> +		(pix->pixelformat >> 24) & 0xff,
>> +		pix->modifier, prt_names(pix->field, v4l2_field_names),
>> +		pix->colorspace, pix->ycbcr_enc,
>> +		pix->quantization, pix->xfer_func);
>> +	for (i = 0; i < VIDEO_MAX_PLANES && pix->plane_fmt[i].sizeimage; i++)
>> +		pr_debug("plane %u: bytesperline=%u sizeimage=%u\n",
>> +			 i, pix->plane_fmt[i].bytesperline,
>> +			 pix->plane_fmt[i].sizeimage);
>> +}
>> +
>>  static void v4l_print_framebuffer(const void *arg, bool write_only)
>>  {
>>  	const struct v4l2_framebuffer *p = arg;
>> @@ -958,11 +981,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>>  	switch (type) {
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>>  		if ((is_vid || is_tch) && is_rx &&
>> -		    (ops->vidioc_g_fmt_vid_cap || ops->vidioc_g_fmt_vid_cap_mplane))
>> +		    (ops->vidioc_g_fmt_vid_cap ||
>> +		     ops->vidioc_g_ext_pix_fmt_vid_cap ||
>> +		     ops->vidioc_g_fmt_vid_cap_mplane))
>>  			return 0;
>>  		break;
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> -		if ((is_vid || is_tch) && is_rx && ops->vidioc_g_fmt_vid_cap_mplane)
>> +		if ((is_vid || is_tch) && is_rx &&
>> +		    (ops->vidioc_g_fmt_vid_cap_mplane ||
>> +		     ops->vidioc_g_ext_pix_fmt_vid_cap))
>>  			return 0;
>>  		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>> @@ -971,11 +998,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>>  		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>>  		if (is_vid && is_tx &&
>> -		    (ops->vidioc_g_fmt_vid_out || ops->vidioc_g_fmt_vid_out_mplane))
>> +		    (ops->vidioc_g_fmt_vid_out ||
>> +		     ops->vidioc_g_ext_pix_fmt_vid_out ||
>> +		     ops->vidioc_g_fmt_vid_out_mplane))
>>  			return 0;
>>  		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> -		if (is_vid && is_tx && ops->vidioc_g_fmt_vid_out_mplane)
>> +		if (is_vid && is_tx &&
>> +		    (ops->vidioc_g_ext_pix_fmt_vid_out ||
>> +		     ops->vidioc_g_fmt_vid_out_mplane))
>>  			return 0;
>>  		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>> @@ -1055,6 +1086,134 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>>  	       sizeof(fmt->fmt.pix) - offset);
>>  }
>>  
>> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
>> +				  struct v4l2_format *f, bool mplane_cap,
>> +				  bool strict)
>> +{
>> +	const struct v4l2_plane_ext_pix_format *pe;
>> +	struct v4l2_plane_pix_format *p;
>> +	unsigned int i;
>> +
>> +	memset(f, 0, sizeof(*f));
>> +
>> +	/*
>> +	 * Make sure no modifier is required before doing the
>> +	 * conversion.
>> +	 */
>> +	if (e->modifier && strict &&
>> +	    e->modifier != DRM_FORMAT_MOD_LINEAR &&
>> +	    e->modifier != DRM_FORMAT_MOD_INVALID)
>> +		return -EINVAL;
>> +
>> +	if (!e->plane_fmt[0].sizeimage && strict)
>> +		return -EINVAL;
>> +
>> +	if (e->plane_fmt[1].sizeimage && !mplane_cap && strict)
>> +		return 0;
>> +
>> +	if (!mplane_cap) {
>> +		f->fmt.pix.width = e->width;
>> +		f->fmt.pix.height = e->height;
>> +		f->fmt.pix.pixelformat = e->pixelformat;
>> +		f->fmt.pix.field = e->field;
>> +		f->fmt.pix.colorspace = e->colorspace;
>> +		f->fmt.pix.ycbcr_enc = e->ycbcr_enc;
>> +		f->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +		f->fmt.pix.quantization = e->quantization;
>> +		pe = &e->plane_fmt[0];
>> +		f->fmt.pix.bytesperline = pe->bytesperline;
>> +		f->fmt.pix.sizeimage = pe->sizeimage;
>> +		f->type = e->type;
>> +		return 0;
>> +	}
>> +
>> +	f->fmt.pix_mp.width = e->width;
>> +	f->fmt.pix_mp.height = e->height;
>> +	f->fmt.pix_mp.pixelformat = e->pixelformat;
>> +	f->fmt.pix_mp.field = e->field;
>> +	f->fmt.pix_mp.colorspace = e->colorspace;
>> +	f->fmt.pix_mp.ycbcr_enc = e->ycbcr_enc;
>> +	f->fmt.pix_mp.quantization = e->quantization;
>> +	if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
>> +		f->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
>> +	else
>> +		f->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
>> +
>> +	for (i = 0; i < VIDEO_MAX_PLANES; i++) {
>> +		pe = &e->plane_fmt[i];
>> +		p = &f->fmt.pix_mp.plane_fmt[i];
>> +		p->bytesperline = pe->bytesperline;
>> +		p->sizeimage = pe->sizeimage;
>> +	}
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_ext_pix_format_to_format);
>> +
>> +int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
>> +				  struct v4l2_ext_pix_format *e, bool strict)
>> +{
>> +	const struct v4l2_plane_pix_format *p;
>> +	struct v4l2_plane_ext_pix_format *pe;
>> +	unsigned int i;
>> +
>> +	memset(e, 0, sizeof(*e));
>> +
>> +	switch (f->type) {
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +		e->width = f->fmt.pix.width;
>> +		e->height = f->fmt.pix.height;
>> +		e->pixelformat = f->fmt.pix.pixelformat;
>> +		e->field = f->fmt.pix.field;
>> +		e->colorspace = f->fmt.pix.colorspace;
>> +		if (f->fmt.pix.flags)
>> +			pr_warn("Ignoring pixelformat flags 0x%x\n",
>> +				f->fmt.pix.flags);
>> +		e->ycbcr_enc = f->fmt.pix.ycbcr_enc;
>> +		e->quantization = f->fmt.pix.quantization;
>> +		e->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
>> +		e->plane_fmt[0].sizeimage = f->fmt.pix.sizeimage;
>> +		e->type = f->type;
>> +		break;
>> +
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> +		if ((f->fmt.pix_mp.num_planes > VIDEO_MAX_PLANES ||
>> +		     !f->fmt.pix_mp.num_planes) && strict)
>> +			return -EINVAL;
>> +
>> +		e->width = f->fmt.pix_mp.width;
>> +		e->height = f->fmt.pix_mp.height;
>> +		e->pixelformat = f->fmt.pix_mp.pixelformat;
>> +		e->field = f->fmt.pix_mp.field;
>> +		e->colorspace = f->fmt.pix_mp.colorspace;
>> +		if (f->fmt.pix.flags)
>> +			pr_warn("Ignoring pixelformat flags 0x%x\n",
>> +				f->fmt.pix.flags);
>> +		e->ycbcr_enc = f->fmt.pix_mp.ycbcr_enc;
>> +		e->quantization = f->fmt.pix_mp.quantization;
>> +		if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
>> +			e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
>> +		else
>> +			e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
>> +
>> +		for (i = 0; i < VIDEO_MAX_PLANES; i++) {
>> +			pe = &e->plane_fmt[i];
>> +			p = &f->fmt.pix_mp.plane_fmt[i];
>> +			pe->bytesperline = p->bytesperline;
>> +			pe->sizeimage = p->sizeimage;
>> +		}
>> +		break;
>> +
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_format_to_ext_pix_format);
>> +
>>  static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
>>  				struct file *file, void *fh, void *arg)
>>  {
>> @@ -1558,6 +1717,38 @@ static void v4l_pix_format_touch(struct v4l2_pix_format *p)
>>  	p->xfer_func = 0;
>>  }
>>  
>> +static int v4l_g_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
>> +			     struct file *file, void *fh,
>> +			     struct v4l2_format *f)
>> +{
>> +	struct v4l2_ext_pix_format ef;
>> +	int ret;
>> +
>> +	switch (f->type) {
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> +		ef.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
>> +		ret = ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, &ef);
>> +		break;
>> +
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> +		ef.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
>> +		ret = ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, &ef);
>> +		break;
>> +
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	if (ret)
>> +		return ret;
>> +
>> +	return v4l2_ext_pix_format_to_format(&ef, f,
>> +					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
>> +					     true);
>> +}
>> +
>>  static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>>  				struct file *file, void *fh, void *arg)
>>  {
>> @@ -1594,17 +1785,27 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>>  
>>  	switch (p->type) {
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> -		if (unlikely(!ops->vidioc_g_fmt_vid_cap))
>> -			break;
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -		ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
>> -		/* just in case the driver zeroed it again */
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -		if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> -			v4l_pix_format_touch(&p->fmt.pix);
>> -		return ret;
>> +		if (ops->vidioc_g_fmt_vid_cap) {
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +			ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
>> +			/* just in case the driver zeroed it again */
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> +				v4l_pix_format_touch(&p->fmt.pix);
>> +			return ret;
>> +		} else if (ops->vidioc_g_ext_pix_fmt_vid_cap) {
>> +			ret = v4l_g_fmt_ext_pix(ops, file, fh, p);
>> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> +				v4l_pix_format_touch(&p->fmt.pix);
>> +			return ret;
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> -		return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
>> +		if (ops->vidioc_g_fmt_vid_cap_mplane)
>> +			return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
>> +		else if (ops->vidioc_g_ext_pix_fmt_vid_cap)
>> +			return v4l_g_fmt_ext_pix(ops, file, fh, p);
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>>  		return ops->vidioc_g_fmt_vid_overlay(file, fh, arg);
>>  	case V4L2_BUF_TYPE_VBI_CAPTURE:
>> @@ -1612,15 +1813,22 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>>  	case V4L2_BUF_TYPE_SLICED_VBI_CAPTURE:
>>  		return ops->vidioc_g_fmt_sliced_vbi_cap(file, fh, arg);
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> -		if (unlikely(!ops->vidioc_g_fmt_vid_out))
>> -			break;
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -		ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
>> -		/* just in case the driver zeroed it again */
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -		return ret;
>> +		if (ops->vidioc_g_fmt_vid_out) {
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +			ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
>> +			/* just in case the driver zeroed it again */
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +			return ret;
>> +		} else if (ops->vidioc_g_ext_pix_fmt_vid_out) {
>> +			return v4l_g_fmt_ext_pix(ops, file, fh, p);
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> -		return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
>> +		if (ops->vidioc_g_fmt_vid_out_mplane)
>> +			return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
>> +		else if (ops->vidioc_g_ext_pix_fmt_vid_out)
>> +			return v4l_g_fmt_ext_pix(ops, file, fh, p);
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>>  		return ops->vidioc_g_fmt_vid_out_overlay(file, fh, arg);
>>  	case V4L2_BUF_TYPE_VBI_OUTPUT:
>> @@ -1639,6 +1847,76 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>>  	return -EINVAL;
>>  }
>>  
>> +static int v4l_g_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
>> +			     struct file *file, void *fh, void *arg)
>> +{
>> +	struct v4l2_ext_pix_format *ef = arg;
>> +	struct v4l2_format f = {
>> +		.type = ef->type,
>> +	};
>> +	int ret;
>> +
>> +	ret = check_fmt(file, ef->type);
>> +	if (ret)
>> +		return ret;
>> +
>> +	memset(ef, 0, sizeof(*ef));
>> +	ef->type = f.type;
>> +
>> +	switch (f.type) {
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +		if (ops->vidioc_g_ext_pix_fmt_vid_cap)
>> +			return ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, ef);
>> +		break;
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +		if (ops->vidioc_g_ext_pix_fmt_vid_out)
>> +			return ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, ef);
>> +		break;
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	ret = v4l_g_fmt(ops, file, fh, &f);
>> +	if (ret)
>> +		return ret;
>> +
>> +	return v4l2_format_to_ext_pix_format(&f, ef, true);
>> +}
>> +
>> +static int v4l_s_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
>> +			     struct file *file, void *fh,
>> +			     struct v4l2_format *f)
>> +{
>> +	struct v4l2_ext_pix_format ef;
>> +	int ret;
>> +
>> +	ret = v4l2_format_to_ext_pix_format(f, &ef, false);
>> +	if (ret)
>> +		return ret;
>> +
>> +	switch (f->type) {
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> +		ret = ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, &ef);
>> +		break;
>> +
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> +		ret = ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, &ef);
>> +		break;
>> +
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	if (ret)
>> +		return ret;
>> +
>> +	return v4l2_ext_pix_format_to_format(&ef, f,
>> +					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
>> +					     true);
>> +}
>> +
>>  static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>>  				struct file *file, void *fh, void *arg)
>>  {
>> @@ -1657,23 +1935,31 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>>  
>>  	switch (p->type) {
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> -		if (unlikely(!ops->vidioc_s_fmt_vid_cap))
>> +		if (ops->vidioc_s_fmt_vid_cap) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix);
>> +			ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
>> +			/* just in case the driver zeroed it again */
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
>> +			ret = v4l_s_fmt_ext_pix(ops, file, fh, arg);
>> +		} else {
>>  			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix);
>> -		ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
>> -		/* just in case the driver zeroed it again */
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +		}
>> +
>>  		if (vfd->vfl_type == VFL_TYPE_TOUCH)
>>  			v4l_pix_format_touch(&p->fmt.pix);
>>  		return ret;
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> -		if (unlikely(!ops->vidioc_s_fmt_vid_cap_mplane))
>> -			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> -					  bytesperline);
>> -		return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
>> +		if (ops->vidioc_s_fmt_vid_cap_mplane) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> +						  bytesperline);
>> +			return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
>> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
>> +			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>>  		if (unlikely(!ops->vidioc_s_fmt_vid_overlay))
>>  			break;
>> @@ -1690,21 +1976,27 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>>  		CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
>>  		return ops->vidioc_s_fmt_sliced_vbi_cap(file, fh, arg);
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> -		if (unlikely(!ops->vidioc_s_fmt_vid_out))
>> -			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix);
>> -		ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
>> -		/* just in case the driver zeroed it again */
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -		return ret;
>> +		if (ops->vidioc_s_fmt_vid_out) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix);
>> +			ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
>> +			/* just in case the driver zeroed it again */
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +			return ret;
>> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
>> +			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> -		if (unlikely(!ops->vidioc_s_fmt_vid_out_mplane))
>> -			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> -					  bytesperline);
>> -		return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
>> +		if (ops->vidioc_s_fmt_vid_out_mplane) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> +						  bytesperline);
>> +			return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
>> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
>> +			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>>  		if (unlikely(!ops->vidioc_s_fmt_vid_out_overlay))
>>  			break;
>> @@ -1744,6 +2036,82 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>>  	return -EINVAL;
>>  }
>>  
>> +static int v4l_s_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
>> +			     struct file *file, void *fh, void *arg)
>> +{
>> +	struct video_device *vfd = video_devdata(file);
>> +	struct v4l2_ext_pix_format *ef = arg;
>> +	struct v4l2_format f;
>> +	int ret;
>> +
>> +	ret = check_fmt(file, ef->type);
>> +	if (ret)
>> +		return ret;
>> +
>> +	switch (ef->type) {
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +		if (ops->vidioc_s_ext_pix_fmt_vid_cap)
>> +			return ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, ef);
>> +		break;
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +		if (ops->vidioc_s_ext_pix_fmt_vid_out)
>> +			return ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, ef);
>> +		break;
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	ret = v4l2_ext_pix_format_to_format(ef, &f,
>> +					    vfd->device_caps &
>> +					    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
>> +					     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
>> +					     V4L2_CAP_VIDEO_M2M_MPLANE),
>> +					    false);
>> +	if (ret)
>> +		return ret;
>> +
>> +	ret = v4l_s_fmt(ops, file, fh, &f);
>> +	if (ret)
>> +		return ret;
>> +
>> +	return v4l2_format_to_ext_pix_format(&f, ef, true);
>> +}
>> +
>> +static int v4l_try_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
>> +			       struct file *file, void *fh,
>> +			       struct v4l2_format *f)
>> +{
>> +	struct v4l2_ext_pix_format ef;
>> +	int ret;
>> +
>> +	ret = v4l2_format_to_ext_pix_format(f, &ef, false);
>> +	if (ret)
>> +		return ret;
>> +
>> +	switch (f->type) {
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> +		ret = ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh, &ef);
>> +		break;
>> +
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> +		ret = ops->vidioc_try_ext_pix_fmt_vid_out(file, fh, &ef);
>> +		break;
>> +
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	if (ret)
>> +		return ret;
>> +
>> +	return v4l2_ext_pix_format_to_format(&ef, f,
>> +					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
>> +					     true);
>> +}
>> +
>> +
>>  static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>>  				struct file *file, void *fh, void *arg)
>>  {
>> @@ -1759,23 +2127,32 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>>  
>>  	switch (p->type) {
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> -		if (unlikely(!ops->vidioc_try_fmt_vid_cap))
>> -			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix);
>> -		ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
>> -		/* just in case the driver zeroed it again */
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -		if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> -			v4l_pix_format_touch(&p->fmt.pix);
>> -		return ret;
>> +		if (ops->vidioc_try_fmt_vid_cap) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix);
>> +			ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
>> +			/* just in case the driver zeroed it again */
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> +				v4l_pix_format_touch(&p->fmt.pix);
>> +			return ret;
>> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
>> +			ret = v4l_try_fmt_ext_pix(ops, file, fh, p);
>> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> +				v4l_pix_format_touch(&p->fmt.pix);
>> +			return ret;
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> -		if (unlikely(!ops->vidioc_try_fmt_vid_cap_mplane))
>> -			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> -					  bytesperline);
>> -		return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
>> +		if (ops->vidioc_try_fmt_vid_cap_mplane) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> +						  bytesperline);
>> +			return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
>> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
>> +			return v4l_try_fmt_ext_pix(ops, file, fh, p);
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>>  		if (unlikely(!ops->vidioc_try_fmt_vid_overlay))
>>  			break;
>> @@ -1792,21 +2169,27 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>>  		CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
>>  		return ops->vidioc_try_fmt_sliced_vbi_cap(file, fh, arg);
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> -		if (unlikely(!ops->vidioc_try_fmt_vid_out))
>> -			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix);
>> -		ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
>> -		/* just in case the driver zeroed it again */
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -		return ret;
>> +		if (ops->vidioc_try_fmt_vid_out) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix);
>> +			ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
>> +			/* just in case the driver zeroed it again */
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +			return ret;
>> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
>> +			return v4l_try_fmt_ext_pix(ops, file, fh, p);
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> -		if (unlikely(!ops->vidioc_try_fmt_vid_out_mplane))
>> -			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> -					  bytesperline);
>> -		return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
>> +		if (ops->vidioc_try_fmt_vid_out_mplane) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> +						  bytesperline);
>> +			return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
>> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
>> +			return v4l_try_fmt_ext_pix(ops, file, fh, p);
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>>  		if (unlikely(!ops->vidioc_try_fmt_vid_out_overlay))
>>  			break;
>> @@ -1846,6 +2229,49 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>>  	return -EINVAL;
>>  }
>>  
>> +static int v4l_try_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
>> +			       struct file *file, void *fh, void *arg)
>> +{
>> +	struct video_device *vfd = video_devdata(file);
>> +	struct v4l2_ext_pix_format *ef = arg;
>> +	struct v4l2_format f;
>> +	int ret;
>> +
>> +	ret = check_fmt(file, ef->type);
>> +	if (ret)
>> +		return ret;
>> +
>> +	switch (ef->type) {
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +		if (ops->vidioc_try_ext_pix_fmt_vid_cap)
>> +			return ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh,
>> +								   ef);
>> +		break;
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +		if (ops->vidioc_try_ext_pix_fmt_vid_out)
>> +			return ops->vidioc_try_ext_pix_fmt_vid_out(file, fh,
>> +								   ef);
>> +		break;
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	ret = v4l2_ext_pix_format_to_format(ef, &f,
>> +					    vfd->device_caps &
>> +					    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
>> +					     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
>> +					     V4L2_CAP_VIDEO_M2M_MPLANE),
>> +					    false);
>> +	if (ret)
>> +		return ret;
>> +
>> +	ret = v4l_try_fmt(ops, file, fh, &f);
>> +	if (ret)
>> +		return ret;
>> +
>> +	return v4l2_format_to_ext_pix_format(&f, ef, true);
>> +}
>> +
>>  static int v4l_streamon(const struct v4l2_ioctl_ops *ops,
>>  				struct file *file, void *fh, void *arg)
>>  {
>> @@ -2765,7 +3191,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>>  	IOCTL_INFO(VIDIOC_QUERYCAP, v4l_querycap, v4l_print_querycap, 0),
>>  	IOCTL_INFO(VIDIOC_ENUM_FMT, v4l_enum_fmt, v4l_print_fmtdesc, 0),
>>  	IOCTL_INFO(VIDIOC_G_FMT, v4l_g_fmt, v4l_print_format, 0),
>> +	IOCTL_INFO(VIDIOC_G_EXT_PIX_FMT, v4l_g_ext_pix_fmt, v4l_print_ext_pix_format, 0),
>>  	IOCTL_INFO(VIDIOC_S_FMT, v4l_s_fmt, v4l_print_format, INFO_FL_PRIO),
>> +	IOCTL_INFO(VIDIOC_S_EXT_PIX_FMT, v4l_s_ext_pix_fmt, v4l_print_ext_pix_format, INFO_FL_PRIO),
>>  	IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>>  	IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
>>  	IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
>> @@ -2812,6 +3240,7 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>>  	IOCTL_INFO(VIDIOC_S_JPEGCOMP, v4l_stub_s_jpegcomp, v4l_print_jpegcompression, INFO_FL_PRIO),
>>  	IOCTL_INFO(VIDIOC_QUERYSTD, v4l_querystd, v4l_print_std, 0),
>>  	IOCTL_INFO(VIDIOC_TRY_FMT, v4l_try_fmt, v4l_print_format, 0),
>> +	IOCTL_INFO(VIDIOC_TRY_EXT_PIX_FMT, v4l_try_ext_pix_fmt, v4l_print_ext_pix_format, 0),
>>  	IOCTL_INFO(VIDIOC_ENUMAUDIO, v4l_stub_enumaudio, v4l_print_audio, INFO_FL_CLEAR(v4l2_audio, index)),
>>  	IOCTL_INFO(VIDIOC_ENUMAUDOUT, v4l_stub_enumaudout, v4l_print_audioout, INFO_FL_CLEAR(v4l2_audioout, index)),
>>  	IOCTL_INFO(VIDIOC_G_PRIORITY, v4l_g_priority, v4l_print_u32, 0),
>> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
>> index 86878fba332b0..525ce86725260 100644
>> --- a/include/media/v4l2-ioctl.h
>> +++ b/include/media/v4l2-ioctl.h
>> @@ -48,11 +48,17 @@ struct v4l2_fh;
>>   * @vidioc_g_fmt_vid_cap: pointer to the function that implements
>>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video capture
>>   *	in single plane mode
>> + * @vidioc_g_ext_pix_fmt_vid_cap: pointer to the function that implements
>> + *	:ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
>> + *	capture
>>   * @vidioc_g_fmt_vid_overlay: pointer to the function that implements
>>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>>   * @vidioc_g_fmt_vid_out: pointer to the function that implements
>>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   *	in single plane mode
>> + * @vidioc_g_ext_pix_fmt_vid_out: pointer to the function that implements
>> + *	:ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
>> + *	out
>>   * @vidioc_g_fmt_vid_out_overlay: pointer to the function that implements
>>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
>>   * @vidioc_g_fmt_vbi_cap: pointer to the function that implements
>> @@ -82,11 +88,16 @@ struct v4l2_fh;
>>   * @vidioc_s_fmt_vid_cap: pointer to the function that implements
>>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video capture
>>   *	in single plane mode
>> + * @vidioc_s_ext_pix_fmt_vid_cap: pointer to the function that implements
>> + *	:ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_s_ext_pix_fmt>` ioctl logic for video
>> + *	capture
>>   * @vidioc_s_fmt_vid_overlay: pointer to the function that implements
>>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>>   * @vidioc_s_fmt_vid_out: pointer to the function that implements
>>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   *	in single plane mode
>> + * @vidioc_s_ext_pix_fmt_vid_out: pointer to the function that implements
>> + *	:ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   * @vidioc_s_fmt_vid_out_overlay: pointer to the function that implements
>>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
>>   * @vidioc_s_fmt_vbi_cap: pointer to the function that implements
>> @@ -116,11 +127,16 @@ struct v4l2_fh;
>>   * @vidioc_try_fmt_vid_cap: pointer to the function that implements
>>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video capture
>>   *	in single plane mode
>> + * @vidioc_try_ext_pix_fmt_vid_cap: pointer to the function that implements
>> + *	:ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_try_ext_pix_fmt>` ioctl logic for
>> +	video capture
>>   * @vidioc_try_fmt_vid_overlay: pointer to the function that implements
>>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>>   * @vidioc_try_fmt_vid_out: pointer to the function that implements
>>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   *	in single plane mode
>> + * @vidioc_try_ext_pix_fmt_vid_out: pointer to the function that implements
>> + *	:ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   * @vidioc_try_fmt_vid_out_overlay: pointer to the function that implements
>>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>>   *	output
>> @@ -319,10 +335,14 @@ struct v4l2_ioctl_ops {
>>  	/* VIDIOC_G_FMT handlers */
>>  	int (*vidioc_g_fmt_vid_cap)(struct file *file, void *fh,
>>  				    struct v4l2_format *f);
>> +	int (*vidioc_g_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
>> +					    struct v4l2_ext_pix_format *f);
>>  	int (*vidioc_g_fmt_vid_overlay)(struct file *file, void *fh,
>>  					struct v4l2_format *f);
>>  	int (*vidioc_g_fmt_vid_out)(struct file *file, void *fh,
>>  				    struct v4l2_format *f);
>> +	int (*vidioc_g_ext_pix_fmt_vid_out)(struct file *file, void *fh,
>> +					    struct v4l2_ext_pix_format *f);
>>  	int (*vidioc_g_fmt_vid_out_overlay)(struct file *file, void *fh,
>>  					    struct v4l2_format *f);
>>  	int (*vidioc_g_fmt_vbi_cap)(struct file *file, void *fh,
>> @@ -349,10 +369,14 @@ struct v4l2_ioctl_ops {
>>  	/* VIDIOC_S_FMT handlers */
>>  	int (*vidioc_s_fmt_vid_cap)(struct file *file, void *fh,
>>  				    struct v4l2_format *f);
>> +	int (*vidioc_s_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
>> +					    struct v4l2_ext_pix_format *f);
>>  	int (*vidioc_s_fmt_vid_overlay)(struct file *file, void *fh,
>>  					struct v4l2_format *f);
>>  	int (*vidioc_s_fmt_vid_out)(struct file *file, void *fh,
>>  				    struct v4l2_format *f);
>> +	int (*vidioc_s_ext_pix_fmt_vid_out)(struct file *file, void *fh,
>> +					    struct v4l2_ext_pix_format *f);
>>  	int (*vidioc_s_fmt_vid_out_overlay)(struct file *file, void *fh,
>>  					    struct v4l2_format *f);
>>  	int (*vidioc_s_fmt_vbi_cap)(struct file *file, void *fh,
>> @@ -379,10 +403,14 @@ struct v4l2_ioctl_ops {
>>  	/* VIDIOC_TRY_FMT handlers */
>>  	int (*vidioc_try_fmt_vid_cap)(struct file *file, void *fh,
>>  				      struct v4l2_format *f);
>> +	int (*vidioc_try_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
>> +					      struct v4l2_ext_pix_format *f);
>>  	int (*vidioc_try_fmt_vid_overlay)(struct file *file, void *fh,
>>  					  struct v4l2_format *f);
>>  	int (*vidioc_try_fmt_vid_out)(struct file *file, void *fh,
>>  				      struct v4l2_format *f);
>> +	int (*vidioc_try_ext_pix_fmt_vid_out)(struct file *file, void *fh,
>> +					      struct v4l2_ext_pix_format *f);
>>  	int (*vidioc_try_fmt_vid_out_overlay)(struct file *file, void *fh,
>>  					     struct v4l2_format *f);
>>  	int (*vidioc_try_fmt_vbi_cap)(struct file *file, void *fh,
>> @@ -724,6 +752,12 @@ long int video_usercopy(struct file *file, unsigned int cmd,
>>  long int video_ioctl2(struct file *file,
>>  		      unsigned int cmd, unsigned long int arg);
>>  
>> +int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
>> +				  struct v4l2_ext_pix_format *e, bool strict);
>> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
>> +				  struct v4l2_format *f,
>> +				  bool mplane_cap, bool strict);
>> +
>>  /*
>>   * The user space interpretation of the 'v4l2_event' differs
>>   * based on the 'time_t' definition on 32-bit architectures, so
>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>> index 303805438814f..fc04c81ce7713 100644
>> --- a/include/uapi/linux/videodev2.h
>> +++ b/include/uapi/linux/videodev2.h
>> @@ -2252,6 +2252,57 @@ struct v4l2_pix_format_mplane {
>>  	__u8				reserved[7];
>>  } __attribute__ ((packed));
>>  
>> +/**
>> + * struct v4l2_plane_ext_pix_format - additional, per-plane format definition
>> + * @sizeimage:		maximum size in bytes required for data, for which
>> + *			this plane will be used
>> + * @bytesperline:	distance in bytes between the leftmost pixels in two
>> + *			adjacent lines
>> + * @reserved:		extra space reserved for future fields, must be set to 0
>> + */
>> +struct v4l2_plane_ext_pix_format {
>> +	__u32 sizeimage;
>> +	__u32 bytesperline;
>> +	__u32 reserved;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct v4l2_ext_pix_format - extended single/multiplanar format definition
>> + * @type:		type of the data stream; V4L2_BUF_TYPE_VIDEO_CAPTURE or
>> + *			V4L2_BUF_TYPE_VIDEO_OUTPUT
>> + * @width:		image width in pixels
>> + * @height:		image height in pixels
>> + * @field:		enum v4l2_field; field order (for interlaced video)
>> + * @pixelformat:	little endian four character code (fourcc)
>> + * @modifier:		modifier applied to the format (used for tiled formats
>> + *			and other kind of HW-specific formats, like compressed
>> + *			formats)
>> + * @colorspace:		enum v4l2_colorspace; supplemental to pixelformat
>> + * @plane_fmt:		per-plane information
>> + * @ycbcr_enc:		enum v4l2_ycbcr_encoding, Y'CbCr encoding
>> + * @hsv_enc:		enum v4l2_hsv_encoding, HSV encoding
>> + * @quantization:	enum v4l2_quantization, colorspace quantization
>> + * @xfer_func:		enum v4l2_xfer_func, colorspace transfer function
>> + * @reserved:		extra space reserved for future fields, must be set to 0
>> + */
>> +struct v4l2_ext_pix_format {
>> +	__u32 type;
>> +	__u32 width;
>> +	__u32 height;
>> +	__u32 field;
>> +	__u32 pixelformat;
>> +	__u64 modifier;
>> +	__u32 colorspace;
> 
> This struct has holes and is not the same for 32 and 64 bit architectures.

This would be true if this struct wasn't packed, but I believe we can remove the
packed attribute, unless I'm missing something.
What was the reason for other format structs to have __attribute__ ((packed)) ?

> 
> Moving modifier to before pixelformat will help a lot.
> 
>> +	struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];
>> +	union {
>> +		__u8 ycbcr_enc;
>> +		__u8 hsv_enc;
>> +	};
>> +	__u8 quantization;
>> +	__u8 xfer_func;
> 
> I'd change u8 to u32 for these fields for easier alignment.

Wouldn't it be better to add more reserved fields instead? So we can use this space
latter in case we need them?

Without __attribute__ ((packed)), moving modifiers and changing reserved, I have
from pahole in both 32 and 64 architectures:

struct v4l2_ext_pix_format {
	__u32                      type;                 /*     0     4 */
	__u32                      width;                /*     4     4 */
	__u32                      height;               /*     8     4 */
	__u32                      field;                /*    12     4 */
	__u64                      modifier;             /*    16     8 */
	__u32                      pixelformat;          /*    24     4 */
	__u32                      colorspace;           /*    28     4 */
	struct v4l2_plane_ext_pix_format plane_fmt[8];   /*    32    96 */
	/* --- cacheline 2 boundary (128 bytes) --- */
	union {
		__u8               ycbcr_enc;            /*   128     1 */
		__u8               hsv_enc;              /*   128     1 */
	};                                               /*   128     1 */
	__u8                       quantization;         /*   129     1 */
	__u8                       xfer_func;            /*   130     1 */
	__u8                       _reserved;            /*   131     1 */
	__u32                      reserved[5];          /*   132    20 */

	/* size: 152, cachelines: 3, members: 13 */
	/* last cacheline: 24 bytes */
};


What do you think?

Regards,
Helen


> 
> Regards,
> 
> 	Hans
> 
>> +	__u32 reserved[4];
>> +} __attribute__ ((packed));
>> +
>>  /**
>>   * struct v4l2_sdr_format - SDR format definition
>>   * @pixelformat:	little endian four character code (fourcc)
>> @@ -2569,6 +2620,10 @@ struct v4l2_create_buffers {
>>  
>>  #define VIDIOC_QUERY_EXT_CTRL	_IOWR('V', 103, struct v4l2_query_ext_ctrl)
>>  
>> +#define VIDIOC_G_EXT_PIX_FMT	_IOWR('V', 104, struct v4l2_ext_pix_format)
>> +#define VIDIOC_S_EXT_PIX_FMT	_IOWR('V', 105, struct v4l2_ext_pix_format)
>> +#define VIDIOC_TRY_EXT_PIX_FMT	_IOWR('V', 106, struct v4l2_ext_pix_format)
>> +
>>  /* Reminder: when adding new ioctls please add support for them to
>>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
>>  
>>
> 
> 

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v4 1/6] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)
  2020-07-28 15:18     ` Helen Koike
@ 2020-07-28 15:30       ` Hans Verkuil
  0 siblings, 0 replies; 29+ messages in thread
From: Hans Verkuil @ 2020-07-28 15:30 UTC (permalink / raw)
  To: Helen Koike, mchehab, hans.verkuil, laurent.pinchart,
	sakari.ailus, linux-media
  Cc: Boris Brezillon, tfiga, hiroh, nicolas, Brian.Starkey, kernel,
	narmstrong, linux-kernel, frkoenig, mjourdan, stanimir.varbanov

On 28/07/2020 17:18, Helen Koike wrote:
> Hi Hans,
> 
> On 7/21/20 7:37 AM, Hans Verkuil wrote:
>> On 17/07/2020 13:54, Helen Koike wrote:
>>>  
>>> +/**
>>> + * struct v4l2_plane_ext_pix_format - additional, per-plane format definition
>>> + * @sizeimage:		maximum size in bytes required for data, for which
>>> + *			this plane will be used
>>> + * @bytesperline:	distance in bytes between the leftmost pixels in two
>>> + *			adjacent lines
>>> + * @reserved:		extra space reserved for future fields, must be set to 0
>>> + */
>>> +struct v4l2_plane_ext_pix_format {
>>> +	__u32 sizeimage;
>>> +	__u32 bytesperline;
>>> +	__u32 reserved;
>>> +} __attribute__ ((packed));
>>> +
>>> +/**
>>> + * struct v4l2_ext_pix_format - extended single/multiplanar format definition
>>> + * @type:		type of the data stream; V4L2_BUF_TYPE_VIDEO_CAPTURE or
>>> + *			V4L2_BUF_TYPE_VIDEO_OUTPUT
>>> + * @width:		image width in pixels
>>> + * @height:		image height in pixels
>>> + * @field:		enum v4l2_field; field order (for interlaced video)
>>> + * @pixelformat:	little endian four character code (fourcc)
>>> + * @modifier:		modifier applied to the format (used for tiled formats
>>> + *			and other kind of HW-specific formats, like compressed
>>> + *			formats)
>>> + * @colorspace:		enum v4l2_colorspace; supplemental to pixelformat
>>> + * @plane_fmt:		per-plane information
>>> + * @ycbcr_enc:		enum v4l2_ycbcr_encoding, Y'CbCr encoding
>>> + * @hsv_enc:		enum v4l2_hsv_encoding, HSV encoding
>>> + * @quantization:	enum v4l2_quantization, colorspace quantization
>>> + * @xfer_func:		enum v4l2_xfer_func, colorspace transfer function
>>> + * @reserved:		extra space reserved for future fields, must be set to 0
>>> + */
>>> +struct v4l2_ext_pix_format {
>>> +	__u32 type;
>>> +	__u32 width;
>>> +	__u32 height;
>>> +	__u32 field;
>>> +	__u32 pixelformat;
>>> +	__u64 modifier;
>>> +	__u32 colorspace;
>>
>> This struct has holes and is not the same for 32 and 64 bit architectures.
> 
> This would be true if this struct wasn't packed, but I believe we can remove the
> packed attribute, unless I'm missing something.
> What was the reason for other format structs to have __attribute__ ((packed)) ?

I've never really analyzed it. For new structs you want to avoid messing with that.

> 
>>
>> Moving modifier to before pixelformat will help a lot.
>>
>>> +	struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];
>>> +	union {
>>> +		__u8 ycbcr_enc;
>>> +		__u8 hsv_enc;
>>> +	};
>>> +	__u8 quantization;
>>> +	__u8 xfer_func;
>>
>> I'd change u8 to u32 for these fields for easier alignment.
> 
> Wouldn't it be better to add more reserved fields instead? So we can use this space
> latter in case we need them?
> 
> Without __attribute__ ((packed)), moving modifiers and changing reserved, I have
> from pahole in both 32 and 64 architectures:
> 
> struct v4l2_ext_pix_format {
> 	__u32                      type;                 /*     0     4 */
> 	__u32                      width;                /*     4     4 */
> 	__u32                      height;               /*     8     4 */
> 	__u32                      field;                /*    12     4 */
> 	__u64                      modifier;             /*    16     8 */
> 	__u32                      pixelformat;          /*    24     4 */
> 	__u32                      colorspace;           /*    28     4 */
> 	struct v4l2_plane_ext_pix_format plane_fmt[8];   /*    32    96 */
> 	/* --- cacheline 2 boundary (128 bytes) --- */
> 	union {
> 		__u8               ycbcr_enc;            /*   128     1 */
> 		__u8               hsv_enc;              /*   128     1 */
> 	};                                               /*   128     1 */
> 	__u8                       quantization;         /*   129     1 */
> 	__u8                       xfer_func;            /*   130     1 */
> 	__u8                       _reserved;            /*   131     1 */

This additional _reserved field is just what you want to avoid.

Just stick to u32 as much as possible with a u32 reserved array at the end.

> 	__u32                      reserved[5];          /*   132    20 */

[5] is definitely not enough. But that's something we can ignore until this
struct is finalized.

Regards,

	Hans

> 
> 	/* size: 152, cachelines: 3, members: 13 */
> 	/* last cacheline: 24 bytes */
> };
> 
> 
> What do you think?
> 
> Regards,
> Helen
> 
> 
>>
>> Regards,
>>
>> 	Hans
>>
>>> +	__u32 reserved[4];
>>> +} __attribute__ ((packed));
>>> +
>>>  /**
>>>   * struct v4l2_sdr_format - SDR format definition
>>>   * @pixelformat:	little endian four character code (fourcc)
>>> @@ -2569,6 +2620,10 @@ struct v4l2_create_buffers {
>>>  
>>>  #define VIDIOC_QUERY_EXT_CTRL	_IOWR('V', 103, struct v4l2_query_ext_ctrl)
>>>  
>>> +#define VIDIOC_G_EXT_PIX_FMT	_IOWR('V', 104, struct v4l2_ext_pix_format)
>>> +#define VIDIOC_S_EXT_PIX_FMT	_IOWR('V', 105, struct v4l2_ext_pix_format)
>>> +#define VIDIOC_TRY_EXT_PIX_FMT	_IOWR('V', 106, struct v4l2_ext_pix_format)
>>> +
>>>  /* Reminder: when adding new ioctls please add support for them to
>>>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
>>>  
>>>
>>
>>


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v4 2/6] media: v4l2: Add extended buffer operations
  2020-07-27 12:01             ` Helen Koike
@ 2020-07-28 18:02               ` Stanimir Varbanov
  0 siblings, 0 replies; 29+ messages in thread
From: Stanimir Varbanov @ 2020-07-28 18:02 UTC (permalink / raw)
  To: Helen Koike, mchehab, hans.verkuil, laurent.pinchart,
	sakari.ailus, linux-media
  Cc: Boris Brezillon, tfiga, hiroh, nicolas, Brian.Starkey, kernel,
	narmstrong, linux-kernel, frkoenig, mjourdan

Hi Helen,

On 7/27/20 3:01 PM, Helen Koike wrote:
> 
> 
> On 7/24/20 10:16 AM, Stanimir Varbanov wrote:
>>
>>
>> On 7/21/20 5:40 PM, Helen Koike wrote:
>>>
>>>
>>> On 7/21/20 11:30 AM, Stanimir Varbanov wrote:
>>>> Hi Helen,
>>>>
>>>> On 7/21/20 4:54 PM, Helen Koike wrote:
>>>>> Hi,
>>>>>
>>>>> On 7/21/20 8:26 AM, Stanimir Varbanov wrote:
>>>>>>
>>>>>>
>>>>>> On 7/17/20 2:54 PM, Helen Koike wrote:
>>>>>>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>>>>>>
>>>>>>> Those extended buffer ops have several purpose:
>>>>>>> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
>>>>>>>    the number of ns elapsed since 1970
>>>>>>> 2/ Unify single/multiplanar handling
>>>>>>> 3/ Add a new start offset field to each v4l2 plane buffer info struct
>>>>>>>    to support the case where a single buffer object is storing all
>>>>>>>    planes data, each one being placed at a different offset
>>>>>>>
>>>>>>> New hooks are created in v4l2_ioctl_ops so that drivers can start using
>>>>>>> these new objects.
>>>>>>>
>>>>>>> The core takes care of converting new ioctls requests to old ones
>>>>>>> if the driver does not support the new hooks, and vice versa.
>>>>>>>
>>>>>>> Note that the timecode field is gone, since there doesn't seem to be
>>>>>>> in-kernel users. We can be added back in the reserved area if needed or
>>>>>>> use the Request API to collect more metadata information from the
>>>>>>> frame.
>>>>>>>
>>>>>>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>>>>>>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
>>>>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>>>>>> ---
>>>>>>> Changes in v4:
>>>>>>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
>>>>>>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
>>>>>>> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
>>>>>>> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
>>>>>>> I think we can add this later, so I removed it from this RFC to simplify it.
>>>>>>> - Remove num_planes field from struct v4l2_ext_buffer
>>>>>>> - Add flags field to struct v4l2_ext_create_buffers
>>>>>>> - Reformulate struct v4l2_ext_plane
>>>>>>> - Fix some bugs caught by v4l2-compliance
>>>>>>> - Rebased on top of media/master (post 5.8-rc1)
>>>>>>>
>>>>>>> Changes in v3:
>>>>>>> - Rebased on top of media/master (post 5.4-rc1)
>>>>>>>
>>>>>>> Changes in v2:
>>>>>>> - Add reserved space to v4l2_ext_buffer so that new fields can be added
>>>>>>>   later on
>>>>>>> ---
>>>>>>>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
>>>>>>>  drivers/media/v4l2-core/v4l2-ioctl.c | 349 +++++++++++++++++++++++++--
>>>>>>>  include/media/v4l2-ioctl.h           |  26 ++
>>>>>>>  include/uapi/linux/videodev2.h       |  89 +++++++
>>>>>>>  4 files changed, 471 insertions(+), 22 deletions(-)
>>>>>>>
>>>>>>
>>>>>> <cut>
>>>>>>
>>>>>>> +/**
>>>>>>> + * struct v4l2_ext_plane - extended plane buffer info
>>>>>>> + * @buffer_length:	size of the entire buffer in bytes, should fit
>>>>>>> + *			@offset + @plane_length
>>>>>>> + * @plane_length:	size of the plane in bytes.
>>>>>>> + * @userptr:		when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
>>>>>>> + *			to this plane.
>>>>>>> + * @dmabuf_fd:		when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
>>>>>>> + *			associated with this plane.
>>>>>>> + * @offset:		offset in the memory buffer where the plane starts. If
>>>>>>> + *			V4L2_MEMORY_MMAP is used, then it can be a "cookie" that
>>>>>>> + *			should be passed to mmap() called on the video node.
>>>>>>> + * @reserved:		extra space reserved for future fields, must be set to 0.
>>>>>>> + *
>>>>>>> + *
>>>>>>> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
>>>>>>> + * can have one plane for Y, and another for interleaved CbCr components.
>>>>>>> + * Each plane can reside in a separate memory buffer, or even in
>>>>>>> + * a completely separate memory node (e.g. in embedded devices).
>>>>>>> + */
>>>>>>> +struct v4l2_ext_plane {
>>>>>>> +	__u32 buffer_length;
>>>>>>> +	__u32 plane_length;
>>>>>>> +	union {
>>>>>>> +		__u64 userptr;
>>>>>>> +		__s32 dmabuf_fd;
>>>>>>> +	} m;
>>>>>>> +	__u32 offset;
>>>>>>> +	__u32 reserved[4];
>>>>>>> +};
>>>>>>> +
>>>>>>>  /**
>>>>>>>   * struct v4l2_buffer - video buffer info
>>>>>>>   * @index:	id number of the buffer
>>>>>>> @@ -1055,6 +1086,36 @@ struct v4l2_buffer {
>>>>>>>  	};
>>>>>>>  };
>>>>>>>  
>>>>>>> +/**
>>>>>>> + * struct v4l2_ext_buffer - extended video buffer info
>>>>>>> + * @index:	id number of the buffer
>>>>>>> + * @type:	V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
>>>>>>> + * @flags:	buffer informational flags
>>>>>>> + * @field:	enum v4l2_field; field order of the image in the buffer
>>>>>>> + * @timestamp:	frame timestamp
>>>>>>> + * @sequence:	sequence count of this frame
>>>>>>> + * @memory:	enum v4l2_memory; the method, in which the actual video data is
>>>>>>> + *		passed
>>>>>>> + * @planes:	per-plane buffer information
>>>>>>> + * @request_fd:	fd of the request that this buffer should use
>>>>>>> + * @reserved:	extra space reserved for future fields, must be set to 0
>>>>>>> + *
>>>>>>> + * Contains data exchanged by application and driver using one of the Streaming
>>>>>>> + * I/O methods.
>>>>>>> + */
>>>>>>> +struct v4l2_ext_buffer {
>>>>>>> +	__u32 index;
>>>>>>> +	__u32 type;
>>>>>>> +	__u32 flags;
>>>>>>> +	__u32 field;
>>>>>>> +	__u64 timestamp;
>>>>>>> +	__u32 sequence;
>>>>>>> +	__u32 memory;
>>>>>>> +	__u32 request_fd;
>>>>>>
>>>>>> This should be __s32, at least for consistency with dmabuf_fd?
>>>>>
>>>>> I see that in struct v4l2_buffer, we have __s32, I don't mind changing it
>>>>> to keep the consistency, I just don't see where this value can be a negative
>>>>> number.
>>>>
>>>> here
>>>> https://elixir.bootlin.com/linux/v5.8-rc4/source/drivers/media/common/videobuf2/videobuf2-v4l2.c#L134
>>>
>>> I saw that -1 is used to signal an invalid value, but I was just wondering when request_fd = 0 is valid.
>>
>> The request_fd is valid system wide file descriptor and request_fd = 0
>> is STDIN_FILENO thus IMO it is valid as far as we call it file descriptor.
> 
> Ack
> 
>>
>>>
>>>>
>>>>>
>>>>>>
>>>>>>> +	struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
>>>>>>> +	__u32 reserved[4];
>>>>>>
>>>>>> I think we have to reserve more words here for future extensions.
>>>>>>
>>>>>> I'd like also to propose to add here __s32 metadata_fd. The idea behind
>>>>>> this is to have a way to pass per-frame metadata dmabuf buffers for
>>>>>> synchronous type of metadata where the metadata is coming at the same
>>>>>> time with data buffers. What would be the format of the metadata buffer
>>>>>> is TBD.
>>>>>>
>>>>>> One option for metadata buffer format could be:
>>>>>>
>>>>>> header {
>>>>>> 	num_ctrls
>>>>>> 	array_of_ctrls [0..N]
>>>>>> 		ctrl_id
>>>>>> 		ctrl_size
>>>>>> 		ctrl_offset
>>>>>> }
>>>>>>
>>>>>> data {
>>>>>> 	cid0	//offset of cid0 in dmabuf buffer
>>>>>> 	cid1
>>>>>> 	cidN
>>>>>> }
>>>>>
>>>>> Would it be better if, instead of adding a medatata_fd inside struct v4l2_ext_buffer,
>>>>> we create a new ioctl that gets this structs for the controls and sync them using the
>>>>> Request API ?
>>
>> New ioctl means new syscall. There are use-cases where encoding
>> framerate is 480 fps (and more in near future, for example 960fps) this
>> means 480 more syscalls per second. I don't think this is optimal and
>> scalable solution at all.
> 
> I feel we have a more general problem then.
> 
> What I propose is to leave reserved fields for now, and we can discuss how to include
> this new feature in the future with a different RFC when we have a better view of requirements,
> what do you think?

Sounds good, thanks.

> 
> Thanks
> Helen
-- 
regards,
Stan

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v4 2/6] media: v4l2: Add extended buffer operations
  2020-07-27 12:35             ` Tomasz Figa
@ 2020-07-28 18:08               ` Stanimir Varbanov
  2020-08-05 14:05                 ` Tomasz Figa
  0 siblings, 1 reply; 29+ messages in thread
From: Stanimir Varbanov @ 2020-07-28 18:08 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: Helen Koike, Mauro Carvalho Chehab, Hans Verkuil,
	Laurent Pinchart, Sakari Ailus, Linux Media Mailing List,
	Boris Brezillon, Hirokazu Honda, Nicolas Dufresne, Brian Starkey,
	kernel, Neil Armstrong, Linux Kernel Mailing List, frkoenig,
	Maxime Jourdan

Hi Tomasz,

On 7/27/20 3:35 PM, Tomasz Figa wrote:
> Hi Stanimir,
> 
> On Fri, Jul 24, 2020 at 3:17 PM Stanimir Varbanov
> <stanimir.varbanov@linaro.org> wrote:
>>
>>
>>
>> On 7/21/20 5:40 PM, Helen Koike wrote:
>>>
>>>
>>> On 7/21/20 11:30 AM, Stanimir Varbanov wrote:
>>>> Hi Helen,
>>>>
>>>> On 7/21/20 4:54 PM, Helen Koike wrote:
>>>>> Hi,
>>>>>
>>>>> On 7/21/20 8:26 AM, Stanimir Varbanov wrote:
>>>>>>
>>>>>>
>>>>>> On 7/17/20 2:54 PM, Helen Koike wrote:
>>>>>>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>>>>>>
>>>>>>> Those extended buffer ops have several purpose:
>>>>>>> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
>>>>>>>    the number of ns elapsed since 1970
>>>>>>> 2/ Unify single/multiplanar handling
>>>>>>> 3/ Add a new start offset field to each v4l2 plane buffer info struct
>>>>>>>    to support the case where a single buffer object is storing all
>>>>>>>    planes data, each one being placed at a different offset
>>>>>>>
>>>>>>> New hooks are created in v4l2_ioctl_ops so that drivers can start using
>>>>>>> these new objects.
>>>>>>>
>>>>>>> The core takes care of converting new ioctls requests to old ones
>>>>>>> if the driver does not support the new hooks, and vice versa.
>>>>>>>
>>>>>>> Note that the timecode field is gone, since there doesn't seem to be
>>>>>>> in-kernel users. We can be added back in the reserved area if needed or
>>>>>>> use the Request API to collect more metadata information from the
>>>>>>> frame.
>>>>>>>
>>>>>>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>>>>>>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
>>>>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>>>>>> ---
>>>>>>> Changes in v4:
>>>>>>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
>>>>>>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
>>>>>>> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
>>>>>>> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
>>>>>>> I think we can add this later, so I removed it from this RFC to simplify it.
>>>>>>> - Remove num_planes field from struct v4l2_ext_buffer
>>>>>>> - Add flags field to struct v4l2_ext_create_buffers
>>>>>>> - Reformulate struct v4l2_ext_plane
>>>>>>> - Fix some bugs caught by v4l2-compliance
>>>>>>> - Rebased on top of media/master (post 5.8-rc1)
>>>>>>>
>>>>>>> Changes in v3:
>>>>>>> - Rebased on top of media/master (post 5.4-rc1)
>>>>>>>
>>>>>>> Changes in v2:
>>>>>>> - Add reserved space to v4l2_ext_buffer so that new fields can be added
>>>>>>>   later on
>>>>>>> ---
>>>>>>>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
>>>>>>>  drivers/media/v4l2-core/v4l2-ioctl.c | 349 +++++++++++++++++++++++++--
>>>>>>>  include/media/v4l2-ioctl.h           |  26 ++
>>>>>>>  include/uapi/linux/videodev2.h       |  89 +++++++
>>>>>>>  4 files changed, 471 insertions(+), 22 deletions(-)
>>>>>>>
>>>>>>
>>>>>> <cut>
>>>>>>
>>>>>>> +/**
>>>>>>> + * struct v4l2_ext_plane - extended plane buffer info
>>>>>>> + * @buffer_length:       size of the entire buffer in bytes, should fit
>>>>>>> + *                       @offset + @plane_length
>>>>>>> + * @plane_length:        size of the plane in bytes.
>>>>>>> + * @userptr:             when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
>>>>>>> + *                       to this plane.
>>>>>>> + * @dmabuf_fd:           when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
>>>>>>> + *                       associated with this plane.
>>>>>>> + * @offset:              offset in the memory buffer where the plane starts. If
>>>>>>> + *                       V4L2_MEMORY_MMAP is used, then it can be a "cookie" that
>>>>>>> + *                       should be passed to mmap() called on the video node.
>>>>>>> + * @reserved:            extra space reserved for future fields, must be set to 0.
>>>>>>> + *
>>>>>>> + *
>>>>>>> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
>>>>>>> + * can have one plane for Y, and another for interleaved CbCr components.
>>>>>>> + * Each plane can reside in a separate memory buffer, or even in
>>>>>>> + * a completely separate memory node (e.g. in embedded devices).
>>>>>>> + */
>>>>>>> +struct v4l2_ext_plane {
>>>>>>> + __u32 buffer_length;
>>>>>>> + __u32 plane_length;
>>>>>>> + union {
>>>>>>> +         __u64 userptr;
>>>>>>> +         __s32 dmabuf_fd;
>>>>>>> + } m;
>>>>>>> + __u32 offset;
>>>>>>> + __u32 reserved[4];
>>>>>>> +};
>>>>>>> +
>>>>>>>  /**
>>>>>>>   * struct v4l2_buffer - video buffer info
>>>>>>>   * @index:       id number of the buffer
>>>>>>> @@ -1055,6 +1086,36 @@ struct v4l2_buffer {
>>>>>>>   };
>>>>>>>  };
>>>>>>>
>>>>>>> +/**
>>>>>>> + * struct v4l2_ext_buffer - extended video buffer info
>>>>>>> + * @index:       id number of the buffer
>>>>>>> + * @type:        V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
>>>>>>> + * @flags:       buffer informational flags
>>>>>>> + * @field:       enum v4l2_field; field order of the image in the buffer
>>>>>>> + * @timestamp:   frame timestamp
>>>>>>> + * @sequence:    sequence count of this frame
>>>>>>> + * @memory:      enum v4l2_memory; the method, in which the actual video data is
>>>>>>> + *               passed
>>>>>>> + * @planes:      per-plane buffer information
>>>>>>> + * @request_fd:  fd of the request that this buffer should use
>>>>>>> + * @reserved:    extra space reserved for future fields, must be set to 0
>>>>>>> + *
>>>>>>> + * Contains data exchanged by application and driver using one of the Streaming
>>>>>>> + * I/O methods.
>>>>>>> + */
>>>>>>> +struct v4l2_ext_buffer {
>>>>>>> + __u32 index;
>>>>>>> + __u32 type;
>>>>>>> + __u32 flags;
>>>>>>> + __u32 field;
>>>>>>> + __u64 timestamp;
>>>>>>> + __u32 sequence;
>>>>>>> + __u32 memory;
>>>>>>> + __u32 request_fd;
>>>>>>
>>>>>> This should be __s32, at least for consistency with dmabuf_fd?
>>>>>
>>>>> I see that in struct v4l2_buffer, we have __s32, I don't mind changing it
>>>>> to keep the consistency, I just don't see where this value can be a negative
>>>>> number.
>>>>
>>>> here
>>>> https://elixir.bootlin.com/linux/v5.8-rc4/source/drivers/media/common/videobuf2/videobuf2-v4l2.c#L134
>>>
>>> I saw that -1 is used to signal an invalid value, but I was just wondering when request_fd = 0 is valid.
>>
>> The request_fd is valid system wide file descriptor and request_fd = 0
>> is STDIN_FILENO thus IMO it is valid as far as we call it file descriptor.
>>
>>>
>>>>
>>>>>
>>>>>>
>>>>>>> + struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
>>>>>>> + __u32 reserved[4];
>>>>>>
>>>>>> I think we have to reserve more words here for future extensions.
>>>>>>
>>>>>> I'd like also to propose to add here __s32 metadata_fd. The idea behind
>>>>>> this is to have a way to pass per-frame metadata dmabuf buffers for
>>>>>> synchronous type of metadata where the metadata is coming at the same
>>>>>> time with data buffers. What would be the format of the metadata buffer
>>>>>> is TBD.
>>>>>>
>>>>>> One option for metadata buffer format could be:
>>>>>>
>>>>>> header {
>>>>>>    num_ctrls
>>>>>>    array_of_ctrls [0..N]
>>>>>>            ctrl_id
>>>>>>            ctrl_size
>>>>>>            ctrl_offset
>>>>>> }
>>>>>>
>>>>>> data {
>>>>>>    cid0    //offset of cid0 in dmabuf buffer
>>>>>>    cid1
>>>>>>    cidN
>>>>>> }
>>>>>
>>>>> Would it be better if, instead of adding a medatata_fd inside struct v4l2_ext_buffer,
>>>>> we create a new ioctl that gets this structs for the controls and sync them using the
>>>>> Request API ?
>>
>> New ioctl means new syscall. There are use-cases where encoding
>> framerate is 480 fps (and more in near future, for example 960fps) this
>> means 480 more syscalls per second. I don't think this is optimal and
>> scalable solution at all.
>>
> 
> Do you happen to have some data to confirm that it's indeed a problem?

If you mean profiling data, unfortunately I don't have such. But isn't
it obvious that increasing metadata size (think of compound v4l2
controls) and new syscalls isn't scalable solution in regards to higher
framerates?

I'm open to consider any suggestion different from v4l2 controls plus
Request API.

> 
> Best regards,
> Tomasz
-- 
regards,
Stan

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v4 2/6] media: v4l2: Add extended buffer operations
  2020-07-28 18:08               ` Stanimir Varbanov
@ 2020-08-05 14:05                 ` Tomasz Figa
  0 siblings, 0 replies; 29+ messages in thread
From: Tomasz Figa @ 2020-08-05 14:05 UTC (permalink / raw)
  To: Stanimir Varbanov
  Cc: Helen Koike, Mauro Carvalho Chehab, Hans Verkuil,
	Laurent Pinchart, Sakari Ailus, Linux Media Mailing List,
	Boris Brezillon, Hirokazu Honda, Nicolas Dufresne, Brian Starkey,
	kernel, Neil Armstrong, Linux Kernel Mailing List, frkoenig,
	Maxime Jourdan

On Tue, Jul 28, 2020 at 09:08:55PM +0300, Stanimir Varbanov wrote:
> Hi Tomasz,
> 
> On 7/27/20 3:35 PM, Tomasz Figa wrote:
> > Hi Stanimir,
> > 
> > On Fri, Jul 24, 2020 at 3:17 PM Stanimir Varbanov
> > <stanimir.varbanov@linaro.org> wrote:
> >>
> >>
> >>
> >> On 7/21/20 5:40 PM, Helen Koike wrote:
> >>>
> >>>
> >>> On 7/21/20 11:30 AM, Stanimir Varbanov wrote:
> >>>> Hi Helen,
> >>>>
> >>>> On 7/21/20 4:54 PM, Helen Koike wrote:
> >>>>> Hi,
> >>>>>
> >>>>> On 7/21/20 8:26 AM, Stanimir Varbanov wrote:
> >>>>>>
> >>>>>>
> >>>>>> On 7/17/20 2:54 PM, Helen Koike wrote:
> >>>>>>> From: Hans Verkuil <hans.verkuil@cisco.com>
> >>>>>>>
> >>>>>>> Those extended buffer ops have several purpose:
> >>>>>>> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
> >>>>>>>    the number of ns elapsed since 1970
> >>>>>>> 2/ Unify single/multiplanar handling
> >>>>>>> 3/ Add a new start offset field to each v4l2 plane buffer info struct
> >>>>>>>    to support the case where a single buffer object is storing all
> >>>>>>>    planes data, each one being placed at a different offset
> >>>>>>>
> >>>>>>> New hooks are created in v4l2_ioctl_ops so that drivers can start using
> >>>>>>> these new objects.
> >>>>>>>
> >>>>>>> The core takes care of converting new ioctls requests to old ones
> >>>>>>> if the driver does not support the new hooks, and vice versa.
> >>>>>>>
> >>>>>>> Note that the timecode field is gone, since there doesn't seem to be
> >>>>>>> in-kernel users. We can be added back in the reserved area if needed or
> >>>>>>> use the Request API to collect more metadata information from the
> >>>>>>> frame.
> >>>>>>>
> >>>>>>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> >>>>>>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
> >>>>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> >>>>>>> ---
> >>>>>>> Changes in v4:
> >>>>>>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
> >>>>>>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
> >>>>>>> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
> >>>>>>> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
> >>>>>>> I think we can add this later, so I removed it from this RFC to simplify it.
> >>>>>>> - Remove num_planes field from struct v4l2_ext_buffer
> >>>>>>> - Add flags field to struct v4l2_ext_create_buffers
> >>>>>>> - Reformulate struct v4l2_ext_plane
> >>>>>>> - Fix some bugs caught by v4l2-compliance
> >>>>>>> - Rebased on top of media/master (post 5.8-rc1)
> >>>>>>>
> >>>>>>> Changes in v3:
> >>>>>>> - Rebased on top of media/master (post 5.4-rc1)
> >>>>>>>
> >>>>>>> Changes in v2:
> >>>>>>> - Add reserved space to v4l2_ext_buffer so that new fields can be added
> >>>>>>>   later on
> >>>>>>> ---
> >>>>>>>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
> >>>>>>>  drivers/media/v4l2-core/v4l2-ioctl.c | 349 +++++++++++++++++++++++++--
> >>>>>>>  include/media/v4l2-ioctl.h           |  26 ++
> >>>>>>>  include/uapi/linux/videodev2.h       |  89 +++++++
> >>>>>>>  4 files changed, 471 insertions(+), 22 deletions(-)
> >>>>>>>
> >>>>>>
> >>>>>> <cut>
> >>>>>>
> >>>>>>> +/**
> >>>>>>> + * struct v4l2_ext_plane - extended plane buffer info
> >>>>>>> + * @buffer_length:       size of the entire buffer in bytes, should fit
> >>>>>>> + *                       @offset + @plane_length
> >>>>>>> + * @plane_length:        size of the plane in bytes.
> >>>>>>> + * @userptr:             when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
> >>>>>>> + *                       to this plane.
> >>>>>>> + * @dmabuf_fd:           when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
> >>>>>>> + *                       associated with this plane.
> >>>>>>> + * @offset:              offset in the memory buffer where the plane starts. If
> >>>>>>> + *                       V4L2_MEMORY_MMAP is used, then it can be a "cookie" that
> >>>>>>> + *                       should be passed to mmap() called on the video node.
> >>>>>>> + * @reserved:            extra space reserved for future fields, must be set to 0.
> >>>>>>> + *
> >>>>>>> + *
> >>>>>>> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
> >>>>>>> + * can have one plane for Y, and another for interleaved CbCr components.
> >>>>>>> + * Each plane can reside in a separate memory buffer, or even in
> >>>>>>> + * a completely separate memory node (e.g. in embedded devices).
> >>>>>>> + */
> >>>>>>> +struct v4l2_ext_plane {
> >>>>>>> + __u32 buffer_length;
> >>>>>>> + __u32 plane_length;
> >>>>>>> + union {
> >>>>>>> +         __u64 userptr;
> >>>>>>> +         __s32 dmabuf_fd;
> >>>>>>> + } m;
> >>>>>>> + __u32 offset;
> >>>>>>> + __u32 reserved[4];
> >>>>>>> +};
> >>>>>>> +
> >>>>>>>  /**
> >>>>>>>   * struct v4l2_buffer - video buffer info
> >>>>>>>   * @index:       id number of the buffer
> >>>>>>> @@ -1055,6 +1086,36 @@ struct v4l2_buffer {
> >>>>>>>   };
> >>>>>>>  };
> >>>>>>>
> >>>>>>> +/**
> >>>>>>> + * struct v4l2_ext_buffer - extended video buffer info
> >>>>>>> + * @index:       id number of the buffer
> >>>>>>> + * @type:        V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
> >>>>>>> + * @flags:       buffer informational flags
> >>>>>>> + * @field:       enum v4l2_field; field order of the image in the buffer
> >>>>>>> + * @timestamp:   frame timestamp
> >>>>>>> + * @sequence:    sequence count of this frame
> >>>>>>> + * @memory:      enum v4l2_memory; the method, in which the actual video data is
> >>>>>>> + *               passed
> >>>>>>> + * @planes:      per-plane buffer information
> >>>>>>> + * @request_fd:  fd of the request that this buffer should use
> >>>>>>> + * @reserved:    extra space reserved for future fields, must be set to 0
> >>>>>>> + *
> >>>>>>> + * Contains data exchanged by application and driver using one of the Streaming
> >>>>>>> + * I/O methods.
> >>>>>>> + */
> >>>>>>> +struct v4l2_ext_buffer {
> >>>>>>> + __u32 index;
> >>>>>>> + __u32 type;
> >>>>>>> + __u32 flags;
> >>>>>>> + __u32 field;
> >>>>>>> + __u64 timestamp;
> >>>>>>> + __u32 sequence;
> >>>>>>> + __u32 memory;
> >>>>>>> + __u32 request_fd;
> >>>>>>
> >>>>>> This should be __s32, at least for consistency with dmabuf_fd?
> >>>>>
> >>>>> I see that in struct v4l2_buffer, we have __s32, I don't mind changing it
> >>>>> to keep the consistency, I just don't see where this value can be a negative
> >>>>> number.
> >>>>
> >>>> here
> >>>> https://elixir.bootlin.com/linux/v5.8-rc4/source/drivers/media/common/videobuf2/videobuf2-v4l2.c#L134
> >>>
> >>> I saw that -1 is used to signal an invalid value, but I was just wondering when request_fd = 0 is valid.
> >>
> >> The request_fd is valid system wide file descriptor and request_fd = 0
> >> is STDIN_FILENO thus IMO it is valid as far as we call it file descriptor.
> >>
> >>>
> >>>>
> >>>>>
> >>>>>>
> >>>>>>> + struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
> >>>>>>> + __u32 reserved[4];
> >>>>>>
> >>>>>> I think we have to reserve more words here for future extensions.
> >>>>>>
> >>>>>> I'd like also to propose to add here __s32 metadata_fd. The idea behind
> >>>>>> this is to have a way to pass per-frame metadata dmabuf buffers for
> >>>>>> synchronous type of metadata where the metadata is coming at the same
> >>>>>> time with data buffers. What would be the format of the metadata buffer
> >>>>>> is TBD.
> >>>>>>
> >>>>>> One option for metadata buffer format could be:
> >>>>>>
> >>>>>> header {
> >>>>>>    num_ctrls
> >>>>>>    array_of_ctrls [0..N]
> >>>>>>            ctrl_id
> >>>>>>            ctrl_size
> >>>>>>            ctrl_offset
> >>>>>> }
> >>>>>>
> >>>>>> data {
> >>>>>>    cid0    //offset of cid0 in dmabuf buffer
> >>>>>>    cid1
> >>>>>>    cidN
> >>>>>> }
> >>>>>
> >>>>> Would it be better if, instead of adding a medatata_fd inside struct v4l2_ext_buffer,
> >>>>> we create a new ioctl that gets this structs for the controls and sync them using the
> >>>>> Request API ?
> >>
> >> New ioctl means new syscall. There are use-cases where encoding
> >> framerate is 480 fps (and more in near future, for example 960fps) this
> >> means 480 more syscalls per second. I don't think this is optimal and
> >> scalable solution at all.
> >>
> > 
> > Do you happen to have some data to confirm that it's indeed a problem?
> 
> If you mean profiling data, unfortunately I don't have such. But isn't
> it obvious that increasing metadata size (think of compound v4l2
> controls) and new syscalls isn't scalable solution in regards to higher
> framerates?

No, it certainly isn't obvious to me and that's why I'd prefer to see
some careful testing being done and hard numbers to justify discarding
the approach proposed and instead going away from the general API
conventions.

I believe it was already pointed out earlier, but direct access to
userspace buffers is problematic from security point of view, as the
kernel has no way to ensure the buffer contents stay valid while the
hardware is accessing them. If the hardware/firmware is buggy, the
userspace could modify the metadata buffer post submission and trigger
exploits. That's one of the reasons for the control framework actually
copying things.

> 
> I'm open to consider any suggestion different from v4l2 controls plus
> Request API.

I think we need to first prove that this approach doesn't work.

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 29+ messages in thread

end of thread, other threads:[~2020-08-05 16:55 UTC | newest]

Thread overview: 29+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-17 11:54 [PATCH v4 0/6] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
2020-07-17 11:54 ` [PATCH v4 1/6] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more) Helen Koike
2020-07-21 10:37   ` Hans Verkuil
2020-07-28 15:18     ` Helen Koike
2020-07-28 15:30       ` Hans Verkuil
2020-07-17 11:54 ` [PATCH v4 2/6] media: v4l2: Add extended buffer operations Helen Koike
2020-07-21 10:48   ` Hans Verkuil
2020-07-21 14:36     ` Helen Koike
2020-07-21 11:26   ` Stanimir Varbanov
2020-07-21 13:54     ` Helen Koike
2020-07-21 14:30       ` Stanimir Varbanov
2020-07-21 14:40         ` Helen Koike
2020-07-24 13:16           ` Stanimir Varbanov
2020-07-27 12:01             ` Helen Koike
2020-07-28 18:02               ` Stanimir Varbanov
2020-07-27 12:35             ` Tomasz Figa
2020-07-28 18:08               ` Stanimir Varbanov
2020-08-05 14:05                 ` Tomasz Figa
2020-07-17 11:54 ` [PATCH v4 3/6] media: videobuf2: Expose helpers to implement the _ext_fmt and _ext_buf hooks Helen Koike
2020-07-17 11:54 ` [PATCH v4 4/6] media: mediabus: Add helpers to convert a ext_pix format to/from a mbus_fmt Helen Koike
2020-07-17 11:54 ` [PATCH v4 5/6] media: vivid: Convert the capture and output drivers to EXT_FMT/EXT_BUF Helen Koike
2020-07-17 11:54 ` [PATCH v4 6/6] media: vimc: Implement the ext_fmt and ext_buf hooks Helen Koike
2020-07-20 11:57 ` [PATCH v4 0/6] media: v4l2: Add extended fmt and buffer ioctls Tomasz Figa
2020-07-20 19:47   ` Helen Koike
2020-07-21 10:24 ` Hans Verkuil
2020-07-21 14:23   ` Helen Koike
2020-07-21 14:34     ` Hans Verkuil
2020-07-21 11:15 ` Boris Brezillon
2020-07-21 13:56   ` Helen Koike

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.