[GIT PULL for v4.20-rc1] new experimental media request API

[GIT PULL for v4.20-rc1] new experimental media request API

[Mauro Carvalho Chehab]

Hi Linus,

Please pull from:

  git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media tags/media/v4.20-2

For a new media API: the request API

This API is needed to support device drivers that can dynamically
change their parameters for each new frame. The latest versions of 
Google camera and codec HAL depends on such feature.

At this stage, it supports only stateless codecs.

It has been discussed for a long time (at least over the last 3-4
years), and we finally reached to something that seem to work.

This series contain both the API and core changes required to support it
and a new m2m decoder driver (cedrus).

As the current API is still experimental, the only real driver using it
(cedrus) was added at staging[1]. We intend to keep it there for a while,
in order to test the API. Only when we're sure that this API works
for other cases (like encoders), we'll move this driver out of staging
and set the API into a stone.

[1] We added support for the vivid virtual driver (used only for testing)
to it too, as it makes easier to test the API for the ones that don't have
the cedrus hardware.


Thanks!
Mauro


The following changes since commit d842a7cf938b6e0f8a1aa9f1aec0476c9a599310:

  media: adv7842: enable reduced fps detection (2018-08-31 10:03:51 -0400)

are available in the Git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media tags/media/v4.20-2

for you to fetch changes up to e4183d3256e3cd668e899d06af66da5aac3a51af:

  media: dt-bindings: Document the Rockchip VPU bindings (2018-10-05 07:00:43 -0400)

----------------------------------------------------------------
media updates for v4.20-rc1

----------------------------------------------------------------
Alexandre Courbot (2):
      media: Documentation: v4l: document request API
      media: videodev2.h: add request_fd field to v4l2_ext_controls

Ezequiel Garcia (1):
      media: dt-bindings: Document the Rockchip VPU bindings

Hans Verkuil (44):
      media: uapi/linux/media.h: add request API
      media: media-request: implement media requests
      media: media-request: add media_request_get_by_fd
      media: media-request: add media_request_object_find
      media: v4l2-device.h: add v4l2_device_supports_requests() helper
      media: v4l2-dev: lock req_queue_mutex
      media: v4l2-ctrls: v4l2_ctrl_add_handler: add from_other_dev
      media: v4l2-ctrls: prepare internal structs for request API
      media: v4l2-ctrls: alloc memory for p_req
      media: v4l2-ctrls: use ref in helper instead of ctrl
      media: v4l2-ctrls: add core request support
      media: v4l2-ctrls: support g/s_ext_ctrls for requests
      media: v4l2-ctrls: add v4l2_ctrl_request_hdl_find/put/ctrl_find functions
      media: videobuf2-v4l2: move __fill_v4l2_buffer() function
      media: videobuf2-v4l2: replace if by switch in __fill_vb2_buffer()
      media: vb2: store userspace data in vb2_v4l2_buffer
      media: davinci_vpfe: remove bogus vb2->state check
      media: vb2: drop VB2_BUF_STATE_PREPARED, use bool prepared/synced instead
      media: videodev2.h: Add request_fd field to v4l2_buffer
      media: vb2: add init_buffer buffer op
      media: videobuf2-core: embed media_request_object
      media: videobuf2-core: integrate with media requests
      media: videobuf2-v4l2: integrate with media requests
      media: videobuf2-core: add request helper functions
      media: videobuf2-v4l2: add vb2_request_queue/validate helpers
      media: videobuf2-core: add uses_requests/qbuf flags
      media: videobuf2-v4l2: refuse qbuf if queue uses requests or vv.
      media: v4l2-mem2mem: add vb2_m2m_request_queue
      media: vim2m: use workqueue
      media: vim2m: support requests
      media: vivid: add mc
      media: vivid: add request support
      media: media-request: return -EINVAL for invalid request_fds
      media: v4l2-ctrls: return -EACCES if request wasn't completed
      media: buffer.rst: only set V4L2_BUF_FLAG_REQUEST_FD for QBUF
      media: videodev2.h: add new capabilities for buffer types
      media: vb2: set reqbufs/create_bufs capabilities
      media: media-request: add media_request_(un)lock_for_access
      media: v4l2-ctrls: use media_request_(un)lock_for_access
      media: v4l2-ctrls: improve media_request_(un)lock_for_update
      media: media-request: EPERM -> EACCES/EBUSY
      media: media-request: update documentation
      media: v4l2-compat-ioctl32.c: add missing documentation for a field
      media: v4l2-ctrls.c: initialize an error return code with zero

Paul Kocialkowski (5):
      media: videobuf2-core: Rework and rename helper for request buffer count
      media: v4l: Add definitions for MPEG-2 slice format and metadata
      media: v4l: Add definition for the Sunxi tiled NV12 format
      media: dt-bindings: media: Document bindings for the Cedrus VPU driver
      media: platform: Add Cedrus VPU decoder driver

Sakari Ailus (1):
      media: doc: Add media-request.h header to documentation build

 Documentation/devicetree/bindings/media/cedrus.txt |  54 ++
 .../devicetree/bindings/media/rockchip-vpu.txt     |  29 +
 Documentation/media/kapi/mc-core.rst               |   2 +
 .../media/uapi/mediactl/media-controller.rst       |   1 +
 Documentation/media/uapi/mediactl/media-funcs.rst  |   6 +
 .../uapi/mediactl/media-ioc-request-alloc.rst      |  66 +++
 .../uapi/mediactl/media-request-ioc-queue.rst      |  78 +++
 .../uapi/mediactl/media-request-ioc-reinit.rst     |  51 ++
 Documentation/media/uapi/mediactl/request-api.rst  | 252 +++++++++
 .../media/uapi/mediactl/request-func-close.rst     |  49 ++
 .../media/uapi/mediactl/request-func-ioctl.rst     |  67 +++
 .../media/uapi/mediactl/request-func-poll.rst      |  77 +++
 Documentation/media/uapi/v4l/buffer.rst            |  29 +-
 Documentation/media/uapi/v4l/extended-controls.rst | 176 ++++++
 Documentation/media/uapi/v4l/pixfmt-compressed.rst |  16 +
 Documentation/media/uapi/v4l/pixfmt-reserved.rst   |  15 +-
 .../media/uapi/v4l/vidioc-create-bufs.rst          |  14 +-
 .../media/uapi/v4l/vidioc-g-ext-ctrls.rst          |  59 +-
 Documentation/media/uapi/v4l/vidioc-qbuf.rst       |  37 +-
 Documentation/media/uapi/v4l/vidioc-queryctrl.rst  |  14 +-
 Documentation/media/uapi/v4l/vidioc-reqbufs.rst    |  42 +-
 Documentation/media/videodev2.h.rst.exceptions     |   3 +
 MAINTAINERS                                        |   7 +
 drivers/media/Makefile                             |   3 +-
 drivers/media/common/videobuf2/videobuf2-core.c    | 260 +++++++--
 drivers/media/common/videobuf2/videobuf2-v4l2.c    | 528 ++++++++++++------
 drivers/media/dvb-core/dvb_vb2.c                   |   5 +-
 drivers/media/dvb-frontends/rtl2832_sdr.c          |   5 +-
 drivers/media/media-device.c                       |  24 +-
 drivers/media/media-request.c                      | 501 +++++++++++++++++
 drivers/media/pci/bt8xx/bttv-driver.c              |   2 +-
 drivers/media/pci/cx23885/cx23885-417.c            |   2 +-
 drivers/media/pci/cx88/cx88-blackbird.c            |   2 +-
 drivers/media/pci/cx88/cx88-video.c                |   2 +-
 drivers/media/pci/saa7134/saa7134-empress.c        |   4 +-
 drivers/media/pci/saa7134/saa7134-video.c          |   2 +-
 drivers/media/platform/exynos4-is/fimc-capture.c   |   2 +-
 drivers/media/platform/omap3isp/ispvideo.c         |   4 +-
 drivers/media/platform/rcar-vin/rcar-core.c        |   2 +-
 drivers/media/platform/rcar_drif.c                 |   2 +-
 drivers/media/platform/s3c-camif/camif-capture.c   |   4 +-
 drivers/media/platform/s5p-mfc/s5p_mfc_dec.c       |   4 +-
 drivers/media/platform/s5p-mfc/s5p_mfc_enc.c       |   4 +-
 drivers/media/platform/soc_camera/soc_camera.c     |   7 +-
 drivers/media/platform/vim2m.c                     |  50 +-
 drivers/media/platform/vivid/vivid-core.c          |  74 +++
 drivers/media/platform/vivid/vivid-core.h          |   8 +
 drivers/media/platform/vivid/vivid-ctrls.c         |  46 +-
 drivers/media/platform/vivid/vivid-kthread-cap.c   |  12 +
 drivers/media/platform/vivid/vivid-kthread-out.c   |  12 +
 drivers/media/platform/vivid/vivid-sdr-cap.c       |  16 +
 drivers/media/platform/vivid/vivid-vbi-cap.c       |  10 +
 drivers/media/platform/vivid/vivid-vbi-out.c       |  10 +
 drivers/media/platform/vivid/vivid-vid-cap.c       |  10 +
 drivers/media/platform/vivid/vivid-vid-out.c       |  10 +
 drivers/media/usb/cpia2/cpia2_v4l.c                |   2 +-
 drivers/media/usb/cx231xx/cx231xx-417.c            |   2 +-
 drivers/media/usb/cx231xx/cx231xx-video.c          |   4 +-
 drivers/media/usb/msi2500/msi2500.c                |   2 +-
 drivers/media/usb/tm6000/tm6000-video.c            |   2 +-
 drivers/media/usb/uvc/uvc_queue.c                  |   5 +-
 drivers/media/usb/uvc/uvc_v4l2.c                   |   3 +-
 drivers/media/usb/uvc/uvcvideo.h                   |   1 +
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c      |  19 +-
 drivers/media/v4l2-core/v4l2-ctrls.c               | 612 ++++++++++++++++++++-
 drivers/media/v4l2-core/v4l2-dev.c                 |  18 +-
 drivers/media/v4l2-core/v4l2-device.c              |   3 +-
 drivers/media/v4l2-core/v4l2-ioctl.c               |  50 +-
 drivers/media/v4l2-core/v4l2-mem2mem.c             |  67 ++-
 drivers/media/v4l2-core/v4l2-subdev.c              |   9 +-
 drivers/staging/media/Kconfig                      |   2 +
 drivers/staging/media/Makefile                     |   1 +
 drivers/staging/media/davinci_vpfe/vpfe_video.c    |   7 +-
 drivers/staging/media/imx/imx-media-dev.c          |   2 +-
 drivers/staging/media/imx/imx-media-fim.c          |   2 +-
 drivers/staging/media/omap4iss/iss_video.c         |   3 +-
 drivers/staging/media/sunxi/Kconfig                |  15 +
 drivers/staging/media/sunxi/Makefile               |   1 +
 drivers/staging/media/sunxi/cedrus/Kconfig         |  14 +
 drivers/staging/media/sunxi/cedrus/Makefile        |   3 +
 drivers/staging/media/sunxi/cedrus/TODO            |   7 +
 drivers/staging/media/sunxi/cedrus/cedrus.c        | 431 +++++++++++++++
 drivers/staging/media/sunxi/cedrus/cedrus.h        | 167 ++++++
 drivers/staging/media/sunxi/cedrus/cedrus_dec.c    |  70 +++
 drivers/staging/media/sunxi/cedrus/cedrus_dec.h    |  27 +
 drivers/staging/media/sunxi/cedrus/cedrus_hw.c     | 327 +++++++++++
 drivers/staging/media/sunxi/cedrus/cedrus_hw.h     |  30 +
 drivers/staging/media/sunxi/cedrus/cedrus_mpeg2.c  | 246 +++++++++
 drivers/staging/media/sunxi/cedrus/cedrus_regs.h   | 235 ++++++++
 drivers/staging/media/sunxi/cedrus/cedrus_video.c  | 542 ++++++++++++++++++
 drivers/staging/media/sunxi/cedrus/cedrus_video.h  |  30 +
 drivers/usb/gadget/function/uvc_queue.c            |   2 +-
 include/media/media-device.h                       |  29 +
 include/media/media-request.h                      | 442 +++++++++++++++
 include/media/v4l2-ctrls.h                         | 141 ++++-
 include/media/v4l2-device.h                        |  11 +
 include/media/v4l2-mem2mem.h                       |   4 +
 include/media/videobuf2-core.h                     |  64 ++-
 include/media/videobuf2-v4l2.h                     |  20 +-
 include/uapi/linux/media.h                         |   8 +
 include/uapi/linux/v4l2-controls.h                 |  65 +++
 include/uapi/linux/videodev2.h                     |  33 +-
 102 files changed, 6166 insertions(+), 385 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/media/cedrus.txt
 create mode 100644 Documentation/devicetree/bindings/media/rockchip-vpu.txt
 create mode 100644 Documentation/media/uapi/mediactl/media-ioc-request-alloc.rst
 create mode 100644 Documentation/media/uapi/mediactl/media-request-ioc-queue.rst
 create mode 100644 Documentation/media/uapi/mediactl/media-request-ioc-reinit.rst
 create mode 100644 Documentation/media/uapi/mediactl/request-api.rst
 create mode 100644 Documentation/media/uapi/mediactl/request-func-close.rst
 create mode 100644 Documentation/media/uapi/mediactl/request-func-ioctl.rst
 create mode 100644 Documentation/media/uapi/mediactl/request-func-poll.rst
 create mode 100644 drivers/media/media-request.c
 create mode 100644 drivers/staging/media/sunxi/Kconfig
 create mode 100644 drivers/staging/media/sunxi/Makefile
 create mode 100644 drivers/staging/media/sunxi/cedrus/Kconfig
 create mode 100644 drivers/staging/media/sunxi/cedrus/Makefile
 create mode 100644 drivers/staging/media/sunxi/cedrus/TODO
 create mode 100644 drivers/staging/media/sunxi/cedrus/cedrus.c
 create mode 100644 drivers/staging/media/sunxi/cedrus/cedrus.h
 create mode 100644 drivers/staging/media/sunxi/cedrus/cedrus_dec.c
 create mode 100644 drivers/staging/media/sunxi/cedrus/cedrus_dec.h
 create mode 100644 drivers/staging/media/sunxi/cedrus/cedrus_hw.c
 create mode 100644 drivers/staging/media/sunxi/cedrus/cedrus_hw.h
 create mode 100644 drivers/staging/media/sunxi/cedrus/cedrus_mpeg2.c
 create mode 100644 drivers/staging/media/sunxi/cedrus/cedrus_regs.h
 create mode 100644 drivers/staging/media/sunxi/cedrus/cedrus_video.c
 create mode 100644 drivers/staging/media/sunxi/cedrus/cedrus_video.h
 create mode 100644 include/media/media-request.h

Re: [GIT PULL for v4.20-rc1] new experimental media request API

[Linus Torvalds]

On Tue, Oct 30, 2018 at 6:53 AM Mauro Carvalho Chehab
<mchehab+samsung@kernel.org> wrote:
>
> For a new media API: the request API

Ugh. I don't know how much being in staging matters - if people start
using it, they start using it.

"Staging" does not mean "regressions don't matter".

But pulled,

                Linus

Re: [GIT PULL for v4.20-rc1] new experimental media request API

[Mauro Carvalho Chehab]

Hi Linus,

Em Wed, 31 Oct 2018 11:05:09 -0700
Linus Torvalds <torvalds@linux-foundation.org> escreveu:

> On Tue, Oct 30, 2018 at 6:53 AM Mauro Carvalho Chehab
> <mchehab+samsung@kernel.org> wrote:
> >
> > For a new media API: the request API  
> 
> Ugh. I don't know how much being in staging matters - if people start
> using it, they start using it.
> 
> "Staging" does not mean "regressions don't matter".

Yes, I know.

This shouldn't affect normal cameras and generic V4L2 apps, as this
is for very advanced use cases. So, we hope that people won't start
using it for a while. 

The main interested party on this is Google CromeOS. We're working 
together with them in order to do upstream first. They are well aware
that the API may change. So, I don't expect any complaints from their
side if the API would require further changes.

The point is that this API is complex and ensuring that it will
work properly is not easy. We've been thinking about a solution for
the Camera HAL 2 for a long time (I guess the first discussions were
done back in 2008).

The big problem is that V4L2 API was designed to work with a stream,
while Google HAL API expects to have control for each individual
frame.

The Google API allows, for example that, inside a stream, the
first frame would have a VGA resolution, the next one a 4K resolution
(for example, when the user clicks on a camera button) and then returning
back to VGA (it actually allows full control for every single frame). 

This is something that it is not possible to do with the "standard" 
V4L2 API without stopping and restarting a stream (with increases
a lot the latency).

Solving it is so complex that we decided to start with a completely
new type of Linux media drivers (stateless decoders). In long term, 
the same API should be used by not only by decoders, but also for 
encoders and complex cameras (those with an image signal processor 
inside a SoC chipset).

In order to be sure that it is possible to implement the way we did,
We need to be able to add it to the Kernel somehow and to have
enough drivers that will let us test all possible scenarios.

That will allow to adapt a version of the camera HAL for testing
and see if it behaves as expected.

> But pulled,

Thanks! Anyway, we'll try to rush the tests for this API in order to
try sending any fixes that might be disruptive before the final
release.

Regards,
Mauro

Re: [GIT PULL for v4.20-rc1] new experimental media request API

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 1220 bytes --]

Hi!

> Em Wed, 31 Oct 2018 11:05:09 -0700
> Linus Torvalds <torvalds@linux-foundation.org> escreveu:
> 
> > On Tue, Oct 30, 2018 at 6:53 AM Mauro Carvalho Chehab
> > <mchehab+samsung@kernel.org> wrote:
> > >
> > > For a new media API: the request API  
> > 
> > Ugh. I don't know how much being in staging matters - if people start
> > using it, they start using it.
> > 
> > "Staging" does not mean "regressions don't matter".
> 
> Yes, I know.
> 
> This shouldn't affect normal cameras and generic V4L2 apps, as this
> is for very advanced use cases. So, we hope that people won't start
> using it for a while. 
> 
> The main interested party on this is Google CromeOS. We're working 
> together with them in order to do upstream first. They are well aware
> that the API may change. So, I don't expect any complaints from their
> side if the API would require further changes.

You may want to simply disable it in Kconfig... ChromeOS people can
enable it easily, and if it never worked in the mainline, you get some
wiggle room :-).
								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [GIT PULL for v4.20-rc1] new experimental media request API

[Hans Verkuil]

On 11/11/2018 09:28 PM, Pavel Machek wrote:
> Hi!
> 
>> Em Wed, 31 Oct 2018 11:05:09 -0700
>> Linus Torvalds <torvalds@linux-foundation.org> escreveu:
>>
>>> On Tue, Oct 30, 2018 at 6:53 AM Mauro Carvalho Chehab
>>> <mchehab+samsung@kernel.org> wrote:
>>>>
>>>> For a new media API: the request API  
>>>
>>> Ugh. I don't know how much being in staging matters - if people start
>>> using it, they start using it.
>>>
>>> "Staging" does not mean "regressions don't matter".
>>
>> Yes, I know.
>>
>> This shouldn't affect normal cameras and generic V4L2 apps, as this
>> is for very advanced use cases. So, we hope that people won't start
>> using it for a while. 
>>
>> The main interested party on this is Google CromeOS. We're working 
>> together with them in order to do upstream first. They are well aware
>> that the API may change. So, I don't expect any complaints from their
>> side if the API would require further changes.
> 
> You may want to simply disable it in Kconfig... ChromeOS people can
> enable it easily, and if it never worked in the mainline, you get some
> wiggle room :-).
> 								Pavel
> 

For the record: there are no known issues with the Request API. The
only outstanding issue is with a control introduced in the staging cedrus
driver which will have to change since the method used to refer to MPEG
reference frames is wrong. We're working on that and should have this
fixed soon (patches have been posted and I'm waiting for test feedback).

This control has nothing to do with the Request API, so I see no reason
to put the Request API under a config option. Drivers needs to explicit
enable support for the Request API anyway, so all existing drivers will
return an error if userspace attempts to use this API.

Regards,

	Hans

Re: [GIT PULL for v4.20-rc1] new experimental media request API

[Linus Torvalds]

On Wed, Oct 31, 2018 at 11:05 AM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> But pulled,

I have no idea how I missed this during the actual test compile after
the pull (and yes, I'm sure I did one), but after doing a couple of
more pulls I finally did notice.

After the media pull I get this warning:

  ./usr/include/linux/v4l2-controls.h:1105: found __[us]{8,16,32,64}
type without #include <linux/types.h>

and sure enough, the recent changes to

  include/uapi/linux/v4l2-controls.h

add those new structures use the "__uXY" types without including the
header to define them.

It's harmless in the short term and the kernel build itself obviously
doesn't care apart from the warning, but please fix it.

                     Linus

Re: [GIT PULL for v4.20-rc1] new experimental media request API

[Mauro Carvalho Chehab]

Em Wed, 31 Oct 2018 15:32:03 -0700
Linus Torvalds <torvalds@linux-foundation.org> escreveu:

> On Wed, Oct 31, 2018 at 11:05 AM Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
> >
> > But pulled,  
> 
> I have no idea how I missed this during the actual test compile after
> the pull (and yes, I'm sure I did one), but after doing a couple of
> more pulls I finally did notice.
> 
> After the media pull I get this warning:
> 
>   ./usr/include/linux/v4l2-controls.h:1105: found __[us]{8,16,32,64}
> type without #include <linux/types.h>
> 
> and sure enough, the recent changes to
> 
>   include/uapi/linux/v4l2-controls.h
> 
> add those new structures use the "__uXY" types without including the
> header to define them.
> 
> It's harmless in the short term and the kernel build itself obviously
> doesn't care apart from the warning, but please fix it.

I also missed this one. Perhaps it depends on gcc version, or is it
a new warning after some changes? I remember there was some patchsets
floating around related to change some warnings.

Anyway, I'll send you a fix for it soon.

Thanks,
Mauro