From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
To: Enrico Granata <egranata@google.com>
Cc: "Changyeon Jo" <changyeon@google.com>,
virtio-dev@lists.oasis-open.org,
"Gerd Hoffmann" <kraxel@redhat.com>,
"Keiichi Watanabe" <keiichiw@chromium.org>,
"Tomasz Figa" <tfiga@chromium.org>,
"Linux Media Mailing List" <linux-media@vger.kernel.org>,
"Alexandre Courbot" <acourbot@chromium.org>,
"Alex Lau" <alexlau@chromium.org>,
"Dylan Reid" <dgreid@chromium.org>,
"Stéphane Marchesin" <marcheu@chromium.org>,
"Pawel Osciak" <posciak@chromium.org>,
"David Stevens" <stevensd@chromium.org>,
"Hans Verkuil" <hverkuil@xs4all.nl>,
"Daniel Vetter" <daniel@ffwll.ch>
Subject: Re: [virtio-dev] [RFC RESEND] virtio-video: Add virtio video device specification
Date: Tue, 10 Dec 2019 14:16:03 +0100 [thread overview]
Message-ID: <2278645.2VI2HTiyL8@os-lin-dmo> (raw)
In-Reply-To: <CAPR809t2X3eEZj14Y-0CdnmzGZFhWKt2vwFSZBrEZbChQpmU_w@mail.gmail.com>
Hi,
Just to start, let's consider this v4l2 control:
V4L2_CID_MPEG_VIDEO_FRAME_RC_ENABLE.
As I can see, this control is referenced as a mandatory one in the Chromium
sources [1].
So could someone from the Chromium team please explain why it is mandatory?
(YouTube?) In fact, almost no encoders implement this control. Do we need it
in virtio-video?
[1] https://chromium.googlesource.com/chromium/src/media/+/refs/heads/master/
gpu/v4l2/v4l2_video_encode_accelerator.cc#1500
Regards,
Dmitry.
On Montag, 9. Dezember 2019 22:12:28 CET Enrico Granata wrote:
> +Changyeon Jo <changyeon@google.com> for his awareness
>
> Thanks,
> - Enrico
>
>
> On Mon, Dec 9, 2019 at 6:20 AM Dmitry Sepp <dmitry.sepp@opensynergy.com>
>
> wrote:
> > Hello,
> >
> > I'd like to invite everyone to share ideas regarding required encoder
> > features
> > in this separate sub-tree.
> >
> > In general, encoder devices are more complex compared to decoders. So the
> > question I'd like to rise is in what way we define the minimal subset of
> > features to be implemented by the virtio-video.
> >
> > We may look at the following to define the set of features:
> > 1. USB video, 2.3.6 Encoding Unit [1].
> > 2. Android 10 Compatibility Definition [2].
> >
> > Would be nice to hear about any specific requirements from the Chromium
> > team as
> > well.
> >
> > [1] https://www.usb.org/sites/default/files/USB_Video_Class_1_5.zip
> > [2]
> > https://source.android.com/compatibility/android-cdd#5_2_video_encoding
> >
> > Thank you.
> >
> > Best regards,
> > Dmitry.
> >
> > On Mittwoch, 4. Dezember 2019 10:16:20 CET Gerd Hoffmann wrote:
> > > Hi,
> > >
> > > > 1. Focus on only decoder/encoder functionalities first.
> > > >
> > > > As Tomasz said earlier in this thread, it'd be too complicated to
> >
> > support
> >
> > > > camera usage at the same time. So, I'd suggest to make it just a
> >
> > generic
> >
> > > > mem-to-mem video processing device protocol for now.
> > > > If we finally decide to support camera in this protocol, we can add it
> > > > later.
> > >
> > > Agree.
> > >
> > > > 2. Only one feature bit can be specified for one device.
> > > >
> > > > I'd like to have a decoder device and encoder device separately.
> > > > It'd be natural to assume it because a decoder and an encoder are
> >
> > provided
> >
> > > > as different hardware.
> > >
> > > Hmm, modern GPUs support both encoding and decoding ...
> > >
> > > I don't think we should bake that restriction into the specification.
> > > It probably makes sense to use one virtqueue per function though, that
> > > will simplify dispatching in both host and guest.
> > >
> > > > 3. Separate buffer allocation functionalities from virtio-video
> >
> > protocol.
> >
> > > > To support various ways of guest/host buffer sharing, we might want to
> > > > have a dedicated buffer sharing device as we're discussing in another
> > > > thread. Until we reach consensus there, it'd be good not to have
> > > > buffer
> > > > allocation
> > > > functionalities in virtio-video.
> > >
> > > I think virtio-video should be able to work as stand-alone device,
> > > so we need some way to allocate buffers ...
> > >
> > > Buffer sharing with other devices can be added later.
> > >
> > > > > +The virtio video device is a virtual video streaming device that
> > > > > supports the +following functions: encoder, decoder, capture,
> > > > > output.
> > > > > +
> > > > > +\subsection{Device ID}\label{sec:Device Types / Video Device /
> >
> > Device
> >
> > > > > ID}
> > > > > +
> > > > > +TBD.
> > > >
> > > > I'm wondering how and when we can determine and reserve this ID?
> > >
> > > Grab the next free, update the spec accordingly, submit the one-line
> > > patch.
> > >
> > > > > +\begin{lstlisting}
> > > > > +enum virtio_video_pixel_format {
> > > > > + VIRTIO_VIDEO_PIX_FMT_UNDEFINED = 0,
> > > > > +
> > > > > + VIRTIO_VIDEO_PIX_FMT_H264 = 0x0100,
> > > > > + VIRTIO_VIDEO_PIX_FMT_NV12,
> > > > > + VIRTIO_VIDEO_PIX_FMT_NV21,
> > > > > + VIRTIO_VIDEO_PIX_FMT_I420,
> > > > > + VIRTIO_VIDEO_PIX_FMT_I422,
> > > > > + VIRTIO_VIDEO_PIX_FMT_XBGR,
> > > > > +};
> > > >
> > > > I'm wondering if we can use FOURCC instead. So, we can avoid
> >
> > reinventing a
> >
> > > > mapping from formats to integers.
> > > > Also, I suppose the word "pixel formats" means only raw (decoded)
> >
> > formats.
> >
> > > > But, it can be encoded format like H.264. So, I guess "image format"
> > > > or
> > > > "fourcc" is a better word choice.
> > >
> > > Use separate pixel_format (fourcc) and stream_format (H.264 etc.) enums?
> > >
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_video_function {
> > > > > + struct virtio_video_desc desc;
> > > > > + __le32 function_type; /* One of VIRTIO_VIDEO_FUNC_* types */
> > > > > + __le32 function_id;
> > > > > + struct virtio_video_params in_params;
> > > > > + struct virtio_video_params out_params;
> > > > > + __le32 num_caps;
> > > > > + __u8 padding[4];
> > > > > + /* Followed by struct virtio_video_capability video_caps[];
> >
> > */
> >
> > > > > +};
> > > > > +\end{lstlisting}
> > > >
> > > > If one device only has one functionality, virtio_video_function's
> >
> > fields
> >
> > > > will be no longer needed except in_params and out_params. So, we'd be
> > > > able to remove virtio_video_function and have in_params and out_params
> >
> > in
> >
> > > > virtio_video_capability instead.
> > >
> > > Same goes for per-function virtqueues (used virtqueue implies function).
> > >
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_video_resource_detach_backing {
> > > > > + struct virtio_video_ctrl_hdr hdr;
> > > > > + __le32 resource_id;
> > > > > + __u8 padding[4];
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > +\end{description}
> > > >
> > > > I suppose that it'd be better not to have the above series of
> >
> > T_RESOURCE
> >
> > > > controls at least until we reach a conclusion in the thread of
> > > > buffer-sharing device. If we end up concluding this type of controls
> > > > is
> > > > the best way, we'll be able to revisit here.
> > >
> > > Well. For buffer management there are a bunch of options.
> > >
> > > (1) Simply stick the buffers (well, pointers to the buffer pages) into
> > >
> > > the virtqueue. This is the standard virtio way.
> > >
> > > (2) Create resources, then put the resource ids into the virtqueue.
> > >
> > > virtio-gpu uses that model. First, because virtio-gpu needs an id
> > > to reference resources in the rendering command stream
> > > (virtio-video doesn't need this). Also because (some kinds of)
> > > resources are around for a long time and the guest-physical ->
> > > host-virtual mapping needs to be done only once that way (which
> > > I think would be the case for virtio-video too because v4l2
> > > re-uses buffers in robin-round fashion). Drawback is this
> > > assumes shared memory between host and guest (which is the case
> > > in typical use cases but it is not mandated by the virtio spec).
> > >
> > > (3) Import external resources (from virtio-gpu for example).
> > >
> > > Out of scope for now, will probably added as optional feature
> > > later.
> > >
> > > I guess long-term we want support either (1)+(3) or (2)+(3).
> > >
> > > cheers,
> > >
> > > Gerd
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
next prev parent reply other threads:[~2019-12-10 13:16 UTC|newest]
Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-11-05 19:19 [RFC RESEND] virtio-video: Add virtio video device specification Dmitry Sepp
2019-11-07 9:56 ` [virtio-dev] " Gerd Hoffmann
2019-11-07 13:09 ` Dmitry Sepp
2019-11-08 7:49 ` Gerd Hoffmann
2019-11-08 7:58 ` Tomasz Figa
2019-11-08 9:51 ` Dmitry Sepp
2019-11-08 7:50 ` Tomasz Figa
2019-11-08 9:05 ` Gerd Hoffmann
2019-11-08 9:28 ` Keiichi Watanabe
2019-11-20 11:29 ` Gerd Hoffmann
2019-11-21 10:54 ` Dmitry Sepp
2019-12-04 7:48 ` Keiichi Watanabe
2019-12-04 9:16 ` Gerd Hoffmann
2019-12-04 19:11 ` Enrico Granata
2019-12-05 8:21 ` Keiichi Watanabe
2019-12-06 7:32 ` Gerd Hoffmann
2019-12-06 12:30 ` Keiichi Watanabe
2019-12-06 15:50 ` Enrico Granata
2019-12-09 13:43 ` Keiichi Watanabe
2019-12-09 10:46 ` Gerd Hoffmann
2019-12-09 11:38 ` Dmitry Sepp
2019-12-09 13:17 ` Keiichi Watanabe
2019-12-09 14:19 ` Dmitry Sepp
[not found] ` <CAPR809t2X3eEZj14Y-0CdnmzGZFhWKt2vwFSZBrEZbChQpmU_w@mail.gmail.com>
2019-12-10 13:16 ` Dmitry Sepp [this message]
2019-12-12 5:39 ` Keiichi Watanabe
2019-12-12 10:34 ` Dmitry Sepp
2019-12-13 14:20 ` Keiichi Watanabe
2019-12-13 16:31 ` Keiichi Watanabe
2019-12-20 14:24 ` Dmitry Sepp
2019-12-20 15:01 ` Keiichi Watanabe
2019-12-13 14:58 ` Christophe de Dinechin
2019-12-16 8:09 ` Tomasz Figa
2019-12-16 10:32 ` Gerd Hoffmann
2019-12-17 13:15 ` Tomasz Figa
2019-12-17 13:39 ` Gerd Hoffmann
2019-12-17 14:09 ` Keiichi Watanabe
2019-12-17 16:13 ` Dmitry Sepp
2019-12-18 6:43 ` Gerd Hoffmann
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=2278645.2VI2HTiyL8@os-lin-dmo \
--to=dmitry.sepp@opensynergy.com \
--cc=acourbot@chromium.org \
--cc=alexlau@chromium.org \
--cc=changyeon@google.com \
--cc=daniel@ffwll.ch \
--cc=dgreid@chromium.org \
--cc=egranata@google.com \
--cc=hverkuil@xs4all.nl \
--cc=keiichiw@chromium.org \
--cc=kraxel@redhat.com \
--cc=linux-media@vger.kernel.org \
--cc=marcheu@chromium.org \
--cc=posciak@chromium.org \
--cc=stevensd@chromium.org \
--cc=tfiga@chromium.org \
--cc=virtio-dev@lists.oasis-open.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).