Linux-Media Archive on lore.kernel.org
 help / color / Atom feed
From: Enrico Granata <egranata@google.com>
To: Gerd Hoffmann <kraxel@redhat.com>
Cc: "Keiichi Watanabe" <keiichiw@chromium.org>,
	"Dmitry Sepp" <dmitry.sepp@opensynergy.com>,
	virtio-dev@lists.oasis-open.org,
	"Tomasz Figa" <tfiga@chromium.org>,
	"Linux Media Mailing List" <linux-media@vger.kernel.org>,
	"Alexandre Courbot" <acourbot@chromium.org>,
	"Alex Lau" <alexlau@chromium.org>,
	"Dylan Reid" <dgreid@chromium.org>,
	"Stéphane Marchesin" <marcheu@chromium.org>,
	"Pawel Osciak" <posciak@chromium.org>,
	"David Stevens" <stevensd@chromium.org>,
	"Hans Verkuil" <hverkuil@xs4all.nl>,
	"Daniel Vetter" <daniel@ffwll.ch>
Subject: Re: [virtio-dev] [RFC RESEND] virtio-video: Add virtio video device specification
Date: Wed, 4 Dec 2019 11:11:49 -0800
Message-ID: <CAPR809uYEjSGqPA57HiDgTf7MESrXnsCKwvSEQQ8LfMvCJwaaQ@mail.gmail.com> (raw)
In-Reply-To: <20191204091620.zpnd7jttkpkduort@sirius.home.kraxel.org>

On Wed, Dec 4, 2019 at 1:16 AM Gerd Hoffmann <kraxel@redhat.com> wrote:
>
>   Hi,
>
> > 1. Focus on only decoder/encoder functionalities first.
> >
> > As Tomasz said earlier in this thread, it'd be too complicated to support camera
> > usage at the same time. So, I'd suggest to make it just a generic mem-to-mem
> > video processing device protocol for now.
> > If we finally decide to support camera in this protocol, we can add it later.
>
> Agree.
>
> > 2. Only one feature bit can be specified for one device.
> >
> > I'd like to have a decoder device and encoder device separately.
> > It'd be natural to assume it because a decoder and an encoder are provided as
> > different hardware.
>
> Hmm, modern GPUs support both encoding and decoding ...
>
> I don't think we should bake that restriction into the specification.
> It probably makes sense to use one virtqueue per function though, that
> will simplify dispatching in both host and guest.
>
> > 3. Separate buffer allocation functionalities from virtio-video protocol.
> >
> > To support various ways of guest/host buffer sharing, we might want to have a
> > dedicated buffer sharing device as we're discussing in another thread. Until we
> > reach consensus there, it'd be good not to have buffer allocation
> > functionalities in virtio-video.
>
> I think virtio-video should be able to work as stand-alone device,
> so we need some way to allocate buffers ...
>
> Buffer sharing with other devices can be added later.
>
> > > +The virtio video device is a virtual video streaming device that supports the
> > > +following functions: encoder, decoder, capture, output.
> > > +
> > > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
> > > +
> > > +TBD.
> >
> > I'm wondering how and when we can determine and reserve this ID?
>
> Grab the next free, update the spec accordingly, submit the one-line
> patch.
>
> > > +\begin{lstlisting}
> > > +enum virtio_video_pixel_format {
> > > +       VIRTIO_VIDEO_PIX_FMT_UNDEFINED = 0,
> > > +
> > > +       VIRTIO_VIDEO_PIX_FMT_H264 = 0x0100,
> > > +       VIRTIO_VIDEO_PIX_FMT_NV12,
> > > +       VIRTIO_VIDEO_PIX_FMT_NV21,
> > > +       VIRTIO_VIDEO_PIX_FMT_I420,
> > > +       VIRTIO_VIDEO_PIX_FMT_I422,
> > > +       VIRTIO_VIDEO_PIX_FMT_XBGR,
> > > +};
> >
> > I'm wondering if we can use FOURCC instead. So, we can avoid reinventing a
> > mapping from formats to integers.
> > Also, I suppose the word "pixel formats" means only raw (decoded) formats.
> > But, it can be encoded format like H.264. So, I guess "image format" or
> > "fourcc" is a better word choice.
>
> Use separate pixel_format (fourcc) and stream_format (H.264 etc.) enums?
>
> > > +\begin{lstlisting}
> > > +struct virtio_video_function {
> > > +       struct virtio_video_desc desc;
> > > +       __le32 function_type; /* One of VIRTIO_VIDEO_FUNC_* types */
> > > +       __le32 function_id;
> > > +       struct virtio_video_params in_params;
> > > +       struct virtio_video_params out_params;
> > > +       __le32 num_caps;
> > > +       __u8 padding[4];
> > > +       /* Followed by struct virtio_video_capability video_caps[]; */
> > > +};
> > > +\end{lstlisting}
> >
> > If one device only has one functionality, virtio_video_function's fields will be
> > no longer needed except in_params and out_params. So, we'd be able to remove
> > virtio_video_function and have in_params and out_params in
> > virtio_video_capability instead.
>
> Same goes for per-function virtqueues (used virtqueue implies function).
>
> > > +\begin{lstlisting}
> > > +struct virtio_video_resource_detach_backing {
> > > +       struct virtio_video_ctrl_hdr hdr;
> > > +       __le32 resource_id;
> > > +       __u8 padding[4];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{resource_id}] internal id of the resource.
> > > +\end{description}
> >
> > I suppose that it'd be better not to have the above series of T_RESOURCE
> > controls at least until we reach a conclusion in the thread of buffer-sharing
> > device. If we end up concluding this type of controls is the best way, we'll be
> > able to revisit here.
>

As an interim solution, this form of "manual resource backing-store
management" could be specified as a feature flag.
Once there is an established solution for buffer sharing, we would
then go ahead and introduce a new feature flag for "works with the
buffer sharing mechanism", as an alternative to this manual mechanism.

wdyt?

> Well.  For buffer management there are a bunch of options.
>
>  (1) Simply stick the buffers (well, pointers to the buffer pages) into
>      the virtqueue.  This is the standard virtio way.
>
>  (2) Create resources, then put the resource ids into the virtqueue.
>      virtio-gpu uses that model.  First, because virtio-gpu needs an id
>      to reference resources in the rendering command stream
>      (virtio-video doesn't need this).  Also because (some kinds of)
>      resources are around for a long time and the guest-physical ->
>      host-virtual mapping needs to be done only once that way (which
>      I think would be the case for virtio-video too because v4l2
>      re-uses buffers in robin-round fashion).  Drawback is this
>      assumes shared memory between host and guest (which is the case
>      in typical use cases but it is not mandated by the virtio spec).
>
>  (3) Import external resources (from virtio-gpu for example).
>      Out of scope for now, will probably added as optional feature
>      later.
>
> I guess long-term we want support either (1)+(3) or (2)+(3).
>
> cheers,
>   Gerd
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>

  reply index

Thread overview: 29+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-11-05 19:19 Dmitry Sepp
2019-11-07  9:56 ` [virtio-dev] " Gerd Hoffmann
2019-11-07 13:09   ` Dmitry Sepp
2019-11-08  7:49     ` Gerd Hoffmann
2019-11-08  7:58       ` Tomasz Figa
2019-11-08  9:51       ` Dmitry Sepp
2019-11-08  7:50     ` Tomasz Figa
2019-11-08  9:05       ` Gerd Hoffmann
2019-11-08  9:28         ` Keiichi Watanabe
2019-11-20 11:29           ` Gerd Hoffmann
2019-11-21 10:54             ` Dmitry Sepp
2019-12-04  7:48               ` Keiichi Watanabe
2019-12-04  9:16                 ` Gerd Hoffmann
2019-12-04 19:11                   ` Enrico Granata [this message]
2019-12-05  8:21                     ` Keiichi Watanabe
2019-12-06  7:32                       ` Gerd Hoffmann
2019-12-06 12:30                         ` Keiichi Watanabe
2019-12-06 15:50                           ` Enrico Granata
2019-12-09 13:43                             ` Keiichi Watanabe
2019-12-09 10:46                           ` Gerd Hoffmann
2019-12-09 11:38                             ` Dmitry Sepp
2019-12-09 13:17                               ` Keiichi Watanabe
2019-12-09 14:19                   ` Dmitry Sepp
     [not found]                     ` <CAPR809t2X3eEZj14Y-0CdnmzGZFhWKt2vwFSZBrEZbChQpmU_w@mail.gmail.com>
2019-12-10 13:16                       ` Dmitry Sepp
2019-12-12  5:39                         ` Keiichi Watanabe
2019-12-12 10:34                           ` Dmitry Sepp
2019-12-13 14:20                             ` Keiichi Watanabe
2019-12-13 16:31                               ` Keiichi Watanabe
2019-12-13 14:58                     ` Christophe de Dinechin

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAPR809uYEjSGqPA57HiDgTf7MESrXnsCKwvSEQQ8LfMvCJwaaQ@mail.gmail.com \
    --to=egranata@google.com \
    --cc=acourbot@chromium.org \
    --cc=alexlau@chromium.org \
    --cc=daniel@ffwll.ch \
    --cc=dgreid@chromium.org \
    --cc=dmitry.sepp@opensynergy.com \
    --cc=hverkuil@xs4all.nl \
    --cc=keiichiw@chromium.org \
    --cc=kraxel@redhat.com \
    --cc=linux-media@vger.kernel.org \
    --cc=marcheu@chromium.org \
    --cc=posciak@chromium.org \
    --cc=stevensd@chromium.org \
    --cc=tfiga@chromium.org \
    --cc=virtio-dev@lists.oasis-open.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Media Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-media/0 linux-media/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-media linux-media/ https://lore.kernel.org/linux-media \
		linux-media@vger.kernel.org
	public-inbox-index linux-media

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-media


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git