All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v2 0/1] VirtIO video device specification
@ 2019-12-18 13:02 Keiichi Watanabe
  2019-12-18 13:02 ` [PATCH v2 1/1] virtio-video: Add virtio " Keiichi Watanabe
  2019-12-20 15:58 ` [PATCH v2 0/1] VirtIO " Dmitry Sepp
  0 siblings, 2 replies; 73+ messages in thread
From: Keiichi Watanabe @ 2019-12-18 13:02 UTC (permalink / raw)
  To: virtio-dev
  Cc: linux-media, acourbot, alexlau, daniel, dgreid, dmitry.sepp,
	egranata, fziglio, hverkuil, keiichiw, kraxel, marcheu, posciak,
	spice-devel, stevensd, tfiga, uril

Hi,
This is the 2nd version of virtio-video patch. The PDF is available in [1].
The first version was sent at [2].

Any feedback would be appreciated. Thank you.

Best,
Keiichi

[1]: https://drive.google.com/drive/folders/1eT5fEckBoor2iHZR4f4GLxYzFMVapOFx?usp=sharing
[2]: https://markmail.org/message/gc6h25acct22niut

Change log:

v2:
* Removed functionalities except encoding and decoding.
* Splited encoder and decoder into different devices that use the same protocol.
* Replaced GET_FUNCS with GET_CAPABILITY.
* Updated structs for capabilities.
  - Defined new structs and enums such as image formats, profiles, range (min,
  max, step), etc
    * For virtio_video_pixel_format, chose a naming convention that is used
      in DRM. We removed XBGR, NV21 and I422, as they are not used in the
      current draft implementation. https://lwn.net/Articles/806416/
  - Removed virtio_video_control, whose usage was not documented yet and which
    is not necessary for the simplest decoding scenario.
  - Removed virtio_video_desc, as it is no longer needed.
* Updated struct virtio_video_config for changes around capabilities.
* Added a way to represent supported combinations of formats.
  - A field "mask" in virtio_video_format_desc plays this role.
* Removed VIRTIO_VIDEO_T_STREAM_{START,STOP} because they don't play any meaningful roles.
* Removed VIRTIO_VIDEO_T_STREAM_{ATTACH, DETACH}_BACKING and merged them into RESOURCE_{CREATE, DESTROY}.
* Added a way to notify/specify resource creation method.
  - Added a feature flag.
  - Defined enum virtio_video_mem_type.
  - Added new fields in video_stream_create.
* Modified fields in virtio_video_params.
  - Added crop information.
* Removed enum virtio_video_channel_type because we can get this information by image format.
* Renamed virtio_video_pin to virtio_video_buf_type.
  - It's similar to V4L2_BUF_TYPE_VIDEO_{OUTPUT, CAPTURE}.
* Added an error event.
* Reordered some subsections.
* Changed styles to make it consistent with other devices.

Dmitry Sepp (1):
  virtio-video: Add virtio video device specification

 content.tex      |   1 +
 virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 580 insertions(+)
 create mode 100644 virtio-video.tex

--
2.24.1.735.g03f4e72817-goog

^ permalink raw reply	[flat|nested] 73+ messages in thread

* [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-18 13:02 [PATCH v2 0/1] VirtIO video device specification Keiichi Watanabe
@ 2019-12-18 13:02 ` Keiichi Watanabe
  2019-12-18 13:40   ` Gerd Hoffmann
                     ` (5 more replies)
  2019-12-20 15:58 ` [PATCH v2 0/1] VirtIO " Dmitry Sepp
  1 sibling, 6 replies; 73+ messages in thread
From: Keiichi Watanabe @ 2019-12-18 13:02 UTC (permalink / raw)
  To: virtio-dev
  Cc: linux-media, acourbot, alexlau, daniel, dgreid, dmitry.sepp,
	egranata, fziglio, hverkuil, keiichiw, kraxel, marcheu, posciak,
	spice-devel, stevensd, tfiga, uril

From: Dmitry Sepp <dmitry.sepp@opensynergy.com>

The virtio video encoder device and decoder device provide functionalities to
encode and decode video stream respectively.
Though video encoder and decoder are provided as different devices, they use a
same protocol.

Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
---
 content.tex      |   1 +
 virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 580 insertions(+)
 create mode 100644 virtio-video.tex

diff --git a/content.tex b/content.tex
index 556b373..9e56839 100644
--- a/content.tex
+++ b/content.tex
@@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
 \input{virtio-vsock.tex}
 \input{virtio-fs.tex}
 \input{virtio-rpmb.tex}
+\input{virtio-video.tex}
 
 \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 
diff --git a/virtio-video.tex b/virtio-video.tex
new file mode 100644
index 0000000..30e728d
--- /dev/null
+++ b/virtio-video.tex
@@ -0,0 +1,579 @@
+\section{Video Device}\label{sec:Device Types / Video Device}
+
+The virtio video encoder device and decoder device are virtual devices that
+supports encoding and decoding respectively. Though the encoder and the decoder
+are different devices, they use the same protocol.
+
+\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
+
+\begin{description}
+\item[30] encoder device
+\item[31] decoder device
+\end{description}
+
+\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues}
+
+\begin{description}
+\item[0] controlq - queue for sending control commands.
+\item[1] eventq - queue for sending events happened in the device.
+\end{description}
+
+\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature bits}
+
+\begin{description}
+\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for video
+  buffers.
+\end{description}
+
+\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits}
+
+The device MUST offer at least one of feature bits.
+
+\subsection{Device configuration layout}\label{sec:Device Types / Video Device / Device configuration layout}
+
+Video device configuration uses the following layout structure:
+
+\begin{lstlisting}
+struct virtio_video_config {
+        le32 max_cap_len;
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{max_cap_len}] defines the maximum length of a descriptor
+  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
+  MUST set this value.
+\end{description}
+
+\subsection{Device Initialization}\label{sec:Device Types / Video Device / Device Initialization}
+
+\devicenormative{\subsubsection}{Device Initialization}{Device Types / Video Device / Device Initialization}
+
+The driver SHOULD query device capability by using the
+VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
+setup.
+
+\subsection{Device Operation}\label{sec:Device Types / Video Device / Device Operation}
+
+The driver allocates input and output buffers and queues the buffers
+to the device. The device performs operations on the buffers according
+to the function in question.
+
+\subsubsection{Device Operation: Create stream}
+
+To process buffers, the device needs to associate them with a certain
+video stream (essentially, a context). Streams are created by
+VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
+determined by the device.
+
+\subsubsection{Device Operation: Create buffers}
+
+Buffers are used to store the actual data as well as the relevant
+metadata. Scatter lists are supported, so the buffer doesn't need to
+be contiguous in guest physical memory.
+
+\begin{itemize*}
+\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
+  resource that is backed by a buffer allocated from the driver's
+  memory.
+\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
+  is no longer needed.
+\end{itemize*}
+
+\subsubsection{Device Operation: Stream parameter control}
+
+\begin{itemize*}
+\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters for
+  input and output streams from the device.
+\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the
+  device.
+\item After setting stream parameters, the driver may issue
+  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can be
+  changed implicitly by the device during the set operation.
+\end{itemize*}
+
+\subsubsection{Device Operation: Process buffers}
+
+\begin{itemize*}
+\item If the function and the buffer type require so, write data to
+the buffer memory.
+\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
+processing in the device.
+\item The request completes asynchronously when the device has
+finished with the buffer.
+\end{itemize*}
+
+\subsubsection{Device Operation: Buffer processing control}
+
+\begin{itemize*}
+\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
+  return all of the already queued buffers.
+\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
+  already queued buffers from the input or the output queue. This also
+  includes input or output buffers that can be currently owned by the
+  device's processing pipeline.
+\end{itemize*}
+
+\subsubsection{Device Operation: Asynchronous events}
+
+While processing buffers, the device can send asynchronous event
+notifications to the driver. The behaviour depends on the exact
+stream. For example, the decoder device sends a resolution change
+event when it encounters new resolution metadata in the stream.
+
+\subsubsection{Device Operation: Request header}
+
+All requests and responses on the control virt queue have a fixed
+header using the following layout structure and definitions:
+
+\begin{lstlisting}
+enum virtio_video_ctrl_type {
+        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
+
+        /* request */
+        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
+        VIRTIO_VIDEO_T_STREAM_CREATE,
+        VIRTIO_VIDEO_T_STREAM_DESTROY,
+        VIRTIO_VIDEO_T_STREAM_DRAIN,
+        VIRTIO_VIDEO_T_RESOURCE_CREATE,
+        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
+        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
+        VIRTIO_VIDEO_T_QUEUE_CLEAR,
+        VIRTIO_VIDEO_T_SET_PARAMS,
+        VIRTIO_VIDEO_T_GET_PARAMS,
+
+        /* response */
+        VIRTIO_VIDEO_S_OK = 0x0200,
+        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
+        VIRTIO_VIDEO_S_OK_GET_PARAMS,
+
+        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
+        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
+        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
+        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
+        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
+};
+
+struct virtio_video_ctrl_hdr {
+        le32 type;
+        le32 stream_id;
+        le32 len; /* Length of the structure in bytes. */
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{type}] is the type of the driver request or the device
+response.
+\item[\field{stream_id}] specifies a target stream.
+\item[\field{len}] is the length of data in bytes, which includes
+length of the header.
+\end{description}
+
+\subsubsection{Device Operation: controlq}
+
+\begin{description}
+
+\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
+supported formats.
+
+The driver uses \field{struct virtio_video_get_capability} to send a
+query request.
+
+\begin{lstlisting}
+enum virtio_video_buf_type {
+        VIRTIO_VIDEO_BUF_TYPE_INPUT,
+        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
+};
+
+struct virtio_video_get_capability {
+        struct virtio_video_ctrl_hdr hdr;
+        enum virtio_video_buf_type buf_type;
+};
+\end{lstlisting}
+\begin{description}
+\item[\field{buf_type}] is the buffer type that the driver asks
+information about. The driver MUST set either
+\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}.
+\end{description}
+
+The device responds a capability by using \field{struct
+virtio_video_get_capability_resp}.
+\begin{lstlisting}
+enum virtio_video_format {
+        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
+        /* Raw formats */
+        VIRTIO_VIDEO_FORMAT_NV12 = 1,
+        VIRTIO_VIDEO_FORMAT_YUV420,
+        VIRTIO_VIDEO_FORMAT_YVU420,
+
+        /* Compressed formats */
+        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
+        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
+        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
+};
+
+enum virtio_video_profile {
+        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
+
+        /* H.264 */
+        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
+        VIRTIO_VIDEO_PROFILE_H264_BASELINE = VIRTIO_VIDEO_PROFILE_H264_BASELINE,
+        VIRTIO_VIDEO_PROFILE_H264_MAIN,
+        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
+        VIRTIO_VIDEO_PROFILE_H264_HIGH,
+        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
+        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
+        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
+        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
+        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
+        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
+        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
+        VIRTIO_VIDEO_PROFILE_H264_MAX = VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
+
+        /* VP8 */
+        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
+        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
+        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
+
+        /* VP9 */
+        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
+        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
+        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
+        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
+        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
+        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
+};
+
+struct virtio_video_format_range {
+        le32 min;
+        le32 max;
+        le32 step;
+        u8 paddings[4];
+};
+
+struct virtio_video_format_desc {
+        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
+        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
+        le64 mask;
+        struct virtio_video_format_range width;
+        struct virtio_video_format_range height;
+        le32 num_rates;
+        u8 padding[4];
+        /* Followed by struct virtio_video_frame_rate frame_rates[] */
+};
+
+struct virtio_video_get_capability_resp {
+        struct virtio_video_ctrl_hdr hdr;
+        le32 num_descs;
+        /* Followed by struct virtio_video_format_desc desc[] */
+};
+\end{lstlisting}
+
+The format description \field{struct virtio_video_format_desc}
+includes the following fields:
+\begin{description}
+\item[\field{format}] specifies an image format. The device MUST set one
+  of \field{enum virtio_video_format}.
+\item[\field{profile}] specifies a profile of the compressed image format
+  specified in \field{format}. The driver SHOULD ignore this value if
+  \field{format} is a raw format.
+\item[\field{mask}] is a bitset that represents the supported
+  combination of input and output format. If \textit{i}-th bit is set
+  in \field{mask} of \textit{j}-th \field{struct
+  virtio_video_format_desc} for input, the device supports encoding or
+  decoding from the \textit{j}-th input format to \textit{i}-th output
+  format.
+\item[\field{width, height}] represents a range of resolutions
+  supported by the device. If its \field{step} is not applicable, its
+  \field{min} is equal to its \field{max}.
+\item[\field{num_rates}] is the length of an array \field{frame_rates}. In case of decoder, the driver SHOULD ignore this value.
+\item[\field{frame_rates}] is an array of supported frame rates.
+\end{description}
+
+\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
+  within the device.
+
+\begin{lstlisting}
+enum virtio_video_mem_type {
+        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
+};
+
+struct virtio_video_stream_create {
+        struct virtio_video_ctrl_hdr hdr;
+        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
+        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
+        char debug_name[64];
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{in_mem_type}] is a type of buffer management for input
+buffers. The driver MUST set a value in \field{enum
+virtio_video_mem_type}.
+\item[\field{out_mem_type}] is a type of buffer management for output
+buffers. The driver MUST set a value in \field{enum
+virtio_video_mem_type}.
+\item[\field{debug_name}] is a text string for a debug purpose.
+\end{description}
+
+\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
+  within the device.
+
+\begin{lstlisting}
+struct virtio_video_stream_destroy {
+        struct virtio_video_ctrl_hdr hdr;
+};
+\end{lstlisting}
+
+\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
+  queued buffers through the pipeline.
+
+\begin{lstlisting}
+struct virtio_video_stream_drain {
+        struct virtio_video_ctrl_hdr hdr;
+};
+\end{lstlisting}
+
+\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
+  within the device.
+
+\begin{lstlisting}
+struct virtio_video_mem_entry {
+        le64 addr;
+        le32 length;
+        u8 padding[4];
+};
+
+struct virtio_video_resource_create {
+        struct virtio_video_ctrl_hdr hdr;
+        le32 resource_id;
+        le32 nr_entries;
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{resource_id}] internal id of the resource.
+\item[\field{nr_entries}] number of \field{struct
+  virtio_video_mem_entry} memory entries.
+\end{description}
+
+\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
+  within the device.
+
+\begin{lstlisting}
+struct virtio_video_resource_destroy {
+        struct virtio_video_ctrl_hdr hdr;
+        le32 resource_id;
+        u8 padding[4];
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{resource_id}] internal id of the resource.
+\end{description}
+
+\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
+queue.
+
+\begin{lstlisting}
+#define VIRTIO_VIDEO_MAX_PLANES 8
+
+struct virtio_video_resource_queue {
+        struct virtio_video_ctrl_hdr hdr;
+        le32 buf_type;
+        le32 resource_id;
+        le64 timestamp;
+        le32 nr_data_size;
+        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{buf_type}] buf_type of the .
+\item[\field{resource_id}] internal id of the resource.
+\item[\field{timestamp}] an abstract sequence counter that can be used
+  for synchronisation.
+\item[\field{nr_data_size}] number of \field{data_size} entries.
+\item[\field{data_size}] number of data bytes within a plane.
+\end{description}
+
+\begin{lstlisting}
+enum virtio_video_buffer_flag {
+        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
+        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
+        /* Encoder only */
+        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
+        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
+        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
+};
+
+struct virtio_video_resource_queue_resp {
+        struct virtio_video_ctrl_hdr hdr;
+        le64 timestamp;
+        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
+        le32 size;  /* Encoded size */
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{timestamp}] an abstract sequence counter that can be used
+  for synchronisation.
+\item[\field{flags}] mark specific buffers in the sequence.
+\item[\field{size}] data size in the buffer (encoder only).
+\end{description}
+
+The device sends a response to the queue request asynchronously when
+it has finished processing the buffer.
+
+The device SHOULD mark a buffer that triggered a processing error with
+the VIRTIO_VIDEO_BUFFER_F_ERR flag.
+
+The device MUST mark the last buffer with the
+VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
+sequence.
+
+In case of encoder, to denote a particular frame type the devie MUST
+mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
+VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
+
+\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
+  buffers back from the input or the output queue of the device. The
+  device SHOULD return all of the buffers from the respective queue as
+  soon as possible without pushing the buffers through the processing
+  pipeline.
+
+\begin{lstlisting}
+struct virtio_video_queue_clear {
+        struct virtio_video_ctrl_hdr hdr;
+        le32 buf_type;
+        u8 padding[4];
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{buf_type}] buffer type.
+\end{description}
+
+\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
+  output of a stream.
+
+\begin{lstlisting}
+struct virtio_video_plane_format {
+        le32 plane_size;
+        le32 stride;
+        u8 padding[4];
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{plane_size}] size of the plane in bytes.
+\item[\field{stride}] stride used for the plane in bytes.
+\end{description}
+
+\begin{lstlisting}
+struct virtio_video_params {
+        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
+        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
+        le32 frame_width;
+        le32 frame_height;
+        le32 min_buffers;
+        le32 max_buffers;
+        le32 frame_rate;
+        struct virtio_video_crop {
+                le32 left;
+                le32 top;
+                le32 width;
+                le32 height;
+        } crop;
+        le32 num_planes;
+        struct virtio_video_plane_format plane_formats[VIRTIO_VIDEO_MAX_PLANES];
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{frame_width}] the value to get/set.
+\item[\field{frame_height}] the value to get/set.
+\item[\field{pixel_format}] the value to get/set.
+\item[\field{min_buffers}] minimum buffers required to handle the
+  format (r/o).
+\item[\field{max_buffers}] maximum buffers required to handle the
+  format (r/o).
+\item[\field{frame_rate}] the value to get/set.
+\item[\field{crop}] cropping (composing) rectangle.
+\item[\field{num_planes}] number of planes used to store pixel data
+(r/o).
+\item[\field{plane_formats}] description of each plane.
+\end{description}
+
+\begin{lstlisting}
+struct virtio_video_get_params {
+        struct virtio_video_ctrl_hdr hdr;
+        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
+};
+
+struct virtio_video_get_params_resp {
+        struct virtio_video_ctrl_hdr hdr;
+        struct virtio_video_params params;
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{buf_type}] buffer type.
+\item[\field{params}] parameter values.
+\end{description}
+
+\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
+
+
+\begin{lstlisting}
+struct virtio_video_set_params {
+        struct virtio_video_ctrl_hdr hdr;
+        struct virtio_video_params params;
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{params}] parameters to set.
+\end{description}
+
+Setting stream parameters might have side effects within the device.
+For example, the device MAY perform alignment of width and height,
+change the number of planes it uses for the format, or do whatever
+changes that are required to continue normal operation using the
+updated parameters. It is up to the driver to check the parameter set
+after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
+
+\end{description}
+
+\subsubsection{Device Operation: eventq}
+
+The device can report events on the event queue. The driver initially
+populates the queue with device-writeable buffers. When the device
+needs to report an event, it fills a buffer and notifies the driver.
+The driver consumes the report and adds a new buffer to the virtqueue.
+
+\begin{lstlisting}
+enum virtio_video_event_type {
+        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
+        /* For all functions */
+        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
+        /* For decoder only */
+        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
+};
+
+struct virtio_video_event {
+        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
+        le32 stream_id;
+        u8 padding[4];
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{event_type}] type of the triggered event .
+\item[\field{stream_id}] id of the source stream.
+\end{description}
+
+The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
+whenever it encounters new resolution data in the stream. This
+includes the case of the initial device configuration after metadata
+has been parsed and the case of dynamic resolution change.
-- 
2.24.1.735.g03f4e72817-goog


^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-18 13:02 ` [PATCH v2 1/1] virtio-video: Add virtio " Keiichi Watanabe
@ 2019-12-18 13:40   ` Gerd Hoffmann
  2019-12-18 14:08     ` [virtio-dev] " Tomasz Figa
  2019-12-19  9:26     ` Dmitry Sepp
  2019-12-18 17:29   ` Frediano Ziglio
                     ` (4 subsequent siblings)
  5 siblings, 2 replies; 73+ messages in thread
From: Gerd Hoffmann @ 2019-12-18 13:40 UTC (permalink / raw)
  To: Keiichi Watanabe
  Cc: virtio-dev, linux-media, acourbot, alexlau, daniel, dgreid,
	dmitry.sepp, egranata, fziglio, hverkuil, marcheu, posciak,
	spice-devel, stevensd, tfiga, uril

  Hi,

> +The device MUST mark the last buffer with the
> +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> +sequence.

No, that would build a race condition into the protocol.  The device
could complete the last buffer after the driver has sent the drain
command but before the device saw it.  So the flag would not be
reliable.

I also can't see why the flag is needed in the first place.  The driver
should know which buffers are queued still and be able to figure
whenever the drain is complete or not without depending on that flag.
So I'd suggest to simply drop it.

That is the only issue I've spotted in the protocol on a first quick
look.  There are a few places where the spec text could be improved.
I'll try to set aside some time to go over this in detail, but I can't
promise I'll find the time to do that before xmas and new year.

cheers,
  Gerd


^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [virtio-dev] Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-18 13:40   ` Gerd Hoffmann
@ 2019-12-18 14:08     ` Tomasz Figa
  2019-12-19  7:46       ` Gerd Hoffmann
  2019-12-19  9:26     ` Dmitry Sepp
  1 sibling, 1 reply; 73+ messages in thread
From: Tomasz Figa @ 2019-12-18 14:08 UTC (permalink / raw)
  To: Gerd Hoffmann
  Cc: Keiichi Watanabe, virtio-dev, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Dmitry Sepp, Enrico Granata, fziglio, Hans Verkuil,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

On Wed, Dec 18, 2019 at 10:40 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
>
>   Hi,
>
> > +The device MUST mark the last buffer with the
> > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > +sequence.
>
> No, that would build a race condition into the protocol.  The device
> could complete the last buffer after the driver has sent the drain
> command but before the device saw it.  So the flag would not be
> reliable.
>
> I also can't see why the flag is needed in the first place.  The driver
> should know which buffers are queued still and be able to figure
> whenever the drain is complete or not without depending on that flag.
> So I'd suggest to simply drop it.

Unfortunately video decoders are not that simple. There are always
going to be some buffers on the decoder side used as reference frames.
Only the decoder knows when to release them, as it continues decoding
the stream. Moreover, with certain features of certain codecs (e.g.
framebuffer reordering in H.264), there could be decoded frames that
need to be held by decoder, because later bitstream may contain
earlier frames.

How we defined this in the V4L2 stateful decoder interface is that if
the decoder happened to return the last framebuffer before the drain
request arrived, it would return one more, empty. From our experience
that was the easiest thing to deal with from both the driver and the
application, so I believe the same should apply to the host device
implementation and the guest driver.

>
> That is the only issue I've spotted in the protocol on a first quick
> look.  There are a few places where the spec text could be improved.
> I'll try to set aside some time to go over this in detail, but I can't
> promise I'll find the time to do that before xmas and new year.

Thanks a lot for review.

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-18 13:02 ` [PATCH v2 1/1] virtio-video: Add virtio " Keiichi Watanabe
  2019-12-18 13:40   ` Gerd Hoffmann
@ 2019-12-18 17:29   ` Frediano Ziglio
  2019-12-20 14:05     ` Keiichi Watanabe
  2019-12-19 13:28   ` Dmitry Sepp
                     ` (3 subsequent siblings)
  5 siblings, 1 reply; 73+ messages in thread
From: Frediano Ziglio @ 2019-12-18 17:29 UTC (permalink / raw)
  To: Keiichi Watanabe
  Cc: virtio-dev, linux-media, acourbot, alexlau, daniel, dgreid,
	dmitry sepp, egranata, hverkuil, kraxel, marcheu, posciak,
	spice-devel, stevensd, tfiga, uril

> 
> From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> 
> The virtio video encoder device and decoder device provide functionalities to
> encode and decode video stream respectively.
> Though video encoder and decoder are provided as different devices, they use
> a
> same protocol.
> 
> Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> ---
>  content.tex      |   1 +
>  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 580 insertions(+)
>  create mode 100644 virtio-video.tex
> 
> diff --git a/content.tex b/content.tex
> index 556b373..9e56839 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> Requirements}\label{sec:Device
>  \input{virtio-vsock.tex}
>  \input{virtio-fs.tex}
>  \input{virtio-rpmb.tex}
> +\input{virtio-video.tex}
>  
>  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>  
> diff --git a/virtio-video.tex b/virtio-video.tex
> new file mode 100644
> index 0000000..30e728d
> --- /dev/null
> +++ b/virtio-video.tex
> @@ -0,0 +1,579 @@
> +\section{Video Device}\label{sec:Device Types / Video Device}
> +
> +The virtio video encoder device and decoder device are virtual devices that
> +supports encoding and decoding respectively. Though the encoder and the
> decoder
> +are different devices, they use the same protocol.
> +
> +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
> +
> +\begin{description}
> +\item[30] encoder device
> +\item[31] decoder device
> +\end{description}
> +
> +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues}
> +
> +\begin{description}
> +\item[0] controlq - queue for sending control commands.
> +\item[1] eventq - queue for sending events happened in the device.
> +\end{description}
> +
> +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature
> bits}
> +
> +\begin{description}
> +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for
> video
> +  buffers.
> +\end{description}
> +
> +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device /
> Feature bits}
> +
> +The device MUST offer at least one of feature bits.
> +
> +\subsection{Device configuration layout}\label{sec:Device Types / Video
> Device / Device configuration layout}
> +
> +Video device configuration uses the following layout structure:
> +
> +\begin{lstlisting}
> +struct virtio_video_config {
> +        le32 max_cap_len;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> +  MUST set this value.
> +\end{description}
> +
> +\subsection{Device Initialization}\label{sec:Device Types / Video Device /
> Device Initialization}
> +
> +\devicenormative{\subsubsection}{Device Initialization}{Device Types / Video
> Device / Device Initialization}
> +
> +The driver SHOULD query device capability by using the
> +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> +setup.
> +
> +\subsection{Device Operation}\label{sec:Device Types / Video Device / Device
> Operation}
> +
> +The driver allocates input and output buffers and queues the buffers
> +to the device. The device performs operations on the buffers according
> +to the function in question.
> +
> +\subsubsection{Device Operation: Create stream}
> +
> +To process buffers, the device needs to associate them with a certain
> +video stream (essentially, a context). Streams are created by
> +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> +determined by the device.
> +
> +\subsubsection{Device Operation: Create buffers}
> +
> +Buffers are used to store the actual data as well as the relevant
> +metadata. Scatter lists are supported, so the buffer doesn't need to
> +be contiguous in guest physical memory.
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> +  resource that is backed by a buffer allocated from the driver's
> +  memory.
> +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> +  is no longer needed.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Stream parameter control}
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters for
> +  input and output streams from the device.
> +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the
> +  device.
> +\item After setting stream parameters, the driver may issue
> +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can
> be
> +  changed implicitly by the device during the set operation.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Process buffers}
> +
> +\begin{itemize*}
> +\item If the function and the buffer type require so, write data to
> +the buffer memory.
> +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> +processing in the device.
> +\item The request completes asynchronously when the device has
> +finished with the buffer.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Buffer processing control}
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> +  return all of the already queued buffers.
> +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> +  already queued buffers from the input or the output queue. This also
> +  includes input or output buffers that can be currently owned by the
> +  device's processing pipeline.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Asynchronous events}
> +
> +While processing buffers, the device can send asynchronous event
> +notifications to the driver. The behaviour depends on the exact
> +stream. For example, the decoder device sends a resolution change
> +event when it encounters new resolution metadata in the stream.
> +
> +\subsubsection{Device Operation: Request header}
> +
> +All requests and responses on the control virt queue have a fixed
> +header using the following layout structure and definitions:
> +
> +\begin{lstlisting}
> +enum virtio_video_ctrl_type {
> +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> +
> +        /* request */
> +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> +        VIRTIO_VIDEO_T_STREAM_CREATE,
> +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> +        VIRTIO_VIDEO_T_SET_PARAMS,
> +        VIRTIO_VIDEO_T_GET_PARAMS,
> +
> +        /* response */
> +        VIRTIO_VIDEO_S_OK = 0x0200,
> +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> +
> +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> +};
> +
> +struct virtio_video_ctrl_hdr {
> +        le32 type;
> +        le32 stream_id;
> +        le32 len; /* Length of the structure in bytes. */
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{type}] is the type of the driver request or the device
> +response.
> +\item[\field{stream_id}] specifies a target stream.
> +\item[\field{len}] is the length of data in bytes, which includes
> +length of the header.

I suppose is implicit that it's the size of the container structure.

> +\end{description}
> +
> +\subsubsection{Device Operation: controlq}
> +
> +\begin{description}
> +
> +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> +supported formats.
> +
> +The driver uses \field{struct virtio_video_get_capability} to send a
> +query request.
> +
> +\begin{lstlisting}
> +enum virtio_video_buf_type {
> +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> +};
> +
> +struct virtio_video_get_capability {
> +        struct virtio_video_ctrl_hdr hdr;
> +        enum virtio_video_buf_type buf_type;

Here you used an enumeration but later you are using just le32 type
with a comment. Why this difference?

> +};
> +\end{lstlisting}
> +\begin{description}
> +\item[\field{buf_type}] is the buffer type that the driver asks
> +information about. The driver MUST set either
> +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}.
> +\end{description}
> +
> +The device responds a capability by using \field{struct
> +virtio_video_get_capability_resp}.
> +\begin{lstlisting}
> +enum virtio_video_format {
> +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,

Really minor: maybe you want to insert an empty line to be coherent
with other enumerations?

> +        /* Raw formats */
> +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> +        VIRTIO_VIDEO_FORMAT_YUV420,
> +        VIRTIO_VIDEO_FORMAT_YVU420,
> +
> +        /* Compressed formats */
> +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> +};
> +
> +enum virtio_video_profile {
> +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> +
> +        /* H.264 */
> +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> VIRTIO_VIDEO_PROFILE_H264_BASELINE,

Maybe you want

VIRTIO_VIDEO_PROFILE_H264_BASELINE = VIRTIO_VIDEO_PROFILE_H264_MIN,

like others?

> +        VIRTIO_VIDEO_PROFILE_H264_MAIN,
> +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> +
> +        /* VP8 */
> +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> +
> +        /* VP9 */
> +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> +        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> +};
> +
> +struct virtio_video_format_range {
> +        le32 min;
> +        le32 max;
> +        le32 step;
> +        u8 paddings[4];

Here you padded at 8-byte like many other structures but not for
virtio_video_ctrl_hdr. Is this expected?

> +};
> +
> +struct virtio_video_format_desc {
> +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> +        le64 mask;

This will limit the formats to 64 while "num_descs" below is a
32 bit.

> +        struct virtio_video_format_range width;
> +        struct virtio_video_format_range height;
> +        le32 num_rates;
> +        u8 padding[4];
> +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> +};
> +
> +struct virtio_video_get_capability_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 num_descs;
> +        /* Followed by struct virtio_video_format_desc desc[] */
> +};
> +\end{lstlisting}
> +
> +The format description \field{struct virtio_video_format_desc}
> +includes the following fields:
> +\begin{description}
> +\item[\field{format}] specifies an image format. The device MUST set one
> +  of \field{enum virtio_video_format}.
> +\item[\field{profile}] specifies a profile of the compressed image format
> +  specified in \field{format}. The driver SHOULD ignore this value if
> +  \field{format} is a raw format.

Why not documenting that MUST be VIRTIO_VIDEO_PROFILE_UNDEFINED for
raw formats so the field could be extended in the future?

> +\item[\field{mask}] is a bitset that represents the supported
> +  combination of input and output format. If \textit{i}-th bit is set
> +  in \field{mask} of \textit{j}-th \field{struct
> +  virtio_video_format_desc} for input, the device supports encoding or
> +  decoding from the \textit{j}-th input format to \textit{i}-th output
> +  format.
> +\item[\field{width, height}] represents a range of resolutions
> +  supported by the device. If its \field{step} is not applicable, its
> +  \field{min} is equal to its \field{max}.
> +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In
> case of decoder, the driver SHOULD ignore this value.
> +\item[\field{frame_rates}] is an array of supported frame rates.

I suppose here we are talking about bitrates, right? Could be confused by
FPS.

> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> +  within the device.
> +
> +\begin{lstlisting}
> +enum virtio_video_mem_type {
> +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> +};
> +
> +struct virtio_video_stream_create {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> +        char debug_name[64];
> +};

This structure has a size not multiple of 8-bytes. Not an issue, but
the same apply to other structures that instead are padded to 8-byte size.

> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{in_mem_type}] is a type of buffer management for input
> +buffers. The driver MUST set a value in \field{enum
> +virtio_video_mem_type}.
> +\item[\field{out_mem_type}] is a type of buffer management for output
> +buffers. The driver MUST set a value in \field{enum
> +virtio_video_mem_type}.
> +\item[\field{debug_name}] is a text string for a debug purpose.

Must be NUL-terminated ? UTF-8 ?

> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_stream_destroy {
> +        struct virtio_video_ctrl_hdr hdr;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> +  queued buffers through the pipeline.
> +
> +\begin{lstlisting}
> +struct virtio_video_stream_drain {
> +        struct virtio_video_ctrl_hdr hdr;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_mem_entry {
> +        le64 addr;
> +        le32 length;
> +        u8 padding[4];
> +};
> +
> +struct virtio_video_resource_create {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 resource_id;
> +        le32 nr_entries;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{resource_id}] internal id of the resource.

Is it input or output?

> +\item[\field{nr_entries}] number of \field{struct
> +  virtio_video_mem_entry} memory entries.

I suppose that the structure is followed by an array of
virtio_video_mem_entry after a padding of 4-bytes for
alignment.

> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_resource_destroy {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 resource_id;
> +        u8 padding[4];

Is this padding for future extensions?

> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{resource_id}] internal id of the resource.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> +queue.
> +
> +\begin{lstlisting}
> +#define VIRTIO_VIDEO_MAX_PLANES 8
> +
> +struct virtio_video_resource_queue {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type;
> +        le32 resource_id;
> +        le64 timestamp;

I suppose you would like this field aligned to avoid
hidden paddings.

> +        le32 nr_data_size;
> +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buf_type of the .
> +\item[\field{resource_id}] internal id of the resource.
> +\item[\field{timestamp}] an abstract sequence counter that can be used
> +  for synchronisation.
> +\item[\field{nr_data_size}] number of \field{data_size} entries.
> +\item[\field{data_size}] number of data bytes within a plane.
> +\end{description}
> +
... omissis ...
> --
> 2.24.1.735.g03f4e72817-goog
> 
> 

Frediano


^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [virtio-dev] Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-18 14:08     ` [virtio-dev] " Tomasz Figa
@ 2019-12-19  7:46       ` Gerd Hoffmann
  2019-12-19  9:48         ` Dmitry Sepp
  0 siblings, 1 reply; 73+ messages in thread
From: Gerd Hoffmann @ 2019-12-19  7:46 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: Keiichi Watanabe, virtio-dev, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Dmitry Sepp, Enrico Granata, fziglio, Hans Verkuil,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

On Wed, Dec 18, 2019 at 11:08:37PM +0900, Tomasz Figa wrote:
> On Wed, Dec 18, 2019 at 10:40 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
> >
> >   Hi,
> >
> > > +The device MUST mark the last buffer with the
> > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > +sequence.
> >
> > No, that would build a race condition into the protocol.  The device
> > could complete the last buffer after the driver has sent the drain
> > command but before the device saw it.  So the flag would not be
> > reliable.
> >
> > I also can't see why the flag is needed in the first place.  The driver
> > should know which buffers are queued still and be able to figure
> > whenever the drain is complete or not without depending on that flag.
> > So I'd suggest to simply drop it.
> 
> Unfortunately video decoders are not that simple. There are always
> going to be some buffers on the decoder side used as reference frames.
> Only the decoder knows when to release them, as it continues decoding
> the stream.

Not clearly defined in the spec:  When is the decoder supposed to send
the response for a queue request?  When it finished decoding (i.e. frame
is ready for playback), or when it doesn't need the buffer any more for
decoding (i.e. buffer can be re-queued or pages can be released)?

> How we defined this in the V4L2 stateful decoder interface is that if
> the decoder happened to return the last framebuffer before the drain
> request arrived, it would return one more, empty.

Ok.  That is not clear from the spec.  I've read the drain request as as
"please stop decoding and complete all queue requests which are in the
virtqueue, even when you didn't process them yet".  In which case
completing the last pending queue request would clearly indicate the
draining request is complete.  Also completing the drain request should
only happen when the operation is finished.

I think the various states a buffer can have and how queuing & deciding
& draining changes these states should be clarified in the
specification.

cheers,
  Gerd


^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-18 13:40   ` Gerd Hoffmann
  2019-12-18 14:08     ` [virtio-dev] " Tomasz Figa
@ 2019-12-19  9:26     ` Dmitry Sepp
  2019-12-19  9:59       ` Tomasz Figa
  2019-12-19 12:54       ` Gerd Hoffmann
  1 sibling, 2 replies; 73+ messages in thread
From: Dmitry Sepp @ 2019-12-19  9:26 UTC (permalink / raw)
  To: Gerd Hoffmann
  Cc: Keiichi Watanabe, virtio-dev, linux-media, acourbot, alexlau,
	daniel, dgreid, egranata, fziglio, hverkuil, marcheu, posciak,
	spice-devel, stevensd, tfiga, uril

Hi Gerd,

On Mittwoch, 18. Dezember 2019 14:40:37 CET Gerd Hoffmann wrote:
>   Hi,
> 
> > +The device MUST mark the last buffer with the
> > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > +sequence.
> 
> No, that would build a race condition into the protocol.  The device
> could complete the last buffer after the driver has sent the drain
> command but before the device saw it.  So the flag would not be
> reliable.
No, then it means the device was not in drain, but, for example, hit a 
resolution change in the stream and tells us that this is the last buffer with 
the old resolution.

> 
> I also can't see why the flag is needed in the first place.  The driver
> should know which buffers are queued still and be able to figure
> whenever the drain is complete or not without depending on that flag.
> So I'd suggest to simply drop it.
This flag is used not for drain only. In marks the completion of whatever 
specific buffer sequence, like a full end-of-stream, resolution change, drain 
etc. We also need this to handle nested sequences. For instance, a resolution 
change event might happen while in drain.

Regards,
Dmitry.

> 
> That is the only issue I've spotted in the protocol on a first quick
> look.  There are a few places where the spec text could be improved.
> I'll try to set aside some time to go over this in detail, but I can't
> promise I'll find the time to do that before xmas and new year.
> 
> cheers,
>   Gerd



^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [virtio-dev] Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-19  7:46       ` Gerd Hoffmann
@ 2019-12-19  9:48         ` Dmitry Sepp
  2019-12-19  9:59           ` Tomasz Figa
  2019-12-19 13:01           ` Gerd Hoffmann
  0 siblings, 2 replies; 73+ messages in thread
From: Dmitry Sepp @ 2019-12-19  9:48 UTC (permalink / raw)
  To: Gerd Hoffmann
  Cc: Tomasz Figa, Keiichi Watanabe, virtio-dev,
	Linux Media Mailing List, Alexandre Courbot, Alex Lau,
	Daniel Vetter, Dylan Reid, Enrico Granata, fziglio, Hans Verkuil,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

Hi,

On Donnerstag, 19. Dezember 2019 08:46:39 CET Gerd Hoffmann wrote:
> On Wed, Dec 18, 2019 at 11:08:37PM +0900, Tomasz Figa wrote:
> > On Wed, Dec 18, 2019 at 10:40 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
> > >   Hi,
> > >   
> > > > +The device MUST mark the last buffer with the
> > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > > +sequence.
> > > 
> > > No, that would build a race condition into the protocol.  The device
> > > could complete the last buffer after the driver has sent the drain
> > > command but before the device saw it.  So the flag would not be
> > > reliable.
> > > 
> > > I also can't see why the flag is needed in the first place.  The driver
> > > should know which buffers are queued still and be able to figure
> > > whenever the drain is complete or not without depending on that flag.
> > > So I'd suggest to simply drop it.
> > 
> > Unfortunately video decoders are not that simple. There are always
> > going to be some buffers on the decoder side used as reference frames.
> > Only the decoder knows when to release them, as it continues decoding
> > the stream.
> 
> Not clearly defined in the spec:  When is the decoder supposed to send
> the response for a queue request?  When it finished decoding (i.e. frame
> is ready for playback), or when it doesn't need the buffer any more for
> decoding (i.e. buffer can be re-queued or pages can be released)?
In my eyes the both statements mean almost the same and both are valid. I 
think whatever underlying libraries are used for decoding on the device side, 
they simply won't return us the buffer as long as the HW device needs them to 
continue its normal operation. So your first sentence applies to output buffers, 
the second - to input buffers.

My understanding is as follows: we send the response for a queue request as 
soon as the HW device on the host side passes the buffer ownership back to the 
client (like when VIDIOC_DQBUF has returned a buffer).

Thanks for reviewing!

Regards,
Dmitry.

> 
> > How we defined this in the V4L2 stateful decoder interface is that if
> > the decoder happened to return the last framebuffer before the drain
> > request arrived, it would return one more, empty.
> 
> Ok.  That is not clear from the spec.  I've read the drain request as as
> "please stop decoding and complete all queue requests which are in the
> virtqueue, even when you didn't process them yet".  In which case
> completing the last pending queue request would clearly indicate the
> draining request is complete.  Also completing the drain request should
> only happen when the operation is finished.
> 
> I think the various states a buffer can have and how queuing & deciding
> & draining changes these states should be clarified in the
> specification.
> 
> cheers,
>   Gerd



^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [virtio-dev] Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-19  9:48         ` Dmitry Sepp
@ 2019-12-19  9:59           ` Tomasz Figa
  2019-12-19 10:54             ` Dmitry Sepp
  2019-12-19 13:01           ` Gerd Hoffmann
  1 sibling, 1 reply; 73+ messages in thread
From: Tomasz Figa @ 2019-12-19  9:59 UTC (permalink / raw)
  To: Dmitry Sepp
  Cc: Gerd Hoffmann, Keiichi Watanabe, virtio-dev,
	Linux Media Mailing List, Alexandre Courbot, Alex Lau,
	Daniel Vetter, Dylan Reid, Enrico Granata, fziglio, Hans Verkuil,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

On Thu, Dec 19, 2019 at 6:48 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
>
> Hi,
>
> On Donnerstag, 19. Dezember 2019 08:46:39 CET Gerd Hoffmann wrote:
> > On Wed, Dec 18, 2019 at 11:08:37PM +0900, Tomasz Figa wrote:
> > > On Wed, Dec 18, 2019 at 10:40 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
> > > >   Hi,
> > > >
> > > > > +The device MUST mark the last buffer with the
> > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > > > +sequence.
> > > >
> > > > No, that would build a race condition into the protocol.  The device
> > > > could complete the last buffer after the driver has sent the drain
> > > > command but before the device saw it.  So the flag would not be
> > > > reliable.
> > > >
> > > > I also can't see why the flag is needed in the first place.  The driver
> > > > should know which buffers are queued still and be able to figure
> > > > whenever the drain is complete or not without depending on that flag.
> > > > So I'd suggest to simply drop it.
> > >
> > > Unfortunately video decoders are not that simple. There are always
> > > going to be some buffers on the decoder side used as reference frames.
> > > Only the decoder knows when to release them, as it continues decoding
> > > the stream.
> >
> > Not clearly defined in the spec:  When is the decoder supposed to send
> > the response for a queue request?  When it finished decoding (i.e. frame
> > is ready for playback), or when it doesn't need the buffer any more for
> > decoding (i.e. buffer can be re-queued or pages can be released)?
> In my eyes the both statements mean almost the same and both are valid. I
> think whatever underlying libraries are used for decoding on the device side,
> they simply won't return us the buffer as long as the HW device needs them to
> continue its normal operation. So your first sentence applies to output buffers,
> the second - to input buffers.
>
> My understanding is as follows: we send the response for a queue request as
> soon as the HW device on the host side passes the buffer ownership back to the
> client (like when VIDIOC_DQBUF has returned a buffer).

That's how it's defined in V4L2 and what makes the most sense from the
video decoding point of view, as one wants to display frames as soon
as they are available.

However that still doesn't let the driver know which buffers will be
dequeued when. A simple example of this scenario is when the guest is
done displaying a frame and requeues the buffer back to the decoder.
Then the decoder will not choose it for decoding next frames into as
long as the frame in that buffer is still used as a reference frame,
even if one sends the drain request.

>
> Thanks for reviewing!
>
> Regards,
> Dmitry.
>
> >
> > > How we defined this in the V4L2 stateful decoder interface is that if
> > > the decoder happened to return the last framebuffer before the drain
> > > request arrived, it would return one more, empty.
> >
> > Ok.  That is not clear from the spec.  I've read the drain request as as
> > "please stop decoding and complete all queue requests which are in the
> > virtqueue, even when you didn't process them yet".  In which case
> > completing the last pending queue request would clearly indicate the
> > draining request is complete.  Also completing the drain request should
> > only happen when the operation is finished.
> >
> > I think the various states a buffer can have and how queuing & deciding
> > & draining changes these states should be clarified in the
> > specification.
> >
> > cheers,
> >   Gerd
>
>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-19  9:26     ` Dmitry Sepp
@ 2019-12-19  9:59       ` Tomasz Figa
  2019-12-19 12:54       ` Gerd Hoffmann
  1 sibling, 0 replies; 73+ messages in thread
From: Tomasz Figa @ 2019-12-19  9:59 UTC (permalink / raw)
  To: Dmitry Sepp
  Cc: Gerd Hoffmann, Keiichi Watanabe, virtio-dev,
	Linux Media Mailing List, Alexandre Courbot, Alex Lau,
	Daniel Vetter, Dylan Reid, Enrico Granata, fziglio, Hans Verkuil,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

On Thu, Dec 19, 2019 at 6:26 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
>
> Hi Gerd,
>
> On Mittwoch, 18. Dezember 2019 14:40:37 CET Gerd Hoffmann wrote:
> >   Hi,
> >
> > > +The device MUST mark the last buffer with the
> > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > +sequence.
> >
> > No, that would build a race condition into the protocol.  The device
> > could complete the last buffer after the driver has sent the drain
> > command but before the device saw it.  So the flag would not be
> > reliable.
> No, then it means the device was not in drain, but, for example, hit a
> resolution change in the stream and tells us that this is the last buffer with
> the old resolution.
>
> >
> > I also can't see why the flag is needed in the first place.  The driver
> > should know which buffers are queued still and be able to figure
> > whenever the drain is complete or not without depending on that flag.
> > So I'd suggest to simply drop it.
> This flag is used not for drain only. In marks the completion of whatever
> specific buffer sequence, like a full end-of-stream, resolution change, drain
> etc. We also need this to handle nested sequences. For instance, a resolution
> change event might happen while in drain.

Good point. The resolution change event is in a different virtqueue,
so it's not serialized with the request completions.

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [virtio-dev] Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-19  9:59           ` Tomasz Figa
@ 2019-12-19 10:54             ` Dmitry Sepp
  2019-12-19 12:05               ` Tomasz Figa
  2019-12-19 13:12               ` Gerd Hoffmann
  0 siblings, 2 replies; 73+ messages in thread
From: Dmitry Sepp @ 2019-12-19 10:54 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: Gerd Hoffmann, Keiichi Watanabe, virtio-dev,
	Linux Media Mailing List, Alexandre Courbot, Alex Lau,
	Daniel Vetter, Dylan Reid, Enrico Granata, fziglio, Hans Verkuil,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

Hi Tomasz,

On Donnerstag, 19. Dezember 2019 10:59:02 CET Tomasz Figa wrote:
> On Thu, Dec 19, 2019 at 6:48 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> 
wrote:
> > Hi,
> > 
> > On Donnerstag, 19. Dezember 2019 08:46:39 CET Gerd Hoffmann wrote:
> > > On Wed, Dec 18, 2019 at 11:08:37PM +0900, Tomasz Figa wrote:
> > > > On Wed, Dec 18, 2019 at 10:40 PM Gerd Hoffmann <kraxel@redhat.com> 
wrote:
> > > > >   Hi,
> > > > >   
> > > > > > +The device MUST mark the last buffer with the
> > > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > > > > +sequence.
> > > > > 
> > > > > No, that would build a race condition into the protocol.  The device
> > > > > could complete the last buffer after the driver has sent the drain
> > > > > command but before the device saw it.  So the flag would not be
> > > > > reliable.
> > > > > 
> > > > > I also can't see why the flag is needed in the first place.  The
> > > > > driver
> > > > > should know which buffers are queued still and be able to figure
> > > > > whenever the drain is complete or not without depending on that
> > > > > flag.
> > > > > So I'd suggest to simply drop it.
> > > > 
> > > > Unfortunately video decoders are not that simple. There are always
> > > > going to be some buffers on the decoder side used as reference frames.
> > > > Only the decoder knows when to release them, as it continues decoding
> > > > the stream.
> > > 
> > > Not clearly defined in the spec:  When is the decoder supposed to send
> > > the response for a queue request?  When it finished decoding (i.e. frame
> > > is ready for playback), or when it doesn't need the buffer any more for
> > > decoding (i.e. buffer can be re-queued or pages can be released)?
> > 
> > In my eyes the both statements mean almost the same and both are valid. I
> > think whatever underlying libraries are used for decoding on the device
> > side, they simply won't return us the buffer as long as the HW device
> > needs them to continue its normal operation. So your first sentence
> > applies to output buffers, the second - to input buffers.
> > 
> > My understanding is as follows: we send the response for a queue request
> > as
> > soon as the HW device on the host side passes the buffer ownership back to
> > the client (like when VIDIOC_DQBUF has returned a buffer).
> 
> That's how it's defined in V4L2 and what makes the most sense from the
> video decoding point of view, as one wants to display frames as soon
> as they are available.
> 
> However that still doesn't let the driver know which buffers will be
> dequeued when. A simple example of this scenario is when the guest is
> done displaying a frame and requeues the buffer back to the decoder.
> Then the decoder will not choose it for decoding next frames into as
> long as the frame in that buffer is still used as a reference frame,
> even if one sends the drain request.
It might be that I'm getting your point wrong, but do you mean some hardware 
can mark a buffer as ready to be displayed yet still using the underlying 
memory to decode other frames? This means, if you occasionally/intentionally 
write to the buffer you mess up the whole decoding pipeline. That would be 
strange at least...

Regds,
Dmitry.

> 
> > Thanks for reviewing!
> > 
> > Regards,
> > Dmitry.
> > 
> > > > How we defined this in the V4L2 stateful decoder interface is that if
> > > > the decoder happened to return the last framebuffer before the drain
> > > > request arrived, it would return one more, empty.
> > > 
> > > Ok.  That is not clear from the spec.  I've read the drain request as as
> > > "please stop decoding and complete all queue requests which are in the
> > > virtqueue, even when you didn't process them yet".  In which case
> > > completing the last pending queue request would clearly indicate the
> > > draining request is complete.  Also completing the drain request should
> > > only happen when the operation is finished.
> > > 
> > > I think the various states a buffer can have and how queuing & deciding
> > > & draining changes these states should be clarified in the
> > > specification.
> > > 
> > > cheers,
> > > 
> > >   Gerd



^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [virtio-dev] Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-19 10:54             ` Dmitry Sepp
@ 2019-12-19 12:05               ` Tomasz Figa
  2019-12-19 13:12               ` Gerd Hoffmann
  1 sibling, 0 replies; 73+ messages in thread
From: Tomasz Figa @ 2019-12-19 12:05 UTC (permalink / raw)
  To: Dmitry Sepp
  Cc: Gerd Hoffmann, Keiichi Watanabe, virtio-dev,
	Linux Media Mailing List, Alexandre Courbot, Alex Lau,
	Daniel Vetter, Dylan Reid, Enrico Granata, fziglio, Hans Verkuil,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

On Thu, Dec 19, 2019 at 7:55 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
>
> Hi Tomasz,
>
> On Donnerstag, 19. Dezember 2019 10:59:02 CET Tomasz Figa wrote:
> > On Thu, Dec 19, 2019 at 6:48 PM Dmitry Sepp <dmitry.sepp@opensynergy.com>
> wrote:
> > > Hi,
> > >
> > > On Donnerstag, 19. Dezember 2019 08:46:39 CET Gerd Hoffmann wrote:
> > > > On Wed, Dec 18, 2019 at 11:08:37PM +0900, Tomasz Figa wrote:
> > > > > On Wed, Dec 18, 2019 at 10:40 PM Gerd Hoffmann <kraxel@redhat.com>
> wrote:
> > > > > >   Hi,
> > > > > >
> > > > > > > +The device MUST mark the last buffer with the
> > > > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > > > > > +sequence.
> > > > > >
> > > > > > No, that would build a race condition into the protocol.  The device
> > > > > > could complete the last buffer after the driver has sent the drain
> > > > > > command but before the device saw it.  So the flag would not be
> > > > > > reliable.
> > > > > >
> > > > > > I also can't see why the flag is needed in the first place.  The
> > > > > > driver
> > > > > > should know which buffers are queued still and be able to figure
> > > > > > whenever the drain is complete or not without depending on that
> > > > > > flag.
> > > > > > So I'd suggest to simply drop it.
> > > > >
> > > > > Unfortunately video decoders are not that simple. There are always
> > > > > going to be some buffers on the decoder side used as reference frames.
> > > > > Only the decoder knows when to release them, as it continues decoding
> > > > > the stream.
> > > >
> > > > Not clearly defined in the spec:  When is the decoder supposed to send
> > > > the response for a queue request?  When it finished decoding (i.e. frame
> > > > is ready for playback), or when it doesn't need the buffer any more for
> > > > decoding (i.e. buffer can be re-queued or pages can be released)?
> > >
> > > In my eyes the both statements mean almost the same and both are valid. I
> > > think whatever underlying libraries are used for decoding on the device
> > > side, they simply won't return us the buffer as long as the HW device
> > > needs them to continue its normal operation. So your first sentence
> > > applies to output buffers, the second - to input buffers.
> > >
> > > My understanding is as follows: we send the response for a queue request
> > > as
> > > soon as the HW device on the host side passes the buffer ownership back to
> > > the client (like when VIDIOC_DQBUF has returned a buffer).
> >
> > That's how it's defined in V4L2 and what makes the most sense from the
> > video decoding point of view, as one wants to display frames as soon
> > as they are available.
> >
> > However that still doesn't let the driver know which buffers will be
> > dequeued when. A simple example of this scenario is when the guest is
> > done displaying a frame and requeues the buffer back to the decoder.
> > Then the decoder will not choose it for decoding next frames into as
> > long as the frame in that buffer is still used as a reference frame,
> > even if one sends the drain request.
> It might be that I'm getting your point wrong, but do you mean some hardware
> can mark a buffer as ready to be displayed yet still using the underlying
> memory to decode other frames? This means, if you occasionally/intentionally
> write to the buffer you mess up the whole decoding pipeline. That would be
> strange at least...

That's correct. It depends on the hardware, but in principle we don't
want to copy the frames decoded to temporary buffers for using them as
reference frames, as that would waste bandwidth and increase latency.
The contract between the kernel and the application is that it must
not write to the frame buffers if it wants to get correct decoding
results. But after all, I don't see a reason why the application would
write to those buffers.

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-19  9:26     ` Dmitry Sepp
  2019-12-19  9:59       ` Tomasz Figa
@ 2019-12-19 12:54       ` Gerd Hoffmann
  1 sibling, 0 replies; 73+ messages in thread
From: Gerd Hoffmann @ 2019-12-19 12:54 UTC (permalink / raw)
  To: Dmitry Sepp
  Cc: Keiichi Watanabe, virtio-dev, linux-media, acourbot, alexlau,
	daniel, dgreid, egranata, fziglio, hverkuil, marcheu, posciak,
	spice-devel, stevensd, tfiga, uril

  Hi,

> > I also can't see why the flag is needed in the first place.  The driver
> > should know which buffers are queued still and be able to figure
> > whenever the drain is complete or not without depending on that flag.
> > So I'd suggest to simply drop it.
> This flag is used not for drain only. In marks the completion of whatever 
> specific buffer sequence, like a full end-of-stream, resolution change, drain 
> etc. We also need this to handle nested sequences. For instance, a resolution 
> change event might happen while in drain.

Ah, ok.  That makes sense (please clarify this in the spec).

cheers,
  Gerd


^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [virtio-dev] Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-19  9:48         ` Dmitry Sepp
  2019-12-19  9:59           ` Tomasz Figa
@ 2019-12-19 13:01           ` Gerd Hoffmann
  2020-01-08 13:50             ` Keiichi Watanabe
  1 sibling, 1 reply; 73+ messages in thread
From: Gerd Hoffmann @ 2019-12-19 13:01 UTC (permalink / raw)
  To: Dmitry Sepp
  Cc: Tomasz Figa, Keiichi Watanabe, virtio-dev,
	Linux Media Mailing List, Alexandre Courbot, Alex Lau,
	Daniel Vetter, Dylan Reid, Enrico Granata, fziglio, Hans Verkuil,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

  Hi,

> > Not clearly defined in the spec:  When is the decoder supposed to send
> > the response for a queue request?  When it finished decoding (i.e. frame
> > is ready for playback), or when it doesn't need the buffer any more for
> > decoding (i.e. buffer can be re-queued or pages can be released)?
> In my eyes the both statements mean almost the same and both are valid.

Well, no.  When the device decoded a P-Frame it can notify the device,
saying "here is your decoded frame".  But the device might still need
the buffer with the decoded frame to properly decode the following B/I
Frames which reference the P-Frame.

cheers,
  Gerd


^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [virtio-dev] Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-19 10:54             ` Dmitry Sepp
  2019-12-19 12:05               ` Tomasz Figa
@ 2019-12-19 13:12               ` Gerd Hoffmann
  2020-01-08 13:52                 ` Keiichi Watanabe
  1 sibling, 1 reply; 73+ messages in thread
From: Gerd Hoffmann @ 2019-12-19 13:12 UTC (permalink / raw)
  To: Dmitry Sepp
  Cc: Tomasz Figa, Keiichi Watanabe, virtio-dev,
	Linux Media Mailing List, Alexandre Courbot, Alex Lau,
	Daniel Vetter, Dylan Reid, Enrico Granata, fziglio, Hans Verkuil,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

  Hi,

> > However that still doesn't let the driver know which buffers will be
> > dequeued when. A simple example of this scenario is when the guest is
> > done displaying a frame and requeues the buffer back to the decoder.
> > Then the decoder will not choose it for decoding next frames into as
> > long as the frame in that buffer is still used as a reference frame,
> > even if one sends the drain request.
> It might be that I'm getting your point wrong, but do you mean some hardware 
> can mark a buffer as ready to be displayed yet still using the underlying 
> memory to decode other frames?

Yes, this is how I understand Tomasz Figa.

> This means, if you occasionally/intentionally 
> write to the buffer you mess up the whole decoding pipeline.

And to avoid this the buffer handling aspect must be clarified in the
specification.  Is the device allowed to continue using the buffer after
finishing decoding and completing the queue request?  If so, how do we
hand over buffer ownership back to the driver so it can free the pages?
drain request?  How do we handle re-using buffers?  Can the driver
simply re-queue them and expect the device figures by itself whenever it
can use the buffer or whenever it is still needed as reference frame?

cheers,
  Gerd


^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-18 13:02 ` [PATCH v2 1/1] virtio-video: Add virtio " Keiichi Watanabe
  2019-12-18 13:40   ` Gerd Hoffmann
  2019-12-18 17:29   ` Frediano Ziglio
@ 2019-12-19 13:28   ` Dmitry Sepp
  2019-12-20 15:26     ` Keiichi Watanabe
  2019-12-30 12:16     ` [virtio-dev] " Dmitry Sepp
  2020-01-03 15:47   ` Dmitry Sepp
                     ` (2 subsequent siblings)
  5 siblings, 2 replies; 73+ messages in thread
From: Dmitry Sepp @ 2019-12-19 13:28 UTC (permalink / raw)
  To: Keiichi Watanabe
  Cc: virtio-dev, linux-media, acourbot, alexlau, daniel, dgreid,
	egranata, fziglio, hverkuil, kraxel, marcheu, posciak,
	spice-devel, stevensd, tfiga, uril

Hi Keiichi,

Thank you for the update. Please see some comments below.

Also, we need to bring the virtio_video_control back as it is in fact used by 
the driver to enumerate supported encoder controls. But yes, it still needs to 
be documemnted, it's true.

On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> 
> The virtio video encoder device and decoder device provide functionalities
> to encode and decode video stream respectively.
> Though video encoder and decoder are provided as different devices, they use
> a same protocol.
> 
> Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> ---
>  content.tex      |   1 +
>  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 580 insertions(+)
>  create mode 100644 virtio-video.tex
> 
> diff --git a/content.tex b/content.tex
> index 556b373..9e56839 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> Requirements}\label{sec:Device \input{virtio-vsock.tex}
>  \input{virtio-fs.tex}
>  \input{virtio-rpmb.tex}
> +\input{virtio-video.tex}
> 
>  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> 
> diff --git a/virtio-video.tex b/virtio-video.tex
> new file mode 100644
> index 0000000..30e728d
> --- /dev/null
> +++ b/virtio-video.tex
> @@ -0,0 +1,579 @@
> +\section{Video Device}\label{sec:Device Types / Video Device}
> +
> +The virtio video encoder device and decoder device are virtual devices that
> +supports encoding and decoding respectively. Though the encoder and the
> decoder +are different devices, they use the same protocol.
> +
> +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
> +
> +\begin{description}
> +\item[30] encoder device
> +\item[31] decoder device
> +\end{description}
> +
> +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues}
> +
> +\begin{description}
> +\item[0] controlq - queue for sending control commands.
> +\item[1] eventq - queue for sending events happened in the device.
> +\end{description}
> +
> +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature
> bits} +
> +\begin{description}
> +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for
> video +  buffers.
> +\end{description}
> +
> +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device
> / Feature bits} +
> +The device MUST offer at least one of feature bits.
> +
> +\subsection{Device configuration layout}\label{sec:Device Types / Video
> Device / Device configuration layout} +
> +Video device configuration uses the following layout structure:
> +
> +\begin{lstlisting}
> +struct virtio_video_config {
> +        le32 max_cap_len;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> +  MUST set this value.
> +\end{description}
> +
> +\subsection{Device Initialization}\label{sec:Device Types / Video Device /
> Device Initialization} +
> +\devicenormative{\subsubsection}{Device Initialization}{Device Types /
> Video Device / Device Initialization} +
> +The driver SHOULD query device capability by using the
> +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> +setup.
> +
> +\subsection{Device Operation}\label{sec:Device Types / Video Device /
> Device Operation} +
> +The driver allocates input and output buffers and queues the buffers
> +to the device. The device performs operations on the buffers according
> +to the function in question.
> +
> +\subsubsection{Device Operation: Create stream}
> +
> +To process buffers, the device needs to associate them with a certain
> +video stream (essentially, a context). Streams are created by
> +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> +determined by the device.
> +
> +\subsubsection{Device Operation: Create buffers}
> +
> +Buffers are used to store the actual data as well as the relevant
> +metadata. Scatter lists are supported, so the buffer doesn't need to
> +be contiguous in guest physical memory.
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> +  resource that is backed by a buffer allocated from the driver's
> +  memory.
> +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> +  is no longer needed.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Stream parameter control}
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters
> for +  input and output streams from the device.
> +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the
> +  device.
> +\item After setting stream parameters, the driver may issue
> +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can
> be +  changed implicitly by the device during the set operation.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Process buffers}
> +
> +\begin{itemize*}
> +\item If the function and the buffer type require so, write data to
> +the buffer memory.
> +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> +processing in the device.
> +\item The request completes asynchronously when the device has
> +finished with the buffer.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Buffer processing control}
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> +  return all of the already queued buffers.
> +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> +  already queued buffers from the input or the output queue. This also
> +  includes input or output buffers that can be currently owned by the
> +  device's processing pipeline.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Asynchronous events}
> +
> +While processing buffers, the device can send asynchronous event
> +notifications to the driver. The behaviour depends on the exact
> +stream. For example, the decoder device sends a resolution change
> +event when it encounters new resolution metadata in the stream.
> +
> +\subsubsection{Device Operation: Request header}
> +
> +All requests and responses on the control virt queue have a fixed
> +header using the following layout structure and definitions:
> +
> +\begin{lstlisting}
> +enum virtio_video_ctrl_type {
> +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> +
> +        /* request */
> +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> +        VIRTIO_VIDEO_T_STREAM_CREATE,
> +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> +        VIRTIO_VIDEO_T_SET_PARAMS,
> +        VIRTIO_VIDEO_T_GET_PARAMS,
> +
> +        /* response */
> +        VIRTIO_VIDEO_S_OK = 0x0200,
> +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> +
> +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> +};
> +
> +struct virtio_video_ctrl_hdr {
> +        le32 type;
> +        le32 stream_id;
> +        le32 len; /* Length of the structure in bytes. */
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{type}] is the type of the driver request or the device
> +response.
> +\item[\field{stream_id}] specifies a target stream.
> +\item[\field{len}] is the length of data in bytes, which includes
> +length of the header.
> +\end{description}
> +
> +\subsubsection{Device Operation: controlq}
> +
> +\begin{description}
> +
> +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> +supported formats.
> +
> +The driver uses \field{struct virtio_video_get_capability} to send a
> +query request.
> +
> +\begin{lstlisting}
> +enum virtio_video_buf_type {
> +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> +};
I personally didn't like the previous term: pin_type. But, to be honest, I 
don't like the buf_type neither. Consider the GET/SET_PARAMS request: buf_type 
there looks a bit unnatural. We are trying to get stream parameters there, not 
some parameters of whatever buffer. Also I don't see any strict reason to mimic 
v4l2 naming scheme.

I'd better rename it to PORT_TYPE or QUEUE_TYPE.

> +
> +struct virtio_video_get_capability {
> +        struct virtio_video_ctrl_hdr hdr;
> +        enum virtio_video_buf_type buf_type;
> +};
> +\end{lstlisting}
> +\begin{description}
> +\item[\field{buf_type}] is the buffer type that the driver asks
> +information about. The driver MUST set either
> +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> +
> +The device responds a capability by using \field{struct
> +virtio_video_get_capability_resp}.
> +\begin{lstlisting}
> +enum virtio_video_format {
> +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> +        /* Raw formats */
> +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> +        VIRTIO_VIDEO_FORMAT_YUV420,
> +        VIRTIO_VIDEO_FORMAT_YVU420,
Let's add some variants of RGB, like RGBA, ARGB. We need it for the encoder in 
particular .

> +
> +        /* Compressed formats */
> +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
Let's add H265, MPEG4, MPEG2. We already support and use them.

Regards,
Dmitry.

> +};
> +
> +enum virtio_video_profile {
> +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> +
> +        /* H.264 */
> +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> VIRTIO_VIDEO_PROFILE_H264_BASELINE, +       
> VIRTIO_VIDEO_PROFILE_H264_MAIN,
> +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> +        /* VP8 */
> +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> +
> +        /* VP9 */
> +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> +        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> +};
> +
> +struct virtio_video_format_range {
> +        le32 min;
> +        le32 max;
> +        le32 step;
> +        u8 paddings[4];
> +};
> +
> +struct virtio_video_format_desc {
> +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> +        le64 mask;
> +        struct virtio_video_format_range width;
> +        struct virtio_video_format_range height;
> +        le32 num_rates;
> +        u8 padding[4];
> +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> +};
> +
> +struct virtio_video_get_capability_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 num_descs;
> +        /* Followed by struct virtio_video_format_desc desc[] */
> +};
> +\end{lstlisting}
> +
> +The format description \field{struct virtio_video_format_desc}
> +includes the following fields:
> +\begin{description}
> +\item[\field{format}] specifies an image format. The device MUST set one
> +  of \field{enum virtio_video_format}.
> +\item[\field{profile}] specifies a profile of the compressed image format
> +  specified in \field{format}. The driver SHOULD ignore this value if
> +  \field{format} is a raw format.
> +\item[\field{mask}] is a bitset that represents the supported
> +  combination of input and output format. If \textit{i}-th bit is set
> +  in \field{mask} of \textit{j}-th \field{struct
> +  virtio_video_format_desc} for input, the device supports encoding or
> +  decoding from the \textit{j}-th input format to \textit{i}-th output
> +  format.
> +\item[\field{width, height}] represents a range of resolutions
> +  supported by the device. If its \field{step} is not applicable, its
> +  \field{min} is equal to its \field{max}.
> +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In
> case of decoder, the driver SHOULD ignore this value.
> +\item[\field{frame_rates}] is an array of supported frame rates.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> +  within the device.
> +
> +\begin{lstlisting}
> +enum virtio_video_mem_type {
> +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> +};
> +
> +struct virtio_video_stream_create {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> +        char debug_name[64];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{in_mem_type}] is a type of buffer management for input
> +buffers. The driver MUST set a value in \field{enum
> +virtio_video_mem_type}.
> +\item[\field{out_mem_type}] is a type of buffer management for output
> +buffers. The driver MUST set a value in \field{enum
> +virtio_video_mem_type}.
> +\item[\field{debug_name}] is a text string for a debug purpose.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_stream_destroy {
> +        struct virtio_video_ctrl_hdr hdr;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> +  queued buffers through the pipeline.
> +
> +\begin{lstlisting}
> +struct virtio_video_stream_drain {
> +        struct virtio_video_ctrl_hdr hdr;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_mem_entry {
> +        le64 addr;
> +        le32 length;
> +        u8 padding[4];
> +};
> +
> +struct virtio_video_resource_create {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 resource_id;
> +        le32 nr_entries;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{resource_id}] internal id of the resource.
> +\item[\field{nr_entries}] number of \field{struct
> +  virtio_video_mem_entry} memory entries.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_resource_destroy {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 resource_id;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{resource_id}] internal id of the resource.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> +queue.
> +
> +\begin{lstlisting}
> +#define VIRTIO_VIDEO_MAX_PLANES 8
> +
> +struct virtio_video_resource_queue {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type;
> +        le32 resource_id;
> +        le64 timestamp;
> +        le32 nr_data_size;
> +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buf_type of the .
> +\item[\field{resource_id}] internal id of the resource.
> +\item[\field{timestamp}] an abstract sequence counter that can be used
> +  for synchronisation.
> +\item[\field{nr_data_size}] number of \field{data_size} entries.
> +\item[\field{data_size}] number of data bytes within a plane.
> +\end{description}
> +
> +\begin{lstlisting}
> +enum virtio_video_buffer_flag {
> +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> +        /* Encoder only */
> +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> +};
> +
> +struct virtio_video_resource_queue_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le64 timestamp;
> +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> +        le32 size;  /* Encoded size */
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{timestamp}] an abstract sequence counter that can be used
> +  for synchronisation.
> +\item[\field{flags}] mark specific buffers in the sequence.
> +\item[\field{size}] data size in the buffer (encoder only).
> +\end{description}
> +
> +The device sends a response to the queue request asynchronously when
> +it has finished processing the buffer.
> +
> +The device SHOULD mark a buffer that triggered a processing error with
> +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> +
> +The device MUST mark the last buffer with the
> +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> +sequence.
> +
> +In case of encoder, to denote a particular frame type the devie MUST
> +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> +  buffers back from the input or the output queue of the device. The
> +  device SHOULD return all of the buffers from the respective queue as
> +  soon as possible without pushing the buffers through the processing
> +  pipeline.
> +
> +\begin{lstlisting}
> +struct virtio_video_queue_clear {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buffer type.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> +  output of a stream.
> +
> +\begin{lstlisting}
> +struct virtio_video_plane_format {
> +        le32 plane_size;
> +        le32 stride;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{plane_size}] size of the plane in bytes.
> +\item[\field{stride}] stride used for the plane in bytes.
> +\end{description}
> +
> +\begin{lstlisting}
> +struct virtio_video_params {
> +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> +        le32 frame_width;
> +        le32 frame_height;
> +        le32 min_buffers;
> +        le32 max_buffers;
> +        le32 frame_rate;
> +        struct virtio_video_crop {
> +                le32 left;
> +                le32 top;
> +                le32 width;
> +                le32 height;
> +        } crop;
> +        le32 num_planes;
> +        struct virtio_video_plane_format
> plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{frame_width}] the value to get/set.
> +\item[\field{frame_height}] the value to get/set.
> +\item[\field{pixel_format}] the value to get/set.
> +\item[\field{min_buffers}] minimum buffers required to handle the
> +  format (r/o).
> +\item[\field{max_buffers}] maximum buffers required to handle the
> +  format (r/o).
> +\item[\field{frame_rate}] the value to get/set.
> +\item[\field{crop}] cropping (composing) rectangle.
> +\item[\field{num_planes}] number of planes used to store pixel data
> +(r/o).
> +\item[\field{plane_formats}] description of each plane.
> +\end{description}
> +
> +\begin{lstlisting}
> +struct virtio_video_get_params {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> +};
> +
> +struct virtio_video_get_params_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        struct virtio_video_params params;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buffer type.
> +\item[\field{params}] parameter values.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> +
> +
> +\begin{lstlisting}
> +struct virtio_video_set_params {
> +        struct virtio_video_ctrl_hdr hdr;
> +        struct virtio_video_params params;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{params}] parameters to set.
> +\end{description}
> +
> +Setting stream parameters might have side effects within the device.
> +For example, the device MAY perform alignment of width and height,
> +change the number of planes it uses for the format, or do whatever
> +changes that are required to continue normal operation using the
> +updated parameters. It is up to the driver to check the parameter set
> +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> +
> +\end{description}
> +
> +\subsubsection{Device Operation: eventq}
> +
> +The device can report events on the event queue. The driver initially
> +populates the queue with device-writeable buffers. When the device
> +needs to report an event, it fills a buffer and notifies the driver.
> +The driver consumes the report and adds a new buffer to the virtqueue.
> +
> +\begin{lstlisting}
> +enum virtio_video_event_type {
> +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> +        /* For all functions */
> +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> +        /* For decoder only */
> +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> +};
> +
> +struct virtio_video_event {
> +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> +        le32 stream_id;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{event_type}] type of the triggered event .
> +\item[\field{stream_id}] id of the source stream.
> +\end{description}
> +
> +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> +whenever it encounters new resolution data in the stream. This
> +includes the case of the initial device configuration after metadata
> +has been parsed and the case of dynamic resolution change.



^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-18 17:29   ` Frediano Ziglio
@ 2019-12-20 14:05     ` Keiichi Watanabe
  2019-12-20 15:33       ` Dmitry Sepp
  0 siblings, 1 reply; 73+ messages in thread
From: Keiichi Watanabe @ 2019-12-20 14:05 UTC (permalink / raw)
  To: Frediano Ziglio
  Cc: virtio-dev, Linux Media Mailing List, Alexandre Courbot,
	Alex Lau, Daniel Vetter, Dylan Reid, dmitry sepp, Enrico Granata,
	Hans Verkuil, Gerd Hoffmann, Stéphane Marchesin,
	Pawel Osciak, spice-devel, David Stevens, Tomasz Figa, uril

Hi Frediano,
Thanks for reviewing!

On Thu, Dec 19, 2019 at 2:29 AM Frediano Ziglio <fziglio@redhat.com> wrote:
>
> >
> > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> >
> > The virtio video encoder device and decoder device provide functionalities to
> > encode and decode video stream respectively.
> > Though video encoder and decoder are provided as different devices, they use
> > a
> > same protocol.
> >
> > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > ---
> >  content.tex      |   1 +
> >  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 580 insertions(+)
> >  create mode 100644 virtio-video.tex
> >
> > diff --git a/content.tex b/content.tex
> > index 556b373..9e56839 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > Requirements}\label{sec:Device
> >  \input{virtio-vsock.tex}
> >  \input{virtio-fs.tex}
> >  \input{virtio-rpmb.tex}
> > +\input{virtio-video.tex}
> >
> >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> >
> > diff --git a/virtio-video.tex b/virtio-video.tex
> > new file mode 100644
> > index 0000000..30e728d
> > --- /dev/null
> > +++ b/virtio-video.tex
> > @@ -0,0 +1,579 @@
> > +\section{Video Device}\label{sec:Device Types / Video Device}
> > +
> > +The virtio video encoder device and decoder device are virtual devices that
> > +supports encoding and decoding respectively. Though the encoder and the
> > decoder
> > +are different devices, they use the same protocol.
> > +
> > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
> > +
> > +\begin{description}
> > +\item[30] encoder device
> > +\item[31] decoder device
> > +\end{description}
> > +
> > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues}
> > +
> > +\begin{description}
> > +\item[0] controlq - queue for sending control commands.
> > +\item[1] eventq - queue for sending events happened in the device.
> > +\end{description}
> > +
> > +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature
> > bits}
> > +
> > +\begin{description}
> > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for
> > video
> > +  buffers.
> > +\end{description}
> > +
> > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device /
> > Feature bits}
> > +
> > +The device MUST offer at least one of feature bits.
> > +
> > +\subsection{Device configuration layout}\label{sec:Device Types / Video
> > Device / Device configuration layout}
> > +
> > +Video device configuration uses the following layout structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_config {
> > +        le32 max_cap_len;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > +  MUST set this value.
> > +\end{description}
> > +
> > +\subsection{Device Initialization}\label{sec:Device Types / Video Device /
> > Device Initialization}
> > +
> > +\devicenormative{\subsubsection}{Device Initialization}{Device Types / Video
> > Device / Device Initialization}
> > +
> > +The driver SHOULD query device capability by using the
> > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> > +setup.
> > +
> > +\subsection{Device Operation}\label{sec:Device Types / Video Device / Device
> > Operation}
> > +
> > +The driver allocates input and output buffers and queues the buffers
> > +to the device. The device performs operations on the buffers according
> > +to the function in question.
> > +
> > +\subsubsection{Device Operation: Create stream}
> > +
> > +To process buffers, the device needs to associate them with a certain
> > +video stream (essentially, a context). Streams are created by
> > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > +determined by the device.
> > +
> > +\subsubsection{Device Operation: Create buffers}
> > +
> > +Buffers are used to store the actual data as well as the relevant
> > +metadata. Scatter lists are supported, so the buffer doesn't need to
> > +be contiguous in guest physical memory.
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > +  resource that is backed by a buffer allocated from the driver's
> > +  memory.
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> > +  is no longer needed.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Stream parameter control}
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters for
> > +  input and output streams from the device.
> > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the
> > +  device.
> > +\item After setting stream parameters, the driver may issue
> > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can
> > be
> > +  changed implicitly by the device during the set operation.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Process buffers}
> > +
> > +\begin{itemize*}
> > +\item If the function and the buffer type require so, write data to
> > +the buffer memory.
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > +processing in the device.
> > +\item The request completes asynchronously when the device has
> > +finished with the buffer.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Buffer processing control}
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> > +  return all of the already queued buffers.
> > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> > +  already queued buffers from the input or the output queue. This also
> > +  includes input or output buffers that can be currently owned by the
> > +  device's processing pipeline.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Asynchronous events}
> > +
> > +While processing buffers, the device can send asynchronous event
> > +notifications to the driver. The behaviour depends on the exact
> > +stream. For example, the decoder device sends a resolution change
> > +event when it encounters new resolution metadata in the stream.
> > +
> > +\subsubsection{Device Operation: Request header}
> > +
> > +All requests and responses on the control virt queue have a fixed
> > +header using the following layout structure and definitions:
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_ctrl_type {
> > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > +
> > +        /* request */
> > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > +
> > +        /* response */
> > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > +
> > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > +};
> > +
> > +struct virtio_video_ctrl_hdr {
> > +        le32 type;
> > +        le32 stream_id;
> > +        le32 len; /* Length of the structure in bytes. */
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{type}] is the type of the driver request or the device
> > +response.
> > +\item[\field{stream_id}] specifies a target stream.
> > +\item[\field{len}] is the length of data in bytes, which includes
> > +length of the header.
>
> I suppose is implicit that it's the size of the container structure.
>

You're right. All struct except virtio_video_get_capability_resp have
fixed size and virtio_video_get_capability_resp has num_descs. So, no
need to have len.
Let me remove this in the next iteration.


> > +\end{description}
> > +
> > +\subsubsection{Device Operation: controlq}
> > +
> > +\begin{description}
> > +
> > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > +supported formats.
> > +
> > +The driver uses \field{struct virtio_video_get_capability} to send a
> > +query request.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_buf_type {
> > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > +};
> > +
> > +struct virtio_video_get_capability {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        enum virtio_video_buf_type buf_type;
>
> Here you used an enumeration but later you are using just le32 type
> with a comment. Why this difference?

No difference. I will change this to use le32 since virtio-gpu's spec does so.

>
> > +};
> > +\end{lstlisting}
> > +\begin{description}
> > +\item[\field{buf_type}] is the buffer type that the driver asks
> > +information about. The driver MUST set either
> > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}.
> > +\end{description}
> > +
> > +The device responds a capability by using \field{struct
> > +virtio_video_get_capability_resp}.
> > +\begin{lstlisting}
> > +enum virtio_video_format {
> > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
>
> Really minor: maybe you want to insert an empty line to be coherent
> with other enumerations?

Yes. Thanks.

>
> > +        /* Raw formats */
> > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > +        VIRTIO_VIDEO_FORMAT_YVU420,
> > +
> > +        /* Compressed formats */
> > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> > +};
> > +
> > +enum virtio_video_profile {
> > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > +
> > +        /* H.264 */
> > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > VIRTIO_VIDEO_PROFILE_H264_BASELINE,
>
> Maybe you want
>
> VIRTIO_VIDEO_PROFILE_H264_BASELINE = VIRTIO_VIDEO_PROFILE_H264_MIN,
>
> like others?

Thanks for catching this. I think we can find this kind of errors once
we update the driver implementation.

>
> > +        VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > +
> > +        /* VP8 */
> > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > +
> > +        /* VP9 */
> > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > +        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > +};
> > +
> > +struct virtio_video_format_range {
> > +        le32 min;
> > +        le32 max;
> > +        le32 step;
> > +        u8 paddings[4];
>
> Here you padded at 8-byte like many other structures but not for
> virtio_video_ctrl_hdr. Is this expected?

I padded here for 64-bits alignment. So, I should have padded for
ctrl_hdr as you said. But, no need anymore as we will remove |len|
from the struct as you suggested above:)

>
> > +};
> > +
> > +struct virtio_video_format_desc {
> > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > +        le64 mask;
>
> This will limit the formats to 64 while "num_descs" below is a
> 32 bit.

It's my intention. I will explicitly write that "num_desc" must not exceed 64.
I guess 64 is big enough as a number of supported input/output format
and couldn't find a better idea to represent this bipartite graph. If
you have another idea, please let me hear.

>
> > +        struct virtio_video_format_range width;
> > +        struct virtio_video_format_range height;
> > +        le32 num_rates;
> > +        u8 padding[4];
> > +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> > +};
> > +
> > +struct virtio_video_get_capability_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 num_descs;
> > +        /* Followed by struct virtio_video_format_desc desc[] */
> > +};
> > +\end{lstlisting}
> > +
> > +The format description \field{struct virtio_video_format_desc}
> > +includes the following fields:
> > +\begin{description}
> > +\item[\field{format}] specifies an image format. The device MUST set one
> > +  of \field{enum virtio_video_format}.
> > +\item[\field{profile}] specifies a profile of the compressed image format
> > +  specified in \field{format}. The driver SHOULD ignore this value if
> > +  \field{format} is a raw format.
>
> Why not documenting that MUST be VIRTIO_VIDEO_PROFILE_UNDEFINED for
> raw formats so the field could be extended in the future?

Sounds good. Will do.

>
> > +\item[\field{mask}] is a bitset that represents the supported
> > +  combination of input and output format. If \textit{i}-th bit is set
> > +  in \field{mask} of \textit{j}-th \field{struct
> > +  virtio_video_format_desc} for input, the device supports encoding or
> > +  decoding from the \textit{j}-th input format to \textit{i}-th output
> > +  format.
> > +\item[\field{width, height}] represents a range of resolutions
> > +  supported by the device. If its \field{step} is not applicable, its
> > +  \field{min} is equal to its \field{max}.
> > +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In
> > case of decoder, the driver SHOULD ignore this value.
> > +\item[\field{frame_rates}] is an array of supported frame rates.
>
> I suppose here we are talking about bitrates, right? Could be confused by
> FPS.

Nope. This is for frame rates or frame intervals. In V4L2 driver, the
values will be used for VIDIOC_ENUM_FRAMEINTERVALS ioctl.
(Dmitry, please correct me if I am wrong)

>
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_mem_type {
> > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > +};
> > +
> > +struct virtio_video_stream_create {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > +        char debug_name[64];
> > +};
>
> This structure has a size not multiple of 8-bytes. Not an issue, but
> the same apply to other structures that instead are padded to 8-byte size.

It's my fault. I tried to add paddings to make all structs' size
multiple of 8-bytes, I overlooked some.

>
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{in_mem_type}] is a type of buffer management for input
> > +buffers. The driver MUST set a value in \field{enum
> > +virtio_video_mem_type}.
> > +\item[\field{out_mem_type}] is a type of buffer management for output
> > +buffers. The driver MUST set a value in \field{enum
> > +virtio_video_mem_type}.
> > +\item[\field{debug_name}] is a text string for a debug purpose.
>
> Must be NUL-terminated ? UTF-8 ?

Yeah, I will add a description similar to one for "tag" in virtio_fs_config.

>
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_stream_destroy {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > +  queued buffers through the pipeline.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_stream_drain {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_mem_entry {
> > +        le64 addr;
> > +        le32 length;
> > +        u8 padding[4];
> > +};
> > +
> > +struct virtio_video_resource_create {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 resource_id;
> > +        le32 nr_entries;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{resource_id}] internal id of the resource.
>
> Is it input or output?

Will add a field for it.

>
> > +\item[\field{nr_entries}] number of \field{struct
> > +  virtio_video_mem_entry} memory entries.
>
> I suppose that the structure is followed by an array of
> virtio_video_mem_entry after a padding of 4-bytes for
> alignment.
>
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_resource_destroy {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 resource_id;
> > +        u8 padding[4];
>
> Is this padding for future extensions?

Nope, it's mistake. When I was editing structs, I forgot to add/remove
padding. I hope we will not have such mistakes after we have an
updated driver implementation.

>
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{resource_id}] internal id of the resource.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > +queue.
> > +
> > +\begin{lstlisting}
> > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > +
> > +struct virtio_video_resource_queue {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type;
> > +        le32 resource_id;
> > +        le64 timestamp;
>
> I suppose you would like this field aligned to avoid
> hidden paddings.

Thanks.

Best regards,
Keiichi

>
> > +        le32 nr_data_size;
> > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buf_type of the .
> > +\item[\field{resource_id}] internal id of the resource.
> > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > +  for synchronisation.
> > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > +\item[\field{data_size}] number of data bytes within a plane.
> > +\end{description}
> > +
> ... omissis ...
> > --
> > 2.24.1.735.g03f4e72817-goog
> >
> >
>
> Frediano
>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-19 13:28   ` Dmitry Sepp
@ 2019-12-20 15:26     ` Keiichi Watanabe
       [not found]       ` <2584386.DF4NACHtsB@os-lin-dmo>
  2019-12-30 12:16     ` [virtio-dev] " Dmitry Sepp
  1 sibling, 1 reply; 73+ messages in thread
From: Keiichi Watanabe @ 2019-12-20 15:26 UTC (permalink / raw)
  To: Dmitry Sepp
  Cc: virtio-dev, Linux Media Mailing List, Alexandre Courbot,
	Alex Lau, Daniel Vetter, Dylan Reid, Enrico Granata,
	Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, Tomasz Figa, uril

Hi Dmitry,
Thanks for the feedback.

On Thu, Dec 19, 2019 at 10:28 PM Dmitry Sepp
<dmitry.sepp@opensynergy.com> wrote:
>
> Hi Keiichi,
>
> Thank you for the update. Please see some comments below.
>
> Also, we need to bring the virtio_video_control back as it is in fact used by
> the driver to enumerate supported encoder controls. But yes, it still needs to
> be documemnted, it's true.

Yeah, we need it eventually, but I'm wondering what is the difference
between values set by SET_PARAMS and ones by controls.
I saw your driver implementation has BITRATE, PROFILE and LEVEL as
controls. What's the reason why you didn't add them in
virtio_video_params?

>
> On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> >
> > The virtio video encoder device and decoder device provide functionalities
> > to encode and decode video stream respectively.
> > Though video encoder and decoder are provided as different devices, they use
> > a same protocol.
> >
> > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > ---
> >  content.tex      |   1 +
> >  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 580 insertions(+)
> >  create mode 100644 virtio-video.tex
> >
> > diff --git a/content.tex b/content.tex
> > index 556b373..9e56839 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> >  \input{virtio-fs.tex}
> >  \input{virtio-rpmb.tex}
> > +\input{virtio-video.tex}
> >
> >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> >
> > diff --git a/virtio-video.tex b/virtio-video.tex
> > new file mode 100644
> > index 0000000..30e728d
> > --- /dev/null
> > +++ b/virtio-video.tex
> > @@ -0,0 +1,579 @@
> > +\section{Video Device}\label{sec:Device Types / Video Device}
> > +
> > +The virtio video encoder device and decoder device are virtual devices that
> > +supports encoding and decoding respectively. Though the encoder and the
> > decoder +are different devices, they use the same protocol.
> > +
> > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
> > +
> > +\begin{description}
> > +\item[30] encoder device
> > +\item[31] decoder device
> > +\end{description}
> > +
> > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues}
> > +
> > +\begin{description}
> > +\item[0] controlq - queue for sending control commands.
> > +\item[1] eventq - queue for sending events happened in the device.
> > +\end{description}
> > +
> > +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature
> > bits} +
> > +\begin{description}
> > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for
> > video +  buffers.
> > +\end{description}
> > +
> > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device
> > / Feature bits} +
> > +The device MUST offer at least one of feature bits.
> > +
> > +\subsection{Device configuration layout}\label{sec:Device Types / Video
> > Device / Device configuration layout} +
> > +Video device configuration uses the following layout structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_config {
> > +        le32 max_cap_len;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > +  MUST set this value.
> > +\end{description}
> > +
> > +\subsection{Device Initialization}\label{sec:Device Types / Video Device /
> > Device Initialization} +
> > +\devicenormative{\subsubsection}{Device Initialization}{Device Types /
> > Video Device / Device Initialization} +
> > +The driver SHOULD query device capability by using the
> > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> > +setup.
> > +
> > +\subsection{Device Operation}\label{sec:Device Types / Video Device /
> > Device Operation} +
> > +The driver allocates input and output buffers and queues the buffers
> > +to the device. The device performs operations on the buffers according
> > +to the function in question.
> > +
> > +\subsubsection{Device Operation: Create stream}
> > +
> > +To process buffers, the device needs to associate them with a certain
> > +video stream (essentially, a context). Streams are created by
> > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > +determined by the device.
> > +
> > +\subsubsection{Device Operation: Create buffers}
> > +
> > +Buffers are used to store the actual data as well as the relevant
> > +metadata. Scatter lists are supported, so the buffer doesn't need to
> > +be contiguous in guest physical memory.
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > +  resource that is backed by a buffer allocated from the driver's
> > +  memory.
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> > +  is no longer needed.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Stream parameter control}
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters
> > for +  input and output streams from the device.
> > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the
> > +  device.
> > +\item After setting stream parameters, the driver may issue
> > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can
> > be +  changed implicitly by the device during the set operation.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Process buffers}
> > +
> > +\begin{itemize*}
> > +\item If the function and the buffer type require so, write data to
> > +the buffer memory.
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > +processing in the device.
> > +\item The request completes asynchronously when the device has
> > +finished with the buffer.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Buffer processing control}
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> > +  return all of the already queued buffers.
> > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> > +  already queued buffers from the input or the output queue. This also
> > +  includes input or output buffers that can be currently owned by the
> > +  device's processing pipeline.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Asynchronous events}
> > +
> > +While processing buffers, the device can send asynchronous event
> > +notifications to the driver. The behaviour depends on the exact
> > +stream. For example, the decoder device sends a resolution change
> > +event when it encounters new resolution metadata in the stream.
> > +
> > +\subsubsection{Device Operation: Request header}
> > +
> > +All requests and responses on the control virt queue have a fixed
> > +header using the following layout structure and definitions:
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_ctrl_type {
> > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > +
> > +        /* request */
> > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > +
> > +        /* response */
> > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > +
> > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > +};
> > +
> > +struct virtio_video_ctrl_hdr {
> > +        le32 type;
> > +        le32 stream_id;
> > +        le32 len; /* Length of the structure in bytes. */
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{type}] is the type of the driver request or the device
> > +response.
> > +\item[\field{stream_id}] specifies a target stream.
> > +\item[\field{len}] is the length of data in bytes, which includes
> > +length of the header.
> > +\end{description}
> > +
> > +\subsubsection{Device Operation: controlq}
> > +
> > +\begin{description}
> > +
> > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > +supported formats.
> > +
> > +The driver uses \field{struct virtio_video_get_capability} to send a
> > +query request.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_buf_type {
> > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > +};
> I personally didn't like the previous term: pin_type. But, to be honest, I
> don't like the buf_type neither. Consider the GET/SET_PARAMS request: buf_type
> there looks a bit unnatural. We are trying to get stream parameters there, not
> some parameters of whatever buffer. Also I don't see any strict reason to mimic
> v4l2 naming scheme.
>
> I'd better rename it to PORT_TYPE or QUEUE_TYPE.

To be honest, I also wasn't satisfied with the name "buf_type".
QUEUE_TYPE sounds better.
Will rename it.

>
> > +
> > +struct virtio_video_get_capability {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        enum virtio_video_buf_type buf_type;
> > +};
> > +\end{lstlisting}
> > +\begin{description}
> > +\item[\field{buf_type}] is the buffer type that the driver asks
> > +information about. The driver MUST set either
> > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > +
> > +The device responds a capability by using \field{struct
> > +virtio_video_get_capability_resp}.
> > +\begin{lstlisting}
> > +enum virtio_video_format {
> > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > +        /* Raw formats */
> > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > +        VIRTIO_VIDEO_FORMAT_YVU420,
> Let's add some variants of RGB, like RGBA, ARGB. We need it for the encoder in
> particular .
>

Sounds good.
BTW, which "ARGB8888" or "A8R8G8B8" is preferred? While the first one
comes from DRM's FourCC, the second one comes from virtio_gpu_formats.
I personally prefer the first one, then we can have a naming convention like:
VIRTIO_VIDEO_FORMAT_<name from drm_fourcc.h>

> > +
> > +        /* Compressed formats */
> > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> Let's add H265, MPEG4, MPEG2. We already support and use them.

Sure.

Best regards,
Keiichi

>
> Regards,
> Dmitry.
>
> > +};
> > +
> > +enum virtio_video_profile {
> > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > +
> > +        /* H.264 */
> > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > +        /* VP8 */
> > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > +
> > +        /* VP9 */
> > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > +        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > +};
> > +
> > +struct virtio_video_format_range {
> > +        le32 min;
> > +        le32 max;
> > +        le32 step;
> > +        u8 paddings[4];
> > +};
> > +
> > +struct virtio_video_format_desc {
> > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > +        le64 mask;
> > +        struct virtio_video_format_range width;
> > +        struct virtio_video_format_range height;
> > +        le32 num_rates;
> > +        u8 padding[4];
> > +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> > +};
> > +
> > +struct virtio_video_get_capability_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 num_descs;
> > +        /* Followed by struct virtio_video_format_desc desc[] */
> > +};
> > +\end{lstlisting}
> > +
> > +The format description \field{struct virtio_video_format_desc}
> > +includes the following fields:
> > +\begin{description}
> > +\item[\field{format}] specifies an image format. The device MUST set one
> > +  of \field{enum virtio_video_format}.
> > +\item[\field{profile}] specifies a profile of the compressed image format
> > +  specified in \field{format}. The driver SHOULD ignore this value if
> > +  \field{format} is a raw format.
> > +\item[\field{mask}] is a bitset that represents the supported
> > +  combination of input and output format. If \textit{i}-th bit is set
> > +  in \field{mask} of \textit{j}-th \field{struct
> > +  virtio_video_format_desc} for input, the device supports encoding or
> > +  decoding from the \textit{j}-th input format to \textit{i}-th output
> > +  format.
> > +\item[\field{width, height}] represents a range of resolutions
> > +  supported by the device. If its \field{step} is not applicable, its
> > +  \field{min} is equal to its \field{max}.
> > +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In
> > case of decoder, the driver SHOULD ignore this value.
> > +\item[\field{frame_rates}] is an array of supported frame rates.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_mem_type {
> > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > +};
> > +
> > +struct virtio_video_stream_create {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > +        char debug_name[64];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{in_mem_type}] is a type of buffer management for input
> > +buffers. The driver MUST set a value in \field{enum
> > +virtio_video_mem_type}.
> > +\item[\field{out_mem_type}] is a type of buffer management for output
> > +buffers. The driver MUST set a value in \field{enum
> > +virtio_video_mem_type}.
> > +\item[\field{debug_name}] is a text string for a debug purpose.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_stream_destroy {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > +  queued buffers through the pipeline.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_stream_drain {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_mem_entry {
> > +        le64 addr;
> > +        le32 length;
> > +        u8 padding[4];
> > +};
> > +
> > +struct virtio_video_resource_create {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 resource_id;
> > +        le32 nr_entries;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{resource_id}] internal id of the resource.
> > +\item[\field{nr_entries}] number of \field{struct
> > +  virtio_video_mem_entry} memory entries.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_resource_destroy {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 resource_id;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{resource_id}] internal id of the resource.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > +queue.
> > +
> > +\begin{lstlisting}
> > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > +
> > +struct virtio_video_resource_queue {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type;
> > +        le32 resource_id;
> > +        le64 timestamp;
> > +        le32 nr_data_size;
> > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buf_type of the .
> > +\item[\field{resource_id}] internal id of the resource.
> > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > +  for synchronisation.
> > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > +\item[\field{data_size}] number of data bytes within a plane.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_buffer_flag {
> > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > +        /* Encoder only */
> > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > +};
> > +
> > +struct virtio_video_resource_queue_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le64 timestamp;
> > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > +        le32 size;  /* Encoded size */
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > +  for synchronisation.
> > +\item[\field{flags}] mark specific buffers in the sequence.
> > +\item[\field{size}] data size in the buffer (encoder only).
> > +\end{description}
> > +
> > +The device sends a response to the queue request asynchronously when
> > +it has finished processing the buffer.
> > +
> > +The device SHOULD mark a buffer that triggered a processing error with
> > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > +
> > +The device MUST mark the last buffer with the
> > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > +sequence.
> > +
> > +In case of encoder, to denote a particular frame type the devie MUST
> > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> > +  buffers back from the input or the output queue of the device. The
> > +  device SHOULD return all of the buffers from the respective queue as
> > +  soon as possible without pushing the buffers through the processing
> > +  pipeline.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_queue_clear {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buffer type.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> > +  output of a stream.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_plane_format {
> > +        le32 plane_size;
> > +        le32 stride;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{plane_size}] size of the plane in bytes.
> > +\item[\field{stride}] stride used for the plane in bytes.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_params {
> > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> > +        le32 frame_width;
> > +        le32 frame_height;
> > +        le32 min_buffers;
> > +        le32 max_buffers;
> > +        le32 frame_rate;
> > +        struct virtio_video_crop {
> > +                le32 left;
> > +                le32 top;
> > +                le32 width;
> > +                le32 height;
> > +        } crop;
> > +        le32 num_planes;
> > +        struct virtio_video_plane_format
> > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{frame_width}] the value to get/set.
> > +\item[\field{frame_height}] the value to get/set.
> > +\item[\field{pixel_format}] the value to get/set.
> > +\item[\field{min_buffers}] minimum buffers required to handle the
> > +  format (r/o).
> > +\item[\field{max_buffers}] maximum buffers required to handle the
> > +  format (r/o).
> > +\item[\field{frame_rate}] the value to get/set.
> > +\item[\field{crop}] cropping (composing) rectangle.
> > +\item[\field{num_planes}] number of planes used to store pixel data
> > +(r/o).
> > +\item[\field{plane_formats}] description of each plane.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_get_params {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > +};
> > +
> > +struct virtio_video_get_params_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        struct virtio_video_params params;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buffer type.
> > +\item[\field{params}] parameter values.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> > +
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_set_params {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        struct virtio_video_params params;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{params}] parameters to set.
> > +\end{description}
> > +
> > +Setting stream parameters might have side effects within the device.
> > +For example, the device MAY perform alignment of width and height,
> > +change the number of planes it uses for the format, or do whatever
> > +changes that are required to continue normal operation using the
> > +updated parameters. It is up to the driver to check the parameter set
> > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > +
> > +\end{description}
> > +
> > +\subsubsection{Device Operation: eventq}
> > +
> > +The device can report events on the event queue. The driver initially
> > +populates the queue with device-writeable buffers. When the device
> > +needs to report an event, it fills a buffer and notifies the driver.
> > +The driver consumes the report and adds a new buffer to the virtqueue.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_event_type {
> > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > +        /* For all functions */
> > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > +        /* For decoder only */
> > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> > +};
> > +
> > +struct virtio_video_event {
> > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> > +        le32 stream_id;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{event_type}] type of the triggered event .
> > +\item[\field{stream_id}] id of the source stream.
> > +\end{description}
> > +
> > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > +whenever it encounters new resolution data in the stream. This
> > +includes the case of the initial device configuration after metadata
> > +has been parsed and the case of dynamic resolution change.
>
>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-20 14:05     ` Keiichi Watanabe
@ 2019-12-20 15:33       ` Dmitry Sepp
  0 siblings, 0 replies; 73+ messages in thread
From: Dmitry Sepp @ 2019-12-20 15:33 UTC (permalink / raw)
  To: Keiichi Watanabe
  Cc: Frediano Ziglio, virtio-dev, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Enrico Granata, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, Tomasz Figa, uril

Hi Keiichi,

On Freitag, 20. Dezember 2019 15:05:02 CET Keiichi Watanabe wrote:
> Hi Frediano,
> Thanks for reviewing!
> 
> On Thu, Dec 19, 2019 at 2:29 AM Frediano Ziglio <fziglio@redhat.com> wrote:
> > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > 
> > > The virtio video encoder device and decoder device provide
> > > functionalities to encode and decode video stream respectively.
> > > Though video encoder and decoder are provided as different devices, they
> > > use a
> > > same protocol.
> > > 
> > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > > ---
> > > 
> > >  content.tex      |   1 +
> > >  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
> > >  2 files changed, 580 insertions(+)
> > >  create mode 100644 virtio-video.tex
> > > 
> > > diff --git a/content.tex b/content.tex
> > > index 556b373..9e56839 100644
> > > --- a/content.tex
> > > +++ b/content.tex
> > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > > Requirements}\label{sec:Device
> > > 
> > >  \input{virtio-vsock.tex}
> > >  \input{virtio-fs.tex}
> > >  \input{virtio-rpmb.tex}
> > > 
> > > +\input{virtio-video.tex}
> > > 
> > >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> > > 
> > > diff --git a/virtio-video.tex b/virtio-video.tex
> > > new file mode 100644
> > > index 0000000..30e728d
> > > --- /dev/null
> > > +++ b/virtio-video.tex
> > > @@ -0,0 +1,579 @@
> > > +\section{Video Device}\label{sec:Device Types / Video Device}
> > > +
> > > +The virtio video encoder device and decoder device are virtual devices
> > > that +supports encoding and decoding respectively. Though the encoder
> > > and the decoder
> > > +are different devices, they use the same protocol.
> > > +
> > > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device
> > > ID}
> > > +
> > > +\begin{description}
> > > +\item[30] encoder device
> > > +\item[31] decoder device
> > > +\end{description}
> > > +
> > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device /
> > > Virtqueues} +
> > > +\begin{description}
> > > +\item[0] controlq - queue for sending control commands.
> > > +\item[1] eventq - queue for sending events happened in the device.
> > > +\end{description}
> > > +
> > > +\subsection{Feature bits}\label{sec:Device Types / Video Device /
> > > Feature
> > > bits}
> > > +
> > > +\begin{description}
> > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used
> > > for
> > > video
> > > +  buffers.
> > > +\end{description}
> > > +
> > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video
> > > Device / Feature bits}
> > > +
> > > +The device MUST offer at least one of feature bits.
> > > +
> > > +\subsection{Device configuration layout}\label{sec:Device Types / Video
> > > Device / Device configuration layout}
> > > +
> > > +Video device configuration uses the following layout structure:
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_config {
> > > +        le32 max_cap_len;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> > > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > > +  MUST set this value.
> > > +\end{description}
> > > +
> > > +\subsection{Device Initialization}\label{sec:Device Types / Video
> > > Device /
> > > Device Initialization}
> > > +
> > > +\devicenormative{\subsubsection}{Device Initialization}{Device Types /
> > > Video Device / Device Initialization}
> > > +
> > > +The driver SHOULD query device capability by using the
> > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> > > +setup.
> > > +
> > > +\subsection{Device Operation}\label{sec:Device Types / Video Device /
> > > Device Operation}
> > > +
> > > +The driver allocates input and output buffers and queues the buffers
> > > +to the device. The device performs operations on the buffers according
> > > +to the function in question.
> > > +
> > > +\subsubsection{Device Operation: Create stream}
> > > +
> > > +To process buffers, the device needs to associate them with a certain
> > > +video stream (essentially, a context). Streams are created by
> > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > > +determined by the device.
> > > +
> > > +\subsubsection{Device Operation: Create buffers}
> > > +
> > > +Buffers are used to store the actual data as well as the relevant
> > > +metadata. Scatter lists are supported, so the buffer doesn't need to
> > > +be contiguous in guest physical memory.
> > > +
> > > +\begin{itemize*}
> > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > > +  resource that is backed by a buffer allocated from the driver's
> > > +  memory.
> > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> > > +  is no longer needed.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Stream parameter control}
> > > +
> > > +\begin{itemize*}
> > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream
> > > parameters for +  input and output streams from the device.
> > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to
> > > the +  device.
> > > +\item After setting stream parameters, the driver may issue
> > > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output
> > > can be
> > > +  changed implicitly by the device during the set operation.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Process buffers}
> > > +
> > > +\begin{itemize*}
> > > +\item If the function and the buffer type require so, write data to
> > > +the buffer memory.
> > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > > +processing in the device.
> > > +\item The request completes asynchronously when the device has
> > > +finished with the buffer.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Buffer processing control}
> > > +
> > > +\begin{itemize*}
> > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> > > +  return all of the already queued buffers.
> > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> > > +  already queued buffers from the input or the output queue. This also
> > > +  includes input or output buffers that can be currently owned by the
> > > +  device's processing pipeline.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Asynchronous events}
> > > +
> > > +While processing buffers, the device can send asynchronous event
> > > +notifications to the driver. The behaviour depends on the exact
> > > +stream. For example, the decoder device sends a resolution change
> > > +event when it encounters new resolution metadata in the stream.
> > > +
> > > +\subsubsection{Device Operation: Request header}
> > > +
> > > +All requests and responses on the control virt queue have a fixed
> > > +header using the following layout structure and definitions:
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_ctrl_type {
> > > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > > +
> > > +        /* request */
> > > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > > +
> > > +        /* response */
> > > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > > +
> > > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > > +};
> > > +
> > > +struct virtio_video_ctrl_hdr {
> > > +        le32 type;
> > > +        le32 stream_id;
> > > +        le32 len; /* Length of the structure in bytes. */
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{type}] is the type of the driver request or the device
> > > +response.
> > > +\item[\field{stream_id}] specifies a target stream.
> > > +\item[\field{len}] is the length of data in bytes, which includes
> > > +length of the header.
> > 
> > I suppose is implicit that it's the size of the container structure.
> 
> You're right. All struct except virtio_video_get_capability_resp have
> fixed size and virtio_video_get_capability_resp has num_descs. So, no
> need to have len.
> Let me remove this in the next iteration.
> 
> > > +\end{description}
> > > +
> > > +\subsubsection{Device Operation: controlq}
> > > +
> > > +\begin{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > > +supported formats.
> > > +
> > > +The driver uses \field{struct virtio_video_get_capability} to send a
> > > +query request.
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_buf_type {
> > > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > > +};
> > > +
> > > +struct virtio_video_get_capability {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        enum virtio_video_buf_type buf_type;
> > 
> > Here you used an enumeration but later you are using just le32 type
> > with a comment. Why this difference?
> 
> No difference. I will change this to use le32 since virtio-gpu's spec does
> so.
> > > +};
> > > +\end{lstlisting}
> > > +\begin{description}
> > > +\item[\field{buf_type}] is the buffer type that the driver asks
> > > +information about. The driver MUST set either
> > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > > +
> > > +The device responds a capability by using \field{struct
> > > +virtio_video_get_capability_resp}.
> > > +\begin{lstlisting}
> > > +enum virtio_video_format {
> > > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > 
> > Really minor: maybe you want to insert an empty line to be coherent
> > with other enumerations?
> 
> Yes. Thanks.
> 
> > > +        /* Raw formats */
> > > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > > +        VIRTIO_VIDEO_FORMAT_YVU420,
> > > +
> > > +        /* Compressed formats */
> > > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> > > +};
> > > +
> > > +enum virtio_video_profile {
> > > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > > +
> > > +        /* H.264 */
> > > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > > VIRTIO_VIDEO_PROFILE_H264_BASELINE,
> > 
> > Maybe you want
> > 
> > VIRTIO_VIDEO_PROFILE_H264_BASELINE = VIRTIO_VIDEO_PROFILE_H264_MIN,
> > 
> > like others?
> 
> Thanks for catching this. I think we can find this kind of errors once
> we update the driver implementation.
> 
> > > +        VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > > +
> > > +        /* VP8 */
> > > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > > +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > > +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > > +
> > > +        /* VP9 */
> > > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 =
> > > VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_MAX =
> > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > +};
> > > +
> > > +struct virtio_video_format_range {
> > > +        le32 min;
> > > +        le32 max;
> > > +        le32 step;
> > > +        u8 paddings[4];
> > 
> > Here you padded at 8-byte like many other structures but not for
> > virtio_video_ctrl_hdr. Is this expected?
> 
> I padded here for 64-bits alignment. So, I should have padded for
> ctrl_hdr as you said. But, no need anymore as we will remove |len|
> from the struct as you suggested above:)
> 
> > > +};
> > > +
> > > +struct virtio_video_format_desc {
> > > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > > +        le64 mask;
> > 
> > This will limit the formats to 64 while "num_descs" below is a
> > 32 bit.
> 
> It's my intention. I will explicitly write that "num_desc" must not exceed
> 64. I guess 64 is big enough as a number of supported input/output format
> and couldn't find a better idea to represent this bipartite graph. If you
> have another idea, please let me hear.
> 
> > > +        struct virtio_video_format_range width;
> > > +        struct virtio_video_format_range height;
> > > +        le32 num_rates;
> > > +        u8 padding[4];
> > > +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> > > +};
> > > +
> > > +struct virtio_video_get_capability_resp {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 num_descs;
> > > +        /* Followed by struct virtio_video_format_desc desc[] */
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +The format description \field{struct virtio_video_format_desc}
> > > +includes the following fields:
> > > +\begin{description}
> > > +\item[\field{format}] specifies an image format. The device MUST set
> > > one
> > > +  of \field{enum virtio_video_format}.
> > > +\item[\field{profile}] specifies a profile of the compressed image
> > > format
> > > +  specified in \field{format}. The driver SHOULD ignore this value if
> > > +  \field{format} is a raw format.
> > 
> > Why not documenting that MUST be VIRTIO_VIDEO_PROFILE_UNDEFINED for
> > raw formats so the field could be extended in the future?
> 
> Sounds good. Will do.
> 
> > > +\item[\field{mask}] is a bitset that represents the supported
> > > +  combination of input and output format. If \textit{i}-th bit is set
> > > +  in \field{mask} of \textit{j}-th \field{struct
> > > +  virtio_video_format_desc} for input, the device supports encoding or
> > > +  decoding from the \textit{j}-th input format to \textit{i}-th output
> > > +  format.
> > > +\item[\field{width, height}] represents a range of resolutions
> > > +  supported by the device. If its \field{step} is not applicable, its
> > > +  \field{min} is equal to its \field{max}.
> > > +\item[\field{num_rates}] is the length of an array \field{frame_rates}.
> > > In
> > > case of decoder, the driver SHOULD ignore this value.
> > > +\item[\field{frame_rates}] is an array of supported frame rates.
> > 
> > I suppose here we are talking about bitrates, right? Could be confused by
> > FPS.
> 
> Nope. This is for frame rates or frame intervals. In V4L2 driver, the
> values will be used for VIDIOC_ENUM_FRAMEINTERVALS ioctl.
> (Dmitry, please correct me if I am wrong)
Right, this is correct.

Regards,
Dmitry.

> 
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_mem_type {
> > > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > > +};
> > > +
> > > +struct virtio_video_stream_create {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > > +        char debug_name[64];
> > > +};
> > 
> > This structure has a size not multiple of 8-bytes. Not an issue, but
> > the same apply to other structures that instead are padded to 8-byte size.
> 
> It's my fault. I tried to add paddings to make all structs' size
> multiple of 8-bytes, I overlooked some.
> 
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{in_mem_type}] is a type of buffer management for input
> > > +buffers. The driver MUST set a value in \field{enum
> > > +virtio_video_mem_type}.
> > > +\item[\field{out_mem_type}] is a type of buffer management for output
> > > +buffers. The driver MUST set a value in \field{enum
> > > +virtio_video_mem_type}.
> > > +\item[\field{debug_name}] is a text string for a debug purpose.
> > 
> > Must be NUL-terminated ? UTF-8 ?
> 
> Yeah, I will add a description similar to one for "tag" in virtio_fs_config.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_stream_destroy {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > > +  queued buffers through the pipeline.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_stream_drain {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_mem_entry {
> > > +        le64 addr;
> > > +        le32 length;
> > > +        u8 padding[4];
> > > +};
> > > +
> > > +struct virtio_video_resource_create {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 resource_id;
> > > +        le32 nr_entries;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{resource_id}] internal id of the resource.
> > 
> > Is it input or output?
> 
> Will add a field for it.
> 
> > > +\item[\field{nr_entries}] number of \field{struct
> > > +  virtio_video_mem_entry} memory entries.
> > 
> > I suppose that the structure is followed by an array of
> > virtio_video_mem_entry after a padding of 4-bytes for
> > alignment.
> > 
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_resource_destroy {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 resource_id;
> > > +        u8 padding[4];
> > 
> > Is this padding for future extensions?
> 
> Nope, it's mistake. When I was editing structs, I forgot to add/remove
> padding. I hope we will not have such mistakes after we have an
> updated driver implementation.
> 
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{resource_id}] internal id of the resource.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > > +queue.
> > > +
> > > +\begin{lstlisting}
> > > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > > +
> > > +struct virtio_video_resource_queue {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 buf_type;
> > > +        le32 resource_id;
> > > +        le64 timestamp;
> > 
> > I suppose you would like this field aligned to avoid
> > hidden paddings.
> 
> Thanks.
> 
> Best regards,
> Keiichi
> 
> > > +        le32 nr_data_size;
> > > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{buf_type}] buf_type of the .
> > > +\item[\field{resource_id}] internal id of the resource.
> > > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > > +  for synchronisation.
> > > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > > +\item[\field{data_size}] number of data bytes within a plane.
> > > +\end{description}
> > > +
> > 
> > ... omissis ...
> > 
> > > --
> > > 2.24.1.735.g03f4e72817-goog
> > 
> > Frediano



^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 0/1] VirtIO video device specification
  2019-12-18 13:02 [PATCH v2 0/1] VirtIO video device specification Keiichi Watanabe
  2019-12-18 13:02 ` [PATCH v2 1/1] virtio-video: Add virtio " Keiichi Watanabe
@ 2019-12-20 15:58 ` Dmitry Sepp
  2019-12-21  4:36   ` Keiichi Watanabe
  1 sibling, 1 reply; 73+ messages in thread
From: Dmitry Sepp @ 2019-12-20 15:58 UTC (permalink / raw)
  To: Keiichi Watanabe
  Cc: virtio-dev, linux-media, acourbot, alexlau, daniel, dgreid,
	egranata, fziglio, hverkuil, kraxel, marcheu, posciak,
	spice-devel, stevensd, tfiga, uril

Hi Keiichi,

On Mittwoch, 18. Dezember 2019 14:02:13 CET Keiichi Watanabe wrote:
> Hi,
> This is the 2nd version of virtio-video patch. The PDF is available in [1].
> The first version was sent at [2].
> 
> Any feedback would be appreciated. Thank you.
> 
> Best,
> Keiichi
> 
> [1]:
> https://drive.google.com/drive/folders/1eT5fEckBoor2iHZR4f4GLxYzFMVapOFx?us
> p=sharing [2]: https://markmail.org/message/gc6h25acct22niut
> 
> Change log:
> 
> v2:
> * Removed functionalities except encoding and decoding.
> * Splited encoder and decoder into different devices that use the same
> protocol. * Replaced GET_FUNCS with GET_CAPABILITY.
> * Updated structs for capabilities.
>   - Defined new structs and enums such as image formats, profiles, range
> (min, max, step), etc
>     * For virtio_video_pixel_format, chose a naming convention that is used
>       in DRM. We removed XBGR, NV21 and I422, as they are not used in the
>       current draft implementation. https://lwn.net/Articles/806416/
>   - Removed virtio_video_control, whose usage was not documented yet and
> which is not necessary for the simplest decoding scenario.
>   - Removed virtio_video_desc, as it is no longer needed.
> * Updated struct virtio_video_config for changes around capabilities.
> * Added a way to represent supported combinations of formats.
>   - A field "mask" in virtio_video_format_desc plays this role.
> * Removed VIRTIO_VIDEO_T_STREAM_{START,STOP} because they don't play any
> meaningful roles. * Removed VIRTIO_VIDEO_T_STREAM_{ATTACH, DETACH}_BACKING
> and merged them into RESOURCE_{CREATE, DESTROY}. * Added a way to
> notify/specify resource creation method.
>   - Added a feature flag.
>   - Defined enum virtio_video_mem_type.
>   - Added new fields in video_stream_create.
> * Modified fields in virtio_video_params.
>   - Added crop information.
> * Removed enum virtio_video_channel_type because we can get this information
> by image format. 
Could you please explain this? How do you get the information?

Suppose you have some piece of HW on the host side that wants I420 as one 
contig buffer w/ some offsets. But on the driver side, say, gralloc gives you 
three separate buffers, one per channel. How do we pass those to the device 
then?

Best regards,
Dmitry.

> * Renamed virtio_video_pin to virtio_video_buf_type.
>   - It's similar to V4L2_BUF_TYPE_VIDEO_{OUTPUT, CAPTURE}.
> * Added an error event.
> * Reordered some subsections.
> * Changed styles to make it consistent with other devices.
> 
> Dmitry Sepp (1):
>   virtio-video: Add virtio video device specification
> 
>  content.tex      |   1 +
>  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 580 insertions(+)
>  create mode 100644 virtio-video.tex
> 
> --
> 2.24.1.735.g03f4e72817-goog



^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 0/1] VirtIO video device specification
  2019-12-20 15:58 ` [PATCH v2 0/1] VirtIO " Dmitry Sepp
@ 2019-12-21  4:36   ` Keiichi Watanabe
  2019-12-21  6:18     ` Tomasz Figa
  0 siblings, 1 reply; 73+ messages in thread
From: Keiichi Watanabe @ 2019-12-21  4:36 UTC (permalink / raw)
  To: Dmitry Sepp
  Cc: virtio-dev, Linux Media Mailing List, Alexandre Courbot,
	Alex Lau, Daniel Vetter, Dylan Reid, Enrico Granata,
	Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, Tomasz Figa, uril

Hi Dmitry,

On Sat, Dec 21, 2019 at 12:59 AM Dmitry Sepp
<dmitry.sepp@opensynergy.com> wrote:
>
> Hi Keiichi,
>
> On Mittwoch, 18. Dezember 2019 14:02:13 CET Keiichi Watanabe wrote:
> > Hi,
> > This is the 2nd version of virtio-video patch. The PDF is available in [1].
> > The first version was sent at [2].
> >
> > Any feedback would be appreciated. Thank you.
> >
> > Best,
> > Keiichi
> >
> > [1]:
> > https://drive.google.com/drive/folders/1eT5fEckBoor2iHZR4f4GLxYzFMVapOFx?us
> > p=sharing [2]: https://markmail.org/message/gc6h25acct22niut
> >
> > Change log:
> >
> > v2:
> > * Removed functionalities except encoding and decoding.
> > * Splited encoder and decoder into different devices that use the same
> > protocol. * Replaced GET_FUNCS with GET_CAPABILITY.
> > * Updated structs for capabilities.
> >   - Defined new structs and enums such as image formats, profiles, range
> > (min, max, step), etc
> >     * For virtio_video_pixel_format, chose a naming convention that is used
> >       in DRM. We removed XBGR, NV21 and I422, as they are not used in the
> >       current draft implementation. https://lwn.net/Articles/806416/
> >   - Removed virtio_video_control, whose usage was not documented yet and
> > which is not necessary for the simplest decoding scenario.
> >   - Removed virtio_video_desc, as it is no longer needed.
> > * Updated struct virtio_video_config for changes around capabilities.
> > * Added a way to represent supported combinations of formats.
> >   - A field "mask" in virtio_video_format_desc plays this role.
> > * Removed VIRTIO_VIDEO_T_STREAM_{START,STOP} because they don't play any
> > meaningful roles. * Removed VIRTIO_VIDEO_T_STREAM_{ATTACH, DETACH}_BACKING
> > and merged them into RESOURCE_{CREATE, DESTROY}. * Added a way to
> > notify/specify resource creation method.
> >   - Added a feature flag.
> >   - Defined enum virtio_video_mem_type.
> >   - Added new fields in video_stream_create.
> > * Modified fields in virtio_video_params.
> >   - Added crop information.
> > * Removed enum virtio_video_channel_type because we can get this information
> > by image format.
> Could you please explain this? How do you get the information?

It means that if image formats are well-defined, channel information
(e.g. the order of channels) is uniquely determined.

>
> Suppose you have some piece of HW on the host side that wants I420 as one
> contig buffer w/ some offsets. But on the driver side, say, gralloc gives you
> three separate buffers, one per channel. How do we pass those to the device
> then?

You're talking about CrOS use case where buffers are allocated by
virtio-gpu, right?
In this case, virtio-gpu allocates one contiguous host-side buffer and
the client regards a pair of (buffer FD, offset) as one channel.
And, we can register this pair to the device when the buffer is imported.
In the virtio-vdec spec draft, this pair corresponds to struct
virtio_vdec_plane in struct virtio_vdec_plane.

So, I suppose we will need similar structs when we add a control to
import buffers. However, I don't think it's necessary when guest pages
are used.

Best regards,
Keiichi


>
> Best regards,
> Dmitry.
>
> > * Renamed virtio_video_pin to virtio_video_buf_type.
> >   - It's similar to V4L2_BUF_TYPE_VIDEO_{OUTPUT, CAPTURE}.
> > * Added an error event.
> > * Reordered some subsections.
> > * Changed styles to make it consistent with other devices.
> >
> > Dmitry Sepp (1):
> >   virtio-video: Add virtio video device specification
> >
> >  content.tex      |   1 +
> >  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 580 insertions(+)
> >  create mode 100644 virtio-video.tex
> >
> > --
> > 2.24.1.735.g03f4e72817-goog
>
>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 0/1] VirtIO video device specification
  2019-12-21  4:36   ` Keiichi Watanabe
@ 2019-12-21  6:18     ` Tomasz Figa
  2019-12-21  6:19       ` Tomasz Figa
  0 siblings, 1 reply; 73+ messages in thread
From: Tomasz Figa @ 2019-12-21  6:18 UTC (permalink / raw)
  To: Keiichi Watanabe
  Cc: Dmitry Sepp, virtio-dev, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Enrico Granata, Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

On Sat, Dec 21, 2019 at 1:36 PM Keiichi Watanabe <keiichiw@chromium.org> wrote:
>
> Hi Dmitry,
>
> On Sat, Dec 21, 2019 at 12:59 AM Dmitry Sepp
> <dmitry.sepp@opensynergy.com> wrote:
> >
> > Hi Keiichi,
> >
> > On Mittwoch, 18. Dezember 2019 14:02:13 CET Keiichi Watanabe wrote:
> > > Hi,
> > > This is the 2nd version of virtio-video patch. The PDF is available in [1].
> > > The first version was sent at [2].
> > >
> > > Any feedback would be appreciated. Thank you.
> > >
> > > Best,
> > > Keiichi
> > >
> > > [1]:
> > > https://drive.google.com/drive/folders/1eT5fEckBoor2iHZR4f4GLxYzFMVapOFx?us
> > > p=sharing [2]: https://markmail.org/message/gc6h25acct22niut
> > >
> > > Change log:
> > >
> > > v2:
> > > * Removed functionalities except encoding and decoding.
> > > * Splited encoder and decoder into different devices that use the same
> > > protocol. * Replaced GET_FUNCS with GET_CAPABILITY.
> > > * Updated structs for capabilities.
> > >   - Defined new structs and enums such as image formats, profiles, range
> > > (min, max, step), etc
> > >     * For virtio_video_pixel_format, chose a naming convention that is used
> > >       in DRM. We removed XBGR, NV21 and I422, as they are not used in the
> > >       current draft implementation. https://lwn.net/Articles/806416/
> > >   - Removed virtio_video_control, whose usage was not documented yet and
> > > which is not necessary for the simplest decoding scenario.
> > >   - Removed virtio_video_desc, as it is no longer needed.
> > > * Updated struct virtio_video_config for changes around capabilities.
> > > * Added a way to represent supported combinations of formats.
> > >   - A field "mask" in virtio_video_format_desc plays this role.
> > > * Removed VIRTIO_VIDEO_T_STREAM_{START,STOP} because they don't play any
> > > meaningful roles. * Removed VIRTIO_VIDEO_T_STREAM_{ATTACH, DETACH}_BACKING
> > > and merged them into RESOURCE_{CREATE, DESTROY}. * Added a way to
> > > notify/specify resource creation method.
> > >   - Added a feature flag.
> > >   - Defined enum virtio_video_mem_type.
> > >   - Added new fields in video_stream_create.
> > > * Modified fields in virtio_video_params.
> > >   - Added crop information.
> > > * Removed enum virtio_video_channel_type because we can get this information
> > > by image format.
> > Could you please explain this? How do you get the information?
>
> It means that if image formats are well-defined, channel information
> (e.g. the order of channels) is uniquely determined.
>
> >
> > Suppose you have some piece of HW on the host side that wants I420 as one
> > contig buffer w/ some offsets. But on the driver side, say, gralloc gives you
> > three separate buffers, one per channel. How do we pass those to the device
> > then?
>
> You're talking about CrOS use case where buffers are allocated by
> virtio-gpu, right?
> In this case, virtio-gpu allocates one contiguous host-side buffer and
> the client regards a pair of (buffer FD, offset) as one channel.
> And, we can register this pair to the device when the buffer is imported.
> In the virtio-vdec spec draft, this pair corresponds to struct
> virtio_vdec_plane in struct virtio_vdec_plane.
>
> So, I suppose we will need similar structs when we add a control to
> import buffers. However, I don't think it's necessary when guest pages
> are used.

I think we need some way for the guest to know whether it can allocate
the planes in separate buffers, even when guest pages are used. This
would be equivalent to V4L2 M and non-M formats, but mixing this into
FourCC in V4L2 is an acknowledged mistake, so we should add a query or
something.

For future V4L2 development we came up with the idea of a format flag
which could mean that the hardware allows putting planes in separate
buffers. We could have a similar per-format flag in the capabilities,
as we already have a list of all the supported formats there.

Best regards,
Tomasz

>
> Best regards,
> Keiichi
>
>
> >
> > Best regards,
> > Dmitry.
> >
> > > * Renamed virtio_video_pin to virtio_video_buf_type.
> > >   - It's similar to V4L2_BUF_TYPE_VIDEO_{OUTPUT, CAPTURE}.
> > > * Added an error event.
> > > * Reordered some subsections.
> > > * Changed styles to make it consistent with other devices.
> > >
> > > Dmitry Sepp (1):
> > >   virtio-video: Add virtio video device specification
> > >
> > >  content.tex      |   1 +
> > >  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
> > >  2 files changed, 580 insertions(+)
> > >  create mode 100644 virtio-video.tex
> > >
> > > --
> > > 2.24.1.735.g03f4e72817-goog
> >
> >

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 0/1] VirtIO video device specification
  2019-12-21  6:18     ` Tomasz Figa
@ 2019-12-21  6:19       ` Tomasz Figa
  2020-01-03 13:05         ` Dmitry Sepp
  0 siblings, 1 reply; 73+ messages in thread
From: Tomasz Figa @ 2019-12-21  6:19 UTC (permalink / raw)
  To: Keiichi Watanabe
  Cc: Dmitry Sepp, virtio-dev, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Enrico Granata, Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

On Sat, Dec 21, 2019 at 3:18 PM Tomasz Figa <tfiga@chromium.org> wrote:
>
> On Sat, Dec 21, 2019 at 1:36 PM Keiichi Watanabe <keiichiw@chromium.org> wrote:
> >
> > Hi Dmitry,
> >
> > On Sat, Dec 21, 2019 at 12:59 AM Dmitry Sepp
> > <dmitry.sepp@opensynergy.com> wrote:
> > >
> > > Hi Keiichi,
> > >
> > > On Mittwoch, 18. Dezember 2019 14:02:13 CET Keiichi Watanabe wrote:
> > > > Hi,
> > > > This is the 2nd version of virtio-video patch. The PDF is available in [1].
> > > > The first version was sent at [2].
> > > >
> > > > Any feedback would be appreciated. Thank you.
> > > >
> > > > Best,
> > > > Keiichi
> > > >
> > > > [1]:
> > > > https://drive.google.com/drive/folders/1eT5fEckBoor2iHZR4f4GLxYzFMVapOFx?us
> > > > p=sharing [2]: https://markmail.org/message/gc6h25acct22niut
> > > >
> > > > Change log:
> > > >
> > > > v2:
> > > > * Removed functionalities except encoding and decoding.
> > > > * Splited encoder and decoder into different devices that use the same
> > > > protocol. * Replaced GET_FUNCS with GET_CAPABILITY.
> > > > * Updated structs for capabilities.
> > > >   - Defined new structs and enums such as image formats, profiles, range
> > > > (min, max, step), etc
> > > >     * For virtio_video_pixel_format, chose a naming convention that is used
> > > >       in DRM. We removed XBGR, NV21 and I422, as they are not used in the
> > > >       current draft implementation. https://lwn.net/Articles/806416/
> > > >   - Removed virtio_video_control, whose usage was not documented yet and
> > > > which is not necessary for the simplest decoding scenario.
> > > >   - Removed virtio_video_desc, as it is no longer needed.
> > > > * Updated struct virtio_video_config for changes around capabilities.
> > > > * Added a way to represent supported combinations of formats.
> > > >   - A field "mask" in virtio_video_format_desc plays this role.
> > > > * Removed VIRTIO_VIDEO_T_STREAM_{START,STOP} because they don't play any
> > > > meaningful roles. * Removed VIRTIO_VIDEO_T_STREAM_{ATTACH, DETACH}_BACKING
> > > > and merged them into RESOURCE_{CREATE, DESTROY}. * Added a way to
> > > > notify/specify resource creation method.
> > > >   - Added a feature flag.
> > > >   - Defined enum virtio_video_mem_type.
> > > >   - Added new fields in video_stream_create.
> > > > * Modified fields in virtio_video_params.
> > > >   - Added crop information.
> > > > * Removed enum virtio_video_channel_type because we can get this information
> > > > by image format.
> > > Could you please explain this? How do you get the information?
> >
> > It means that if image formats are well-defined, channel information
> > (e.g. the order of channels) is uniquely determined.
> >
> > >
> > > Suppose you have some piece of HW on the host side that wants I420 as one
> > > contig buffer w/ some offsets. But on the driver side, say, gralloc gives you
> > > three separate buffers, one per channel. How do we pass those to the device
> > > then?
> >
> > You're talking about CrOS use case where buffers are allocated by
> > virtio-gpu, right?
> > In this case, virtio-gpu allocates one contiguous host-side buffer and
> > the client regards a pair of (buffer FD, offset) as one channel.
> > And, we can register this pair to the device when the buffer is imported.
> > In the virtio-vdec spec draft, this pair corresponds to struct
> > virtio_vdec_plane in struct virtio_vdec_plane.
> >
> > So, I suppose we will need similar structs when we add a control to
> > import buffers. However, I don't think it's necessary when guest pages
> > are used.
>
> I think we need some way for the guest to know whether it can allocate
> the planes in separate buffers, even when guest pages are used. This
> would be equivalent to V4L2 M and non-M formats, but mixing this into
> FourCC in V4L2 is an acknowledged mistake, so we should add a query or
> something.
>
> For future V4L2 development we came up with the idea of a format flag
> which could mean that the hardware allows putting planes in separate
> buffers. We could have a similar per-format flag in the capabilities,
> as we already have a list of all the supported formats there.

Sorry, forgot to paste the link from future V4L2 work notes from this year ELCE:
https://www.spinics.net/lists/linux-media/msg159789.html

>
> Best regards,
> Tomasz
>
> >
> > Best regards,
> > Keiichi
> >
> >
> > >
> > > Best regards,
> > > Dmitry.
> > >
> > > > * Renamed virtio_video_pin to virtio_video_buf_type.
> > > >   - It's similar to V4L2_BUF_TYPE_VIDEO_{OUTPUT, CAPTURE}.
> > > > * Added an error event.
> > > > * Reordered some subsections.
> > > > * Changed styles to make it consistent with other devices.
> > > >
> > > > Dmitry Sepp (1):
> > > >   virtio-video: Add virtio video device specification
> > > >
> > > >  content.tex      |   1 +
> > > >  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
> > > >  2 files changed, 580 insertions(+)
> > > >  create mode 100644 virtio-video.tex
> > > >
> > > > --
> > > > 2.24.1.735.g03f4e72817-goog
> > >
> > >

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
       [not found]       ` <2584386.DF4NACHtsB@os-lin-dmo>
@ 2019-12-21  6:46         ` Tomasz Figa
  0 siblings, 0 replies; 73+ messages in thread
From: Tomasz Figa @ 2019-12-21  6:46 UTC (permalink / raw)
  To: Dmitry Sepp
  Cc: Keiichi Watanabe, virtio-dev, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Enrico Granata, Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

On Sat, Dec 21, 2019 at 12:46 AM Dmitry Sepp
<dmitry.sepp@opensynergy.com> wrote:
> On Freitag, 20. Dezember 2019 16:26:50 CET Keiichi Watanabe wrote:
> > On Thu, Dec 19, 2019 at 10:28 PM Dmitry Sepp
> > <dmitry.sepp@opensynergy.com> wrote:
> > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
[snip]
> > > > +enum virtio_video_format {
> > > > + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > > > + /* Raw formats */
> > > > + VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > > > + VIRTIO_VIDEO_FORMAT_YUV420,
> > > > + VIRTIO_VIDEO_FORMAT_YVU420,
> > >
> > > Let's add some variants of RGB, like RGBA, ARGB. We need it for the
> > > encoder in particular .
> >
> > Sounds good.
> > BTW, which "ARGB8888" or "A8R8G8B8" is preferred? While the first one
> > comes from DRM's FourCC, the second one comes from virtio_gpu_formats.
> > I personally prefer the first one, then we can have a naming convention
> > like: VIRTIO_VIDEO_FORMAT_<name from drm_fourcc.h>
> >
>
> I'd go with ARGB8888 and BGRA8888 (might be with X variants).

Just to make sure we're talking about the same formats. DRM naming
convention is based on the little endian convention, which for 32-bit
formats means that you interpret the whole pixel as a packed 32-bit
word on a little endian system. For ARGB8888 that would mean (bit 31)
ARGB (bit 0) in the 32-bit word and (byte 0) B, G, R, A (byte 3) when
looking at separate bytes in memory. Does that correspond to your
expected format?

We also have to be specific about the A and X formats, as the A format
should be supported only if the hardware (host) doesn't ignore the
alpha channel. I haven't seen any hardware capable of encoding alpha
channel yet, but apparently for WebM the standard is to just encode
the alpha channel into another stream as Y, together with dummy U and
V values. [1] That sounds like something that would be handled by two
separate encoding streams and not just one that accepts RGBA on the
input.

[1] http://wiki.webmproject.org/alpha-channel

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [virtio-dev] Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-19 13:28   ` Dmitry Sepp
  2019-12-20 15:26     ` Keiichi Watanabe
@ 2019-12-30 12:16     ` Dmitry Sepp
  2020-01-06  6:31       ` Tomasz Figa
  1 sibling, 1 reply; 73+ messages in thread
From: Dmitry Sepp @ 2019-12-30 12:16 UTC (permalink / raw)
  To: virtio-dev
  Cc: Keiichi Watanabe, linux-media, acourbot, alexlau, daniel, dgreid,
	egranata, fziglio, hverkuil, kraxel, marcheu, posciak,
	spice-devel, stevensd, tfiga, uril


On Donnerstag, 19. Dezember 2019 14:28:23 CET Dmitry Sepp wrote:
> Hi Keiichi,
> 
> Thank you for the update. Please see some comments below.
> 
> Also, we need to bring the virtio_video_control back as it is in fact used
> by the driver to enumerate supported encoder controls. But yes, it still
> needs to be documemnted, it's true.
> 
> On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > 
> > The virtio video encoder device and decoder device provide functionalities
> > to encode and decode video stream respectively.
> > Though video encoder and decoder are provided as different devices, they
> > use a same protocol.
> > 
> > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > ---
> > 
> >  content.tex      |   1 +
> >  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 580 insertions(+)
> >  create mode 100644 virtio-video.tex
> > 
> > diff --git a/content.tex b/content.tex
> > index 556b373..9e56839 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> > 
> >  \input{virtio-fs.tex}
> >  \input{virtio-rpmb.tex}
> > 
> > +\input{virtio-video.tex}
> > 
> >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> > 
> > diff --git a/virtio-video.tex b/virtio-video.tex
> > new file mode 100644
> > index 0000000..30e728d
> > --- /dev/null
> > +++ b/virtio-video.tex
> > @@ -0,0 +1,579 @@
> > +\section{Video Device}\label{sec:Device Types / Video Device}
> > +
> > +The virtio video encoder device and decoder device are virtual devices
> > that +supports encoding and decoding respectively. Though the encoder and
> > the decoder +are different devices, they use the same protocol.
> > +
> > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
> > +
> > +\begin{description}
> > +\item[30] encoder device
> > +\item[31] decoder device
> > +\end{description}
> > +
> > +\subsection{Virtqueues}\label{sec:Device Types / Video Device /
> > Virtqueues} +
> > +\begin{description}
> > +\item[0] controlq - queue for sending control commands.
> > +\item[1] eventq - queue for sending events happened in the device.
> > +\end{description}
> > +
> > +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature
> > bits} +
> > +\begin{description}
> > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used
> > for
> > video +  buffers.
> > +\end{description}
> > +
> > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video
> > Device
> > / Feature bits} +
> > +The device MUST offer at least one of feature bits.
> > +
> > +\subsection{Device configuration layout}\label{sec:Device Types / Video
> > Device / Device configuration layout} +
> > +Video device configuration uses the following layout structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_config {
> > +        le32 max_cap_len;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > +  MUST set this value.
> > +\end{description}
> > +
> > +\subsection{Device Initialization}\label{sec:Device Types / Video Device
> > /
> > Device Initialization} +
> > +\devicenormative{\subsubsection}{Device Initialization}{Device Types /
> > Video Device / Device Initialization} +
> > +The driver SHOULD query device capability by using the
> > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> > +setup.
> > +
> > +\subsection{Device Operation}\label{sec:Device Types / Video Device /
> > Device Operation} +
> > +The driver allocates input and output buffers and queues the buffers
> > +to the device. The device performs operations on the buffers according
> > +to the function in question.
> > +
> > +\subsubsection{Device Operation: Create stream}
> > +
> > +To process buffers, the device needs to associate them with a certain
> > +video stream (essentially, a context). Streams are created by
> > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > +determined by the device.
> > +
> > +\subsubsection{Device Operation: Create buffers}
> > +
> > +Buffers are used to store the actual data as well as the relevant
> > +metadata. Scatter lists are supported, so the buffer doesn't need to
> > +be contiguous in guest physical memory.
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > +  resource that is backed by a buffer allocated from the driver's
> > +  memory.
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> > +  is no longer needed.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Stream parameter control}
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters
> > for +  input and output streams from the device.
> > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to
> > the +  device.
> > +\item After setting stream parameters, the driver may issue
> > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output
> > can be +  changed implicitly by the device during the set operation.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Process buffers}
> > +
> > +\begin{itemize*}
> > +\item If the function and the buffer type require so, write data to
> > +the buffer memory.
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > +processing in the device.
> > +\item The request completes asynchronously when the device has
> > +finished with the buffer.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Buffer processing control}
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> > +  return all of the already queued buffers.
> > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> > +  already queued buffers from the input or the output queue. This also
> > +  includes input or output buffers that can be currently owned by the
> > +  device's processing pipeline.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Asynchronous events}
> > +
> > +While processing buffers, the device can send asynchronous event
> > +notifications to the driver. The behaviour depends on the exact
> > +stream. For example, the decoder device sends a resolution change
> > +event when it encounters new resolution metadata in the stream.
> > +
> > +\subsubsection{Device Operation: Request header}
> > +
> > +All requests and responses on the control virt queue have a fixed
> > +header using the following layout structure and definitions:
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_ctrl_type {
> > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > +
> > +        /* request */
> > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > +
> > +        /* response */
> > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > +
> > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > +};
> > +
> > +struct virtio_video_ctrl_hdr {
> > +        le32 type;
> > +        le32 stream_id;
> > +        le32 len; /* Length of the structure in bytes. */
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{type}] is the type of the driver request or the device
> > +response.
> > +\item[\field{stream_id}] specifies a target stream.
> > +\item[\field{len}] is the length of data in bytes, which includes
> > +length of the header.
> > +\end{description}
> > +
> > +\subsubsection{Device Operation: controlq}
> > +
> > +\begin{description}
> > +
> > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > +supported formats.
> > +
> > +The driver uses \field{struct virtio_video_get_capability} to send a
> > +query request.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_buf_type {
> > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > +};
> 
> I personally didn't like the previous term: pin_type. But, to be honest, I
> don't like the buf_type neither. Consider the GET/SET_PARAMS request:
> buf_type there looks a bit unnatural. We are trying to get stream
> parameters there, not some parameters of whatever buffer. Also I don't see
> any strict reason to mimic v4l2 naming scheme.
> 
> I'd better rename it to PORT_TYPE or QUEUE_TYPE.
> 
> > +
> > +struct virtio_video_get_capability {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        enum virtio_video_buf_type buf_type;
> > +};
> > +\end{lstlisting}
> > +\begin{description}
> > +\item[\field{buf_type}] is the buffer type that the driver asks
> > +information about. The driver MUST set either
> > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > +
> > +The device responds a capability by using \field{struct
> > +virtio_video_get_capability_resp}.
> > +\begin{lstlisting}
> > +enum virtio_video_format {
> > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > +        /* Raw formats */
> > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > +        VIRTIO_VIDEO_FORMAT_YVU420,
> 
> Let's add some variants of RGB, like RGBA, ARGB. We need it for the encoder
> in particular .
> 
> > +
> > +        /* Compressed formats */
> > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> 
> Let's add H265, MPEG4, MPEG2. We already support and use them.
> 
> Regards,
> Dmitry.
> 
> > +};
> > +
> > +enum virtio_video_profile {
> > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > +
> > +        /* H.264 */
> > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > +        /* VP8 */
> > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > +
> > +        /* VP9 */
> > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > +        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > +};
> > +
> > +struct virtio_video_format_range {
> > +        le32 min;
> > +        le32 max;
> > +        le32 step;
> > +        u8 paddings[4];
> > +};
> > +
> > +struct virtio_video_format_desc {
> > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > +        le64 mask;
> > +        struct virtio_video_format_range width;
> > +        struct virtio_video_format_range height;
> > +        le32 num_rates;
> > +        u8 padding[4];
> > +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> > +};
> > +
> > +struct virtio_video_get_capability_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 num_descs;
> > +        /* Followed by struct virtio_video_format_desc desc[] */
> > +};
> > +\end{lstlisting}
> > +
We also see the need to add a max_streams value to this structure so as to 
explicitly provide a limit on the number of streams the guest can create.

Regards,
Dmitry.

> > +The format description \field{struct virtio_video_format_desc}
> > +includes the following fields:
> > +\begin{description}
> > +\item[\field{format}] specifies an image format. The device MUST set one
> > +  of \field{enum virtio_video_format}.
> > +\item[\field{profile}] specifies a profile of the compressed image format
> > +  specified in \field{format}. The driver SHOULD ignore this value if
> > +  \field{format} is a raw format.
> > +\item[\field{mask}] is a bitset that represents the supported
> > +  combination of input and output format. If \textit{i}-th bit is set
> > +  in \field{mask} of \textit{j}-th \field{struct
> > +  virtio_video_format_desc} for input, the device supports encoding or
> > +  decoding from the \textit{j}-th input format to \textit{i}-th output
> > +  format.
> > +\item[\field{width, height}] represents a range of resolutions
> > +  supported by the device. If its \field{step} is not applicable, its
> > +  \field{min} is equal to its \field{max}.
> > +\item[\field{num_rates}] is the length of an array \field{frame_rates}.
> > In
> > case of decoder, the driver SHOULD ignore this value.
> > +\item[\field{frame_rates}] is an array of supported frame rates.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_mem_type {
> > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > +};
> > +
> > +struct virtio_video_stream_create {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > +        char debug_name[64];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{in_mem_type}] is a type of buffer management for input
> > +buffers. The driver MUST set a value in \field{enum
> > +virtio_video_mem_type}.
> > +\item[\field{out_mem_type}] is a type of buffer management for output
> > +buffers. The driver MUST set a value in \field{enum
> > +virtio_video_mem_type}.
> > +\item[\field{debug_name}] is a text string for a debug purpose.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_stream_destroy {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > +  queued buffers through the pipeline.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_stream_drain {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_mem_entry {
> > +        le64 addr;
> > +        le32 length;
> > +        u8 padding[4];
> > +};
> > +
> > +struct virtio_video_resource_create {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 resource_id;
> > +        le32 nr_entries;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{resource_id}] internal id of the resource.
> > +\item[\field{nr_entries}] number of \field{struct
> > +  virtio_video_mem_entry} memory entries.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_resource_destroy {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 resource_id;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{resource_id}] internal id of the resource.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > +queue.
> > +
> > +\begin{lstlisting}
> > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > +
> > +struct virtio_video_resource_queue {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type;
> > +        le32 resource_id;
> > +        le64 timestamp;
> > +        le32 nr_data_size;
> > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buf_type of the .
> > +\item[\field{resource_id}] internal id of the resource.
> > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > +  for synchronisation.
> > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > +\item[\field{data_size}] number of data bytes within a plane.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_buffer_flag {
> > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > +        /* Encoder only */
> > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > +};
> > +
> > +struct virtio_video_resource_queue_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le64 timestamp;
> > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > +        le32 size;  /* Encoded size */
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > +  for synchronisation.
> > +\item[\field{flags}] mark specific buffers in the sequence.
> > +\item[\field{size}] data size in the buffer (encoder only).
> > +\end{description}
> > +
> > +The device sends a response to the queue request asynchronously when
> > +it has finished processing the buffer.
> > +
> > +The device SHOULD mark a buffer that triggered a processing error with
> > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > +
> > +The device MUST mark the last buffer with the
> > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > +sequence.
> > +
> > +In case of encoder, to denote a particular frame type the devie MUST
> > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> > +  buffers back from the input or the output queue of the device. The
> > +  device SHOULD return all of the buffers from the respective queue as
> > +  soon as possible without pushing the buffers through the processing
> > +  pipeline.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_queue_clear {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buffer type.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> > +  output of a stream.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_plane_format {
> > +        le32 plane_size;
> > +        le32 stride;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{plane_size}] size of the plane in bytes.
> > +\item[\field{stride}] stride used for the plane in bytes.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_params {
> > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> > +        le32 frame_width;
> > +        le32 frame_height;
> > +        le32 min_buffers;
> > +        le32 max_buffers;
> > +        le32 frame_rate;
> > +        struct virtio_video_crop {
> > +                le32 left;
> > +                le32 top;
> > +                le32 width;
> > +                le32 height;
> > +        } crop;
> > +        le32 num_planes;
> > +        struct virtio_video_plane_format
> > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{frame_width}] the value to get/set.
> > +\item[\field{frame_height}] the value to get/set.
> > +\item[\field{pixel_format}] the value to get/set.
> > +\item[\field{min_buffers}] minimum buffers required to handle the
> > +  format (r/o).
> > +\item[\field{max_buffers}] maximum buffers required to handle the
> > +  format (r/o).
> > +\item[\field{frame_rate}] the value to get/set.
> > +\item[\field{crop}] cropping (composing) rectangle.
> > +\item[\field{num_planes}] number of planes used to store pixel data
> > +(r/o).
> > +\item[\field{plane_formats}] description of each plane.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_get_params {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > +};
> > +
> > +struct virtio_video_get_params_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        struct virtio_video_params params;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buffer type.
> > +\item[\field{params}] parameter values.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> > +
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_set_params {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        struct virtio_video_params params;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{params}] parameters to set.
> > +\end{description}
> > +
> > +Setting stream parameters might have side effects within the device.
> > +For example, the device MAY perform alignment of width and height,
> > +change the number of planes it uses for the format, or do whatever
> > +changes that are required to continue normal operation using the
> > +updated parameters. It is up to the driver to check the parameter set
> > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > +
> > +\end{description}
> > +
> > +\subsubsection{Device Operation: eventq}
> > +
> > +The device can report events on the event queue. The driver initially
> > +populates the queue with device-writeable buffers. When the device
> > +needs to report an event, it fills a buffer and notifies the driver.
> > +The driver consumes the report and adds a new buffer to the virtqueue.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_event_type {
> > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > +        /* For all functions */
> > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > +        /* For decoder only */
> > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> > +};
> > +
> > +struct virtio_video_event {
> > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> > +        le32 stream_id;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{event_type}] type of the triggered event .
> > +\item[\field{stream_id}] id of the source stream.
> > +\end{description}
> > +
> > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > +whenever it encounters new resolution data in the stream. This
> > +includes the case of the initial device configuration after metadata
> > +has been parsed and the case of dynamic resolution change.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org



^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 0/1] VirtIO video device specification
  2019-12-21  6:19       ` Tomasz Figa
@ 2020-01-03 13:05         ` Dmitry Sepp
  2020-01-06 10:30           ` Keiichi Watanabe
  0 siblings, 1 reply; 73+ messages in thread
From: Dmitry Sepp @ 2020-01-03 13:05 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: Keiichi Watanabe, virtio-dev, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Enrico Granata, Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

Hi Tomasz, Keiichi,

On Samstag, 21. Dezember 2019 07:19:23 CET Tomasz Figa wrote:
> On Sat, Dec 21, 2019 at 3:18 PM Tomasz Figa <tfiga@chromium.org> wrote:
> > On Sat, Dec 21, 2019 at 1:36 PM Keiichi Watanabe <keiichiw@chromium.org> 
wrote:
> > > Hi Dmitry,
> > > 
> > > On Sat, Dec 21, 2019 at 12:59 AM Dmitry Sepp
> > > 
> > > <dmitry.sepp@opensynergy.com> wrote:
> > > > Hi Keiichi,
> > > > 
> > > > On Mittwoch, 18. Dezember 2019 14:02:13 CET Keiichi Watanabe wrote:
> > > > > Hi,
> > > > > This is the 2nd version of virtio-video patch. The PDF is available
> > > > > in [1].
> > > > > The first version was sent at [2].
> > > > > 
> > > > > Any feedback would be appreciated. Thank you.
> > > > > 
> > > > > Best,
> > > > > Keiichi
> > > > > 
> > > > > [1]:
> > > > > https://drive.google.com/drive/folders/1eT5fEckBoor2iHZR4f4GLxYzFMVa
> > > > > pOFx?us
> > > > > p=sharing [2]: https://markmail.org/message/gc6h25acct22niut
> > > > > 
> > > > > Change log:
> > > > > 
> > > > > v2:
> > > > > * Removed functionalities except encoding and decoding.
> > > > > * Splited encoder and decoder into different devices that use the
> > > > > same
> > > > > protocol. * Replaced GET_FUNCS with GET_CAPABILITY.
> > > > > * Updated structs for capabilities.
> > > > > 
> > > > >   - Defined new structs and enums such as image formats, profiles,
> > > > >   range
> > > > > 
> > > > > (min, max, step), etc
> > > > > 
> > > > >     * For virtio_video_pixel_format, chose a naming convention that
> > > > >     is used
> > > > >     
> > > > >       in DRM. We removed XBGR, NV21 and I422, as they are not used
> > > > >       in the
> > > > >       current draft implementation. https://lwn.net/Articles/806416/
> > > > >   
> > > > >   - Removed virtio_video_control, whose usage was not documented yet
> > > > >   and
> > > > > 
> > > > > which is not necessary for the simplest decoding scenario.
> > > > > 
> > > > >   - Removed virtio_video_desc, as it is no longer needed.
> > > > > 
> > > > > * Updated struct virtio_video_config for changes around
> > > > > capabilities.
> > > > > * Added a way to represent supported combinations of formats.
> > > > > 
> > > > >   - A field "mask" in virtio_video_format_desc plays this role.
> > > > > 
> > > > > * Removed VIRTIO_VIDEO_T_STREAM_{START,STOP} because they don't play
> > > > > any
> > > > > meaningful roles. * Removed VIRTIO_VIDEO_T_STREAM_{ATTACH,
> > > > > DETACH}_BACKING
> > > > > and merged them into RESOURCE_{CREATE, DESTROY}. * Added a way to
> > > > > notify/specify resource creation method.
> > > > > 
> > > > >   - Added a feature flag.
> > > > >   - Defined enum virtio_video_mem_type.
> > > > >   - Added new fields in video_stream_create.
> > > > > 
> > > > > * Modified fields in virtio_video_params.
> > > > > 
> > > > >   - Added crop information.
> > > > > 
> > > > > * Removed enum virtio_video_channel_type because we can get this
> > > > > information by image format.
> > > > 
> > > > Could you please explain this? How do you get the information?
> > > 
> > > It means that if image formats are well-defined, channel information
> > > (e.g. the order of channels) is uniquely determined.
> > > 
> > > > Suppose you have some piece of HW on the host side that wants I420 as
> > > > one
> > > > contig buffer w/ some offsets. But on the driver side, say, gralloc
> > > > gives you three separate buffers, one per channel. How do we pass
> > > > those to the device then?
> > > 
> > > You're talking about CrOS use case where buffers are allocated by
> > > virtio-gpu, right?
> > > In this case, virtio-gpu allocates one contiguous host-side buffer and
> > > the client regards a pair of (buffer FD, offset) as one channel.
> > > And, we can register this pair to the device when the buffer is
> > > imported.
> > > In the virtio-vdec spec draft, this pair corresponds to struct
> > > virtio_vdec_plane in struct virtio_vdec_plane.
> > > 
> > > So, I suppose we will need similar structs when we add a control to
> > > import buffers. However, I don't think it's necessary when guest pages
> > > are used.
> > 
> > I think we need some way for the guest to know whether it can allocate
> > the planes in separate buffers, even when guest pages are used. This
> > would be equivalent to V4L2 M and non-M formats, but mixing this into
> > FourCC in V4L2 is an acknowledged mistake, so we should add a query or
> > something.
> > 

Yes, this is what I mean. In fact, we already do face the situation when the 
device side is not happy with the sgt and wants contig. I think we'll add a 
module parameter for now.

Regards,
Dmitry.

> > For future V4L2 development we came up with the idea of a format flag
> > which could mean that the hardware allows putting planes in separate
> > buffers. We could have a similar per-format flag in the capabilities,
> > as we already have a list of all the supported formats there.
> 
> Sorry, forgot to paste the link from future V4L2 work notes from this year
> ELCE: https://www.spinics.net/lists/linux-media/msg159789.html
> 
> > Best regards,
> > Tomasz
> > 
> > > Best regards,
> > > Keiichi
> > > 
> > > > Best regards,
> > > > Dmitry.
> > > > 
> > > > > * Renamed virtio_video_pin to virtio_video_buf_type.
> > > > > 
> > > > >   - It's similar to V4L2_BUF_TYPE_VIDEO_{OUTPUT, CAPTURE}.
> > > > > 
> > > > > * Added an error event.
> > > > > * Reordered some subsections.
> > > > > * Changed styles to make it consistent with other devices.
> > > > > 
> > > > > Dmitry Sepp (1):
> > > > >   virtio-video: Add virtio video device specification
> > > > >  
> > > > >  content.tex      |   1 +
> > > > >  virtio-video.tex | 579
> > > > >  +++++++++++++++++++++++++++++++++++++++++++++++
> > > > >  2 files changed, 580 insertions(+)
> > > > >  create mode 100644 virtio-video.tex
> > > > > 
> > > > > --
> > > > > 2.24.1.735.g03f4e72817-goog



^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-18 13:02 ` [PATCH v2 1/1] virtio-video: Add virtio " Keiichi Watanabe
                     ` (2 preceding siblings ...)
  2019-12-19 13:28   ` Dmitry Sepp
@ 2020-01-03 15:47   ` Dmitry Sepp
  2020-01-06  8:47     ` Gerd Hoffmann
  2020-01-06 10:21     ` Keiichi Watanabe
  2020-01-06 14:59   ` Dmitry Sepp
  2020-01-08 12:23   ` Keiichi Watanabe
  5 siblings, 2 replies; 73+ messages in thread
From: Dmitry Sepp @ 2020-01-03 15:47 UTC (permalink / raw)
  To: Keiichi Watanabe
  Cc: virtio-dev, linux-media, acourbot, alexlau, daniel, dgreid,
	egranata, fziglio, hverkuil, kraxel, marcheu, posciak,
	spice-devel, stevensd, tfiga, uril

Hi Keiichi,

On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> 
> The virtio video encoder device and decoder device provide functionalities
> to encode and decode video stream respectively.
> Though video encoder and decoder are provided as different devices, they use
> a same protocol.
> 
> Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> ---
>  content.tex      |   1 +
>  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 580 insertions(+)
>  create mode 100644 virtio-video.tex
> 
> diff --git a/content.tex b/content.tex
> index 556b373..9e56839 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> Requirements}\label{sec:Device \input{virtio-vsock.tex}
>  \input{virtio-fs.tex}
>  \input{virtio-rpmb.tex}
> +\input{virtio-video.tex}
> 
>  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> 
> diff --git a/virtio-video.tex b/virtio-video.tex
> new file mode 100644
> index 0000000..30e728d
> --- /dev/null
> +++ b/virtio-video.tex
> @@ -0,0 +1,579 @@
> +\section{Video Device}\label{sec:Device Types / Video Device}
> +
> +The virtio video encoder device and decoder device are virtual devices that
> +supports encoding and decoding respectively. Though the encoder and the
> decoder +are different devices, they use the same protocol.
> +
> +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
> +
> +\begin{description}
> +\item[30] encoder device
> +\item[31] decoder device
> +\end{description}
> +
> +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues}
> +
> +\begin{description}
> +\item[0] controlq - queue for sending control commands.
> +\item[1] eventq - queue for sending events happened in the device.
> +\end{description}
> +
> +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature
> bits} +
> +\begin{description}
> +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for
> video +  buffers.
> +\end{description}
> +
> +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device
> / Feature bits} +
> +The device MUST offer at least one of feature bits.
> +
> +\subsection{Device configuration layout}\label{sec:Device Types / Video
> Device / Device configuration layout} +
> +Video device configuration uses the following layout structure:
> +
> +\begin{lstlisting}
> +struct virtio_video_config {
> +        le32 max_cap_len;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> +  MUST set this value.
> +\end{description}
> +
> +\subsection{Device Initialization}\label{sec:Device Types / Video Device /
> Device Initialization} +
> +\devicenormative{\subsubsection}{Device Initialization}{Device Types /
> Video Device / Device Initialization} +
> +The driver SHOULD query device capability by using the
> +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> +setup.
> +
> +\subsection{Device Operation}\label{sec:Device Types / Video Device /
> Device Operation} +
> +The driver allocates input and output buffers and queues the buffers
> +to the device. The device performs operations on the buffers according
> +to the function in question.
> +
> +\subsubsection{Device Operation: Create stream}
> +
> +To process buffers, the device needs to associate them with a certain
> +video stream (essentially, a context). Streams are created by
> +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> +determined by the device.
> +
> +\subsubsection{Device Operation: Create buffers}
> +
> +Buffers are used to store the actual data as well as the relevant
> +metadata. Scatter lists are supported, so the buffer doesn't need to
> +be contiguous in guest physical memory.
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> +  resource that is backed by a buffer allocated from the driver's
> +  memory.
> +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> +  is no longer needed.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Stream parameter control}
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters
> for +  input and output streams from the device.
> +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the
> +  device.
> +\item After setting stream parameters, the driver may issue
> +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can
> be +  changed implicitly by the device during the set operation.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Process buffers}
> +
> +\begin{itemize*}
> +\item If the function and the buffer type require so, write data to
> +the buffer memory.
> +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> +processing in the device.
> +\item The request completes asynchronously when the device has
> +finished with the buffer.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Buffer processing control}
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> +  return all of the already queued buffers.
> +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> +  already queued buffers from the input or the output queue. This also
> +  includes input or output buffers that can be currently owned by the
> +  device's processing pipeline.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Asynchronous events}
> +
> +While processing buffers, the device can send asynchronous event
> +notifications to the driver. The behaviour depends on the exact
> +stream. For example, the decoder device sends a resolution change
> +event when it encounters new resolution metadata in the stream.
> +
> +\subsubsection{Device Operation: Request header}
> +
> +All requests and responses on the control virt queue have a fixed
> +header using the following layout structure and definitions:
> +
> +\begin{lstlisting}
> +enum virtio_video_ctrl_type {
> +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> +
> +        /* request */
> +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> +        VIRTIO_VIDEO_T_STREAM_CREATE,
> +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> +        VIRTIO_VIDEO_T_SET_PARAMS,
> +        VIRTIO_VIDEO_T_GET_PARAMS,
> +
> +        /* response */
> +        VIRTIO_VIDEO_S_OK = 0x0200,
> +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> +
> +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> +};
> +
> +struct virtio_video_ctrl_hdr {
> +        le32 type;
> +        le32 stream_id;
> +        le32 len; /* Length of the structure in bytes. */
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{type}] is the type of the driver request or the device
> +response.
> +\item[\field{stream_id}] specifies a target stream.
> +\item[\field{len}] is the length of data in bytes, which includes
> +length of the header.
> +\end{description}
> +
> +\subsubsection{Device Operation: controlq}
> +
> +\begin{description}
> +
> +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> +supported formats.
> +
> +The driver uses \field{struct virtio_video_get_capability} to send a
> +query request.
> +
> +\begin{lstlisting}
> +enum virtio_video_buf_type {
> +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> +};
> +
> +struct virtio_video_get_capability {
> +        struct virtio_video_ctrl_hdr hdr;
> +        enum virtio_video_buf_type buf_type;
> +};
> +\end{lstlisting}
> +\begin{description}
> +\item[\field{buf_type}] is the buffer type that the driver asks
> +information about. The driver MUST set either
> +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> +
> +The device responds a capability by using \field{struct
> +virtio_video_get_capability_resp}.
> +\begin{lstlisting}
> +enum virtio_video_format {
> +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> +        /* Raw formats */
> +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> +        VIRTIO_VIDEO_FORMAT_YUV420,
> +        VIRTIO_VIDEO_FORMAT_YVU420,
> +
> +        /* Compressed formats */
> +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> +};
> +
> +enum virtio_video_profile {
> +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> +
> +        /* H.264 */
> +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> VIRTIO_VIDEO_PROFILE_H264_BASELINE, +       
> VIRTIO_VIDEO_PROFILE_H264_MAIN,
> +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> +        /* VP8 */
> +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> +
> +        /* VP9 */
> +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> +        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> +};
> +
> +struct virtio_video_format_range {
> +        le32 min;
> +        le32 max;
> +        le32 step;
> +        u8 paddings[4];
> +};
> +
> +struct virtio_video_format_desc {
> +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> +        le64 mask;
> +        struct virtio_video_format_range width;
> +        struct virtio_video_format_range height;
> +        le32 num_rates;
> +        u8 padding[4];
> +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> +};
> +
> +struct virtio_video_get_capability_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 num_descs;
> +        /* Followed by struct virtio_video_format_desc desc[] */
> +};
> +\end{lstlisting}
> +
> +The format description \field{struct virtio_video_format_desc}
> +includes the following fields:
> +\begin{description}
> +\item[\field{format}] specifies an image format. The device MUST set one
> +  of \field{enum virtio_video_format}.
> +\item[\field{profile}] specifies a profile of the compressed image format
> +  specified in \field{format}. The driver SHOULD ignore this value if
> +  \field{format} is a raw format.
> +\item[\field{mask}] is a bitset that represents the supported
> +  combination of input and output format. If \textit{i}-th bit is set
> +  in \field{mask} of \textit{j}-th \field{struct
> +  virtio_video_format_desc} for input, the device supports encoding or
> +  decoding from the \textit{j}-th input format to \textit{i}-th output
> +  format.
> +\item[\field{width, height}] represents a range of resolutions
> +  supported by the device. If its \field{step} is not applicable, its
> +  \field{min} is equal to its \field{max}.
> +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In
> case of decoder, the driver SHOULD ignore this value.
> +\item[\field{frame_rates}] is an array of supported frame rates.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> +  within the device.
> +
> +\begin{lstlisting}
> +enum virtio_video_mem_type {
> +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> +};
> +
> +struct virtio_video_stream_create {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> +        char debug_name[64];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{in_mem_type}] is a type of buffer management for input
> +buffers. The driver MUST set a value in \field{enum
> +virtio_video_mem_type}.
> +\item[\field{out_mem_type}] is a type of buffer management for output
> +buffers. The driver MUST set a value in \field{enum
> +virtio_video_mem_type}.
> +\item[\field{debug_name}] is a text string for a debug purpose.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_stream_destroy {
> +        struct virtio_video_ctrl_hdr hdr;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> +  queued buffers through the pipeline.
> +
> +\begin{lstlisting}
> +struct virtio_video_stream_drain {
> +        struct virtio_video_ctrl_hdr hdr;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_mem_entry {
> +        le64 addr;
> +        le32 length;
> +        u8 padding[4];
> +};
> +
> +struct virtio_video_resource_create {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 resource_id;
> +        le32 nr_entries;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{resource_id}] internal id of the resource.
> +\item[\field{nr_entries}] number of \field{struct
> +  virtio_video_mem_entry} memory entries.
> +\end{description}
> +

How should one deal with multiplanar formats? Do we create one resource per 
plane? Otherwise we need a way to send mem entries for each plane in one 
request.

Currently the v1 driver attaches backing to the same resource multiple times 
when a buffer has multiple planes. Not the nicest way to be honest.

Regards,
Dmitry.

> +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_resource_destroy {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 resource_id;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{resource_id}] internal id of the resource.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> +queue.
> +
> +\begin{lstlisting}
> +#define VIRTIO_VIDEO_MAX_PLANES 8
> +
> +struct virtio_video_resource_queue {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type;
> +        le32 resource_id;
> +        le64 timestamp;
> +        le32 nr_data_size;
> +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buf_type of the .
> +\item[\field{resource_id}] internal id of the resource.
> +\item[\field{timestamp}] an abstract sequence counter that can be used
> +  for synchronisation.
> +\item[\field{nr_data_size}] number of \field{data_size} entries.
> +\item[\field{data_size}] number of data bytes within a plane.
> +\end{description}
> +
> +\begin{lstlisting}
> +enum virtio_video_buffer_flag {
> +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> +        /* Encoder only */
> +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> +};
> +
> +struct virtio_video_resource_queue_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le64 timestamp;
> +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> +        le32 size;  /* Encoded size */
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{timestamp}] an abstract sequence counter that can be used
> +  for synchronisation.
> +\item[\field{flags}] mark specific buffers in the sequence.
> +\item[\field{size}] data size in the buffer (encoder only).
> +\end{description}
> +
> +The device sends a response to the queue request asynchronously when
> +it has finished processing the buffer.
> +
> +The device SHOULD mark a buffer that triggered a processing error with
> +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> +
> +The device MUST mark the last buffer with the
> +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> +sequence.
> +
> +In case of encoder, to denote a particular frame type the devie MUST
> +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> +  buffers back from the input or the output queue of the device. The
> +  device SHOULD return all of the buffers from the respective queue as
> +  soon as possible without pushing the buffers through the processing
> +  pipeline.
> +
> +\begin{lstlisting}
> +struct virtio_video_queue_clear {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buffer type.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> +  output of a stream.
> +
> +\begin{lstlisting}
> +struct virtio_video_plane_format {
> +        le32 plane_size;
> +        le32 stride;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{plane_size}] size of the plane in bytes.
> +\item[\field{stride}] stride used for the plane in bytes.
> +\end{description}
> +
> +\begin{lstlisting}
> +struct virtio_video_params {
> +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> +        le32 frame_width;
> +        le32 frame_height;
> +        le32 min_buffers;
> +        le32 max_buffers;
> +        le32 frame_rate;
> +        struct virtio_video_crop {
> +                le32 left;
> +                le32 top;
> +                le32 width;
> +                le32 height;
> +        } crop;
> +        le32 num_planes;
> +        struct virtio_video_plane_format
> plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{frame_width}] the value to get/set.
> +\item[\field{frame_height}] the value to get/set.
> +\item[\field{pixel_format}] the value to get/set.
> +\item[\field{min_buffers}] minimum buffers required to handle the
> +  format (r/o).
> +\item[\field{max_buffers}] maximum buffers required to handle the
> +  format (r/o).
> +\item[\field{frame_rate}] the value to get/set.
> +\item[\field{crop}] cropping (composing) rectangle.
> +\item[\field{num_planes}] number of planes used to store pixel data
> +(r/o).
> +\item[\field{plane_formats}] description of each plane.
> +\end{description}
> +
> +\begin{lstlisting}
> +struct virtio_video_get_params {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> +};
> +
> +struct virtio_video_get_params_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        struct virtio_video_params params;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buffer type.
> +\item[\field{params}] parameter values.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> +
> +
> +\begin{lstlisting}
> +struct virtio_video_set_params {
> +        struct virtio_video_ctrl_hdr hdr;
> +        struct virtio_video_params params;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{params}] parameters to set.
> +\end{description}
> +
> +Setting stream parameters might have side effects within the device.
> +For example, the device MAY perform alignment of width and height,
> +change the number of planes it uses for the format, or do whatever
> +changes that are required to continue normal operation using the
> +updated parameters. It is up to the driver to check the parameter set
> +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> +
> +\end{description}
> +
> +\subsubsection{Device Operation: eventq}
> +
> +The device can report events on the event queue. The driver initially
> +populates the queue with device-writeable buffers. When the device
> +needs to report an event, it fills a buffer and notifies the driver.
> +The driver consumes the report and adds a new buffer to the virtqueue.
> +
> +\begin{lstlisting}
> +enum virtio_video_event_type {
> +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> +        /* For all functions */
> +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> +        /* For decoder only */
> +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> +};
> +
> +struct virtio_video_event {
> +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> +        le32 stream_id;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{event_type}] type of the triggered event .
> +\item[\field{stream_id}] id of the source stream.
> +\end{description}
> +
> +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> +whenever it encounters new resolution data in the stream. This
> +includes the case of the initial device configuration after metadata
> +has been parsed and the case of dynamic resolution change.



^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [virtio-dev] Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-30 12:16     ` [virtio-dev] " Dmitry Sepp
@ 2020-01-06  6:31       ` Tomasz Figa
  2020-01-06  8:33         ` Gerd Hoffmann
  0 siblings, 1 reply; 73+ messages in thread
From: Tomasz Figa @ 2020-01-06  6:31 UTC (permalink / raw)
  To: Dmitry Sepp
  Cc: virtio-dev, Keiichi Watanabe, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Enrico Granata, Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

On Mon, Dec 30, 2019 at 9:16 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
>
>
> On Donnerstag, 19. Dezember 2019 14:28:23 CET Dmitry Sepp wrote:
> > Hi Keiichi,
> >
> > Thank you for the update. Please see some comments below.
> >
> > Also, we need to bring the virtio_video_control back as it is in fact used
> > by the driver to enumerate supported encoder controls. But yes, it still
> > needs to be documemnted, it's true.
> >
> > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > >
> > > The virtio video encoder device and decoder device provide functionalities
> > > to encode and decode video stream respectively.
> > > Though video encoder and decoder are provided as different devices, they
> > > use a same protocol.
> > >
> > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > > ---
> > >
> > >  content.tex      |   1 +
> > >  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
> > >  2 files changed, 580 insertions(+)
> > >  create mode 100644 virtio-video.tex
> > >
> > > diff --git a/content.tex b/content.tex
> > > index 556b373..9e56839 100644
> > > --- a/content.tex
> > > +++ b/content.tex
> > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> > >
> > >  \input{virtio-fs.tex}
> > >  \input{virtio-rpmb.tex}
> > >
> > > +\input{virtio-video.tex}
> > >
> > >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> > >
> > > diff --git a/virtio-video.tex b/virtio-video.tex
> > > new file mode 100644
> > > index 0000000..30e728d
> > > --- /dev/null
> > > +++ b/virtio-video.tex
> > > @@ -0,0 +1,579 @@
> > > +\section{Video Device}\label{sec:Device Types / Video Device}
> > > +
> > > +The virtio video encoder device and decoder device are virtual devices
> > > that +supports encoding and decoding respectively. Though the encoder and
> > > the decoder +are different devices, they use the same protocol.
> > > +
> > > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
> > > +
> > > +\begin{description}
> > > +\item[30] encoder device
> > > +\item[31] decoder device
> > > +\end{description}
> > > +
> > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device /
> > > Virtqueues} +
> > > +\begin{description}
> > > +\item[0] controlq - queue for sending control commands.
> > > +\item[1] eventq - queue for sending events happened in the device.
> > > +\end{description}
> > > +
> > > +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature
> > > bits} +
> > > +\begin{description}
> > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used
> > > for
> > > video +  buffers.
> > > +\end{description}
> > > +
> > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video
> > > Device
> > > / Feature bits} +
> > > +The device MUST offer at least one of feature bits.
> > > +
> > > +\subsection{Device configuration layout}\label{sec:Device Types / Video
> > > Device / Device configuration layout} +
> > > +Video device configuration uses the following layout structure:
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_config {
> > > +        le32 max_cap_len;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> > > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > > +  MUST set this value.
> > > +\end{description}
> > > +
> > > +\subsection{Device Initialization}\label{sec:Device Types / Video Device
> > > /
> > > Device Initialization} +
> > > +\devicenormative{\subsubsection}{Device Initialization}{Device Types /
> > > Video Device / Device Initialization} +
> > > +The driver SHOULD query device capability by using the
> > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> > > +setup.
> > > +
> > > +\subsection{Device Operation}\label{sec:Device Types / Video Device /
> > > Device Operation} +
> > > +The driver allocates input and output buffers and queues the buffers
> > > +to the device. The device performs operations on the buffers according
> > > +to the function in question.
> > > +
> > > +\subsubsection{Device Operation: Create stream}
> > > +
> > > +To process buffers, the device needs to associate them with a certain
> > > +video stream (essentially, a context). Streams are created by
> > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > > +determined by the device.
> > > +
> > > +\subsubsection{Device Operation: Create buffers}
> > > +
> > > +Buffers are used to store the actual data as well as the relevant
> > > +metadata. Scatter lists are supported, so the buffer doesn't need to
> > > +be contiguous in guest physical memory.
> > > +
> > > +\begin{itemize*}
> > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > > +  resource that is backed by a buffer allocated from the driver's
> > > +  memory.
> > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> > > +  is no longer needed.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Stream parameter control}
> > > +
> > > +\begin{itemize*}
> > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters
> > > for +  input and output streams from the device.
> > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to
> > > the +  device.
> > > +\item After setting stream parameters, the driver may issue
> > > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output
> > > can be +  changed implicitly by the device during the set operation.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Process buffers}
> > > +
> > > +\begin{itemize*}
> > > +\item If the function and the buffer type require so, write data to
> > > +the buffer memory.
> > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > > +processing in the device.
> > > +\item The request completes asynchronously when the device has
> > > +finished with the buffer.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Buffer processing control}
> > > +
> > > +\begin{itemize*}
> > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> > > +  return all of the already queued buffers.
> > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> > > +  already queued buffers from the input or the output queue. This also
> > > +  includes input or output buffers that can be currently owned by the
> > > +  device's processing pipeline.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Asynchronous events}
> > > +
> > > +While processing buffers, the device can send asynchronous event
> > > +notifications to the driver. The behaviour depends on the exact
> > > +stream. For example, the decoder device sends a resolution change
> > > +event when it encounters new resolution metadata in the stream.
> > > +
> > > +\subsubsection{Device Operation: Request header}
> > > +
> > > +All requests and responses on the control virt queue have a fixed
> > > +header using the following layout structure and definitions:
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_ctrl_type {
> > > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > > +
> > > +        /* request */
> > > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > > +
> > > +        /* response */
> > > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > > +
> > > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > > +};
> > > +
> > > +struct virtio_video_ctrl_hdr {
> > > +        le32 type;
> > > +        le32 stream_id;
> > > +        le32 len; /* Length of the structure in bytes. */
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{type}] is the type of the driver request or the device
> > > +response.
> > > +\item[\field{stream_id}] specifies a target stream.
> > > +\item[\field{len}] is the length of data in bytes, which includes
> > > +length of the header.
> > > +\end{description}
> > > +
> > > +\subsubsection{Device Operation: controlq}
> > > +
> > > +\begin{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > > +supported formats.
> > > +
> > > +The driver uses \field{struct virtio_video_get_capability} to send a
> > > +query request.
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_buf_type {
> > > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > > +};
> >
> > I personally didn't like the previous term: pin_type. But, to be honest, I
> > don't like the buf_type neither. Consider the GET/SET_PARAMS request:
> > buf_type there looks a bit unnatural. We are trying to get stream
> > parameters there, not some parameters of whatever buffer. Also I don't see
> > any strict reason to mimic v4l2 naming scheme.
> >
> > I'd better rename it to PORT_TYPE or QUEUE_TYPE.
> >
> > > +
> > > +struct virtio_video_get_capability {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        enum virtio_video_buf_type buf_type;
> > > +};
> > > +\end{lstlisting}
> > > +\begin{description}
> > > +\item[\field{buf_type}] is the buffer type that the driver asks
> > > +information about. The driver MUST set either
> > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > > +
> > > +The device responds a capability by using \field{struct
> > > +virtio_video_get_capability_resp}.
> > > +\begin{lstlisting}
> > > +enum virtio_video_format {
> > > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > > +        /* Raw formats */
> > > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > > +        VIRTIO_VIDEO_FORMAT_YVU420,
> >
> > Let's add some variants of RGB, like RGBA, ARGB. We need it for the encoder
> > in particular .
> >
> > > +
> > > +        /* Compressed formats */
> > > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> >
> > Let's add H265, MPEG4, MPEG2. We already support and use them.
> >
> > Regards,
> > Dmitry.
> >
> > > +};
> > > +
> > > +enum virtio_video_profile {
> > > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > > +
> > > +        /* H.264 */
> > > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > > +        /* VP8 */
> > > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > > +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > > +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > > +
> > > +        /* VP9 */
> > > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > +};
> > > +
> > > +struct virtio_video_format_range {
> > > +        le32 min;
> > > +        le32 max;
> > > +        le32 step;
> > > +        u8 paddings[4];
> > > +};
> > > +
> > > +struct virtio_video_format_desc {
> > > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > > +        le64 mask;
> > > +        struct virtio_video_format_range width;
> > > +        struct virtio_video_format_range height;
> > > +        le32 num_rates;
> > > +        u8 padding[4];
> > > +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> > > +};
> > > +
> > > +struct virtio_video_get_capability_resp {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 num_descs;
> > > +        /* Followed by struct virtio_video_format_desc desc[] */
> > > +};
> > > +\end{lstlisting}
> > > +
> We also see the need to add a max_streams value to this structure so as to
> explicitly provide a limit on the number of streams the guest can create.

What would be the advantage over just trying to create one and
failing? The maximum number would be only meaningful for the special
case when the streams are always only created by one user space
process. Otherwise, if several processes do that, they are not aware
of each other and the number could be higher than they can actually
create, because other processes could have some streams created
already.

Best regards,
Tomasz

>
> Regards,
> Dmitry.
>
> > > +The format description \field{struct virtio_video_format_desc}
> > > +includes the following fields:
> > > +\begin{description}
> > > +\item[\field{format}] specifies an image format. The device MUST set one
> > > +  of \field{enum virtio_video_format}.
> > > +\item[\field{profile}] specifies a profile of the compressed image format
> > > +  specified in \field{format}. The driver SHOULD ignore this value if
> > > +  \field{format} is a raw format.
> > > +\item[\field{mask}] is a bitset that represents the supported
> > > +  combination of input and output format. If \textit{i}-th bit is set
> > > +  in \field{mask} of \textit{j}-th \field{struct
> > > +  virtio_video_format_desc} for input, the device supports encoding or
> > > +  decoding from the \textit{j}-th input format to \textit{i}-th output
> > > +  format.
> > > +\item[\field{width, height}] represents a range of resolutions
> > > +  supported by the device. If its \field{step} is not applicable, its
> > > +  \field{min} is equal to its \field{max}.
> > > +\item[\field{num_rates}] is the length of an array \field{frame_rates}.
> > > In
> > > case of decoder, the driver SHOULD ignore this value.
> > > +\item[\field{frame_rates}] is an array of supported frame rates.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_mem_type {
> > > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > > +};
> > > +
> > > +struct virtio_video_stream_create {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > > +        char debug_name[64];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{in_mem_type}] is a type of buffer management for input
> > > +buffers. The driver MUST set a value in \field{enum
> > > +virtio_video_mem_type}.
> > > +\item[\field{out_mem_type}] is a type of buffer management for output
> > > +buffers. The driver MUST set a value in \field{enum
> > > +virtio_video_mem_type}.
> > > +\item[\field{debug_name}] is a text string for a debug purpose.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_stream_destroy {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > > +  queued buffers through the pipeline.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_stream_drain {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_mem_entry {
> > > +        le64 addr;
> > > +        le32 length;
> > > +        u8 padding[4];
> > > +};
> > > +
> > > +struct virtio_video_resource_create {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 resource_id;
> > > +        le32 nr_entries;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{resource_id}] internal id of the resource.
> > > +\item[\field{nr_entries}] number of \field{struct
> > > +  virtio_video_mem_entry} memory entries.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_resource_destroy {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 resource_id;
> > > +        u8 padding[4];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{resource_id}] internal id of the resource.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > > +queue.
> > > +
> > > +\begin{lstlisting}
> > > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > > +
> > > +struct virtio_video_resource_queue {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 buf_type;
> > > +        le32 resource_id;
> > > +        le64 timestamp;
> > > +        le32 nr_data_size;
> > > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{buf_type}] buf_type of the .
> > > +\item[\field{resource_id}] internal id of the resource.
> > > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > > +  for synchronisation.
> > > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > > +\item[\field{data_size}] number of data bytes within a plane.
> > > +\end{description}
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_buffer_flag {
> > > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > > +        /* Encoder only */
> > > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > > +};
> > > +
> > > +struct virtio_video_resource_queue_resp {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le64 timestamp;
> > > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > > +        le32 size;  /* Encoded size */
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > > +  for synchronisation.
> > > +\item[\field{flags}] mark specific buffers in the sequence.
> > > +\item[\field{size}] data size in the buffer (encoder only).
> > > +\end{description}
> > > +
> > > +The device sends a response to the queue request asynchronously when
> > > +it has finished processing the buffer.
> > > +
> > > +The device SHOULD mark a buffer that triggered a processing error with
> > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > > +
> > > +The device MUST mark the last buffer with the
> > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > +sequence.
> > > +
> > > +In case of encoder, to denote a particular frame type the devie MUST
> > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > > +
> > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> > > +  buffers back from the input or the output queue of the device. The
> > > +  device SHOULD return all of the buffers from the respective queue as
> > > +  soon as possible without pushing the buffers through the processing
> > > +  pipeline.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_queue_clear {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 buf_type;
> > > +        u8 padding[4];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{buf_type}] buffer type.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> > > +  output of a stream.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_plane_format {
> > > +        le32 plane_size;
> > > +        le32 stride;
> > > +        u8 padding[4];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{plane_size}] size of the plane in bytes.
> > > +\item[\field{stride}] stride used for the plane in bytes.
> > > +\end{description}
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_params {
> > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> > > +        le32 frame_width;
> > > +        le32 frame_height;
> > > +        le32 min_buffers;
> > > +        le32 max_buffers;
> > > +        le32 frame_rate;
> > > +        struct virtio_video_crop {
> > > +                le32 left;
> > > +                le32 top;
> > > +                le32 width;
> > > +                le32 height;
> > > +        } crop;
> > > +        le32 num_planes;
> > > +        struct virtio_video_plane_format
> > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{frame_width}] the value to get/set.
> > > +\item[\field{frame_height}] the value to get/set.
> > > +\item[\field{pixel_format}] the value to get/set.
> > > +\item[\field{min_buffers}] minimum buffers required to handle the
> > > +  format (r/o).
> > > +\item[\field{max_buffers}] maximum buffers required to handle the
> > > +  format (r/o).
> > > +\item[\field{frame_rate}] the value to get/set.
> > > +\item[\field{crop}] cropping (composing) rectangle.
> > > +\item[\field{num_planes}] number of planes used to store pixel data
> > > +(r/o).
> > > +\item[\field{plane_formats}] description of each plane.
> > > +\end{description}
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_get_params {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > +};
> > > +
> > > +struct virtio_video_get_params_resp {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        struct virtio_video_params params;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{buf_type}] buffer type.
> > > +\item[\field{params}] parameter values.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> > > +
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_set_params {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        struct virtio_video_params params;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{params}] parameters to set.
> > > +\end{description}
> > > +
> > > +Setting stream parameters might have side effects within the device.
> > > +For example, the device MAY perform alignment of width and height,
> > > +change the number of planes it uses for the format, or do whatever
> > > +changes that are required to continue normal operation using the
> > > +updated parameters. It is up to the driver to check the parameter set
> > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > > +
> > > +\end{description}
> > > +
> > > +\subsubsection{Device Operation: eventq}
> > > +
> > > +The device can report events on the event queue. The driver initially
> > > +populates the queue with device-writeable buffers. When the device
> > > +needs to report an event, it fills a buffer and notifies the driver.
> > > +The driver consumes the report and adds a new buffer to the virtqueue.
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_event_type {
> > > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > > +        /* For all functions */
> > > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > > +        /* For decoder only */
> > > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> > > +};
> > > +
> > > +struct virtio_video_event {
> > > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> > > +        le32 stream_id;
> > > +        u8 padding[4];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{event_type}] type of the triggered event .
> > > +\item[\field{stream_id}] id of the source stream.
> > > +\end{description}
> > > +
> > > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > > +whenever it encounters new resolution data in the stream. This
> > > +includes the case of the initial device configuration after metadata
> > > +has been parsed and the case of dynamic resolution change.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>
>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [virtio-dev] Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2020-01-06  6:31       ` Tomasz Figa
@ 2020-01-06  8:33         ` Gerd Hoffmann
  2020-01-06  9:29           ` Dmitry Sepp
  0 siblings, 1 reply; 73+ messages in thread
From: Gerd Hoffmann @ 2020-01-06  8:33 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: Dmitry Sepp, virtio-dev, Keiichi Watanabe,
	Linux Media Mailing List, Alexandre Courbot, Alex Lau,
	Daniel Vetter, Dylan Reid, Enrico Granata, Frediano Ziglio,
	Hans Verkuil, Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

  Hi,

> > We also see the need to add a max_streams value to this structure so as to
> > explicitly provide a limit on the number of streams the guest can create.
> 
> What would be the advantage over just trying to create one and
> failing? The maximum number would be only meaningful for the special
> case when the streams are always only created by one user space
> process. Otherwise, if several processes do that, they are not aware
> of each other and the number could be higher than they can actually
> create, because other processes could have some streams created
> already.

Also the number of streams might not be fixed but depend on stream
parameters, i.e. hardware can decode one hd or two sd streams ...

cheers,
  Gerd


^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2020-01-03 15:47   ` Dmitry Sepp
@ 2020-01-06  8:47     ` Gerd Hoffmann
  2020-01-06 10:21     ` Keiichi Watanabe
  1 sibling, 0 replies; 73+ messages in thread
From: Gerd Hoffmann @ 2020-01-06  8:47 UTC (permalink / raw)
  To: Dmitry Sepp
  Cc: Keiichi Watanabe, virtio-dev, linux-media, acourbot, alexlau,
	daniel, dgreid, egranata, fziglio, hverkuil, marcheu, posciak,
	spice-devel, stevensd, tfiga, uril

  Hi,

> How should one deal with multiplanar formats? Do we create one resource per 
> plane? Otherwise we need a way to send mem entries for each plane in one 
> request.

DRM uses arrays of handles and offsets (see struct drm_framebuffer).  A
handle references a gem object (roughly the same as a resource), and the
offset specifies the start of the plane within the gem object.  That
allows both a single gem object with planes stored at different offsets
and one gem object per plane.  virtio-video could do the same, or pick
one of the two approaches and support only that.

cheers,
  Gerd


^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [virtio-dev] Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2020-01-06  8:33         ` Gerd Hoffmann
@ 2020-01-06  9:29           ` Dmitry Sepp
  0 siblings, 0 replies; 73+ messages in thread
From: Dmitry Sepp @ 2020-01-06  9:29 UTC (permalink / raw)
  To: Gerd Hoffmann
  Cc: Tomasz Figa, virtio-dev, Keiichi Watanabe,
	Linux Media Mailing List, Alexandre Courbot, Alex Lau,
	Daniel Vetter, Dylan Reid, Enrico Granata, Frediano Ziglio,
	Hans Verkuil, Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

Hi,

On Montag, 6. Januar 2020 09:33:35 CET Gerd Hoffmann wrote:
>   Hi,
> 
> > > We also see the need to add a max_streams value to this structure so as
> > > to
> > > explicitly provide a limit on the number of streams the guest can
> > > create.
> > 
> > What would be the advantage over just trying to create one and
> > failing? The maximum number would be only meaningful for the special
> > case when the streams are always only created by one user space
> > process. Otherwise, if several processes do that, they are not aware
> > of each other and the number could be higher than they can actually
> > create, because other processes could have some streams created
> > already.
> 
> Also the number of streams might not be fixed but depend on stream
> parameters, i.e. hardware can decode one hd or two sd streams ...
> 
Ok, you are right. We'd better return an error from the device side.

Regards,
Dmitry.

> cheers,
>   Gerd



^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2020-01-03 15:47   ` Dmitry Sepp
  2020-01-06  8:47     ` Gerd Hoffmann
@ 2020-01-06 10:21     ` Keiichi Watanabe
  1 sibling, 0 replies; 73+ messages in thread
From: Keiichi Watanabe @ 2020-01-06 10:21 UTC (permalink / raw)
  To: Dmitry Sepp
  Cc: virtio-dev, Linux Media Mailing List, Alexandre Courbot,
	Alex Lau, Daniel Vetter, Dylan Reid, Enrico Granata,
	Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, Tomasz Figa, uril

Hi Dmitry,

On Sat, Jan 4, 2020 at 12:47 AM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
>
> Hi Keiichi,
>
> On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> >
> > The virtio video encoder device and decoder device provide functionalities
> > to encode and decode video stream respectively.
> > Though video encoder and decoder are provided as different devices, they use
> > a same protocol.
> >
> > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > ---
> >  content.tex      |   1 +
> >  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 580 insertions(+)
> >  create mode 100644 virtio-video.tex
> >
> > diff --git a/content.tex b/content.tex
> > index 556b373..9e56839 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> >  \input{virtio-fs.tex}
> >  \input{virtio-rpmb.tex}
> > +\input{virtio-video.tex}
> >
> >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> >
> > diff --git a/virtio-video.tex b/virtio-video.tex
> > new file mode 100644
> > index 0000000..30e728d
> > --- /dev/null
> > +++ b/virtio-video.tex
> > @@ -0,0 +1,579 @@
> > +\section{Video Device}\label{sec:Device Types / Video Device}
> > +
> > +The virtio video encoder device and decoder device are virtual devices that
> > +supports encoding and decoding respectively. Though the encoder and the
> > decoder +are different devices, they use the same protocol.
> > +
> > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
> > +
> > +\begin{description}
> > +\item[30] encoder device
> > +\item[31] decoder device
> > +\end{description}
> > +
> > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues}
> > +
> > +\begin{description}
> > +\item[0] controlq - queue for sending control commands.
> > +\item[1] eventq - queue for sending events happened in the device.
> > +\end{description}
> > +
> > +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature
> > bits} +
> > +\begin{description}
> > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for
> > video +  buffers.
> > +\end{description}
> > +
> > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device
> > / Feature bits} +
> > +The device MUST offer at least one of feature bits.
> > +
> > +\subsection{Device configuration layout}\label{sec:Device Types / Video
> > Device / Device configuration layout} +
> > +Video device configuration uses the following layout structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_config {
> > +        le32 max_cap_len;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > +  MUST set this value.
> > +\end{description}
> > +
> > +\subsection{Device Initialization}\label{sec:Device Types / Video Device /
> > Device Initialization} +
> > +\devicenormative{\subsubsection}{Device Initialization}{Device Types /
> > Video Device / Device Initialization} +
> > +The driver SHOULD query device capability by using the
> > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> > +setup.
> > +
> > +\subsection{Device Operation}\label{sec:Device Types / Video Device /
> > Device Operation} +
> > +The driver allocates input and output buffers and queues the buffers
> > +to the device. The device performs operations on the buffers according
> > +to the function in question.
> > +
> > +\subsubsection{Device Operation: Create stream}
> > +
> > +To process buffers, the device needs to associate them with a certain
> > +video stream (essentially, a context). Streams are created by
> > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > +determined by the device.
> > +
> > +\subsubsection{Device Operation: Create buffers}
> > +
> > +Buffers are used to store the actual data as well as the relevant
> > +metadata. Scatter lists are supported, so the buffer doesn't need to
> > +be contiguous in guest physical memory.
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > +  resource that is backed by a buffer allocated from the driver's
> > +  memory.
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> > +  is no longer needed.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Stream parameter control}
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters
> > for +  input and output streams from the device.
> > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the
> > +  device.
> > +\item After setting stream parameters, the driver may issue
> > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can
> > be +  changed implicitly by the device during the set operation.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Process buffers}
> > +
> > +\begin{itemize*}
> > +\item If the function and the buffer type require so, write data to
> > +the buffer memory.
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > +processing in the device.
> > +\item The request completes asynchronously when the device has
> > +finished with the buffer.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Buffer processing control}
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> > +  return all of the already queued buffers.
> > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> > +  already queued buffers from the input or the output queue. This also
> > +  includes input or output buffers that can be currently owned by the
> > +  device's processing pipeline.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Asynchronous events}
> > +
> > +While processing buffers, the device can send asynchronous event
> > +notifications to the driver. The behaviour depends on the exact
> > +stream. For example, the decoder device sends a resolution change
> > +event when it encounters new resolution metadata in the stream.
> > +
> > +\subsubsection{Device Operation: Request header}
> > +
> > +All requests and responses on the control virt queue have a fixed
> > +header using the following layout structure and definitions:
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_ctrl_type {
> > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > +
> > +        /* request */
> > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > +
> > +        /* response */
> > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > +
> > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > +};
> > +
> > +struct virtio_video_ctrl_hdr {
> > +        le32 type;
> > +        le32 stream_id;
> > +        le32 len; /* Length of the structure in bytes. */
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{type}] is the type of the driver request or the device
> > +response.
> > +\item[\field{stream_id}] specifies a target stream.
> > +\item[\field{len}] is the length of data in bytes, which includes
> > +length of the header.
> > +\end{description}
> > +
> > +\subsubsection{Device Operation: controlq}
> > +
> > +\begin{description}
> > +
> > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > +supported formats.
> > +
> > +The driver uses \field{struct virtio_video_get_capability} to send a
> > +query request.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_buf_type {
> > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > +};
> > +
> > +struct virtio_video_get_capability {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        enum virtio_video_buf_type buf_type;
> > +};
> > +\end{lstlisting}
> > +\begin{description}
> > +\item[\field{buf_type}] is the buffer type that the driver asks
> > +information about. The driver MUST set either
> > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > +
> > +The device responds a capability by using \field{struct
> > +virtio_video_get_capability_resp}.
> > +\begin{lstlisting}
> > +enum virtio_video_format {
> > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > +        /* Raw formats */
> > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > +        VIRTIO_VIDEO_FORMAT_YVU420,
> > +
> > +        /* Compressed formats */
> > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> > +};
> > +
> > +enum virtio_video_profile {
> > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > +
> > +        /* H.264 */
> > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > +        /* VP8 */
> > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > +
> > +        /* VP9 */
> > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > +        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > +};
> > +
> > +struct virtio_video_format_range {
> > +        le32 min;
> > +        le32 max;
> > +        le32 step;
> > +        u8 paddings[4];
> > +};
> > +
> > +struct virtio_video_format_desc {
> > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > +        le64 mask;
> > +        struct virtio_video_format_range width;
> > +        struct virtio_video_format_range height;
> > +        le32 num_rates;
> > +        u8 padding[4];
> > +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> > +};
> > +
> > +struct virtio_video_get_capability_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 num_descs;
> > +        /* Followed by struct virtio_video_format_desc desc[] */
> > +};
> > +\end{lstlisting}
> > +
> > +The format description \field{struct virtio_video_format_desc}
> > +includes the following fields:
> > +\begin{description}
> > +\item[\field{format}] specifies an image format. The device MUST set one
> > +  of \field{enum virtio_video_format}.
> > +\item[\field{profile}] specifies a profile of the compressed image format
> > +  specified in \field{format}. The driver SHOULD ignore this value if
> > +  \field{format} is a raw format.
> > +\item[\field{mask}] is a bitset that represents the supported
> > +  combination of input and output format. If \textit{i}-th bit is set
> > +  in \field{mask} of \textit{j}-th \field{struct
> > +  virtio_video_format_desc} for input, the device supports encoding or
> > +  decoding from the \textit{j}-th input format to \textit{i}-th output
> > +  format.
> > +\item[\field{width, height}] represents a range of resolutions
> > +  supported by the device. If its \field{step} is not applicable, its
> > +  \field{min} is equal to its \field{max}.
> > +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In
> > case of decoder, the driver SHOULD ignore this value.
> > +\item[\field{frame_rates}] is an array of supported frame rates.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_mem_type {
> > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > +};
> > +
> > +struct virtio_video_stream_create {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > +        char debug_name[64];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{in_mem_type}] is a type of buffer management for input
> > +buffers. The driver MUST set a value in \field{enum
> > +virtio_video_mem_type}.
> > +\item[\field{out_mem_type}] is a type of buffer management for output
> > +buffers. The driver MUST set a value in \field{enum
> > +virtio_video_mem_type}.
> > +\item[\field{debug_name}] is a text string for a debug purpose.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_stream_destroy {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > +  queued buffers through the pipeline.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_stream_drain {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_mem_entry {
> > +        le64 addr;
> > +        le32 length;
> > +        u8 padding[4];
> > +};
> > +
> > +struct virtio_video_resource_create {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 resource_id;
> > +        le32 nr_entries;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{resource_id}] internal id of the resource.
> > +\item[\field{nr_entries}] number of \field{struct
> > +  virtio_video_mem_entry} memory entries.
> > +\end{description}
> > +
>
> How should one deal with multiplanar formats? Do we create one resource per
> plane? Otherwise we need a way to send mem entries for each plane in one
> request.

That's a good point. We should be able to create a resource that
corresponds multiple planes.
How about changing the struct like this:

struct virtio_video_resource_create {
        struct virtio_video_ctrl_hdr hdr;
        le32 resource_id;
        le32 nums_entries[VIRTIO_VIDEO_MAX_PLANES];
        u8 padding[4];
        /* Followed by struct virtio_video_mem_entry entries[].
           |length of entries[]| == |sum of nums_entries[i] for each i| */
};

Best regards,
Keiichi

>
> Currently the v1 driver attaches backing to the same resource multiple times
> when a buffer has multiple planes. Not the nicest way to be honest.
>
> Regards,
> Dmitry.
>
> > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_resource_destroy {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 resource_id;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{resource_id}] internal id of the resource.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > +queue.
> > +
> > +\begin{lstlisting}
> > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > +
> > +struct virtio_video_resource_queue {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type;
> > +        le32 resource_id;
> > +        le64 timestamp;
> > +        le32 nr_data_size;
> > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buf_type of the .
> > +\item[\field{resource_id}] internal id of the resource.
> > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > +  for synchronisation.
> > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > +\item[\field{data_size}] number of data bytes within a plane.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_buffer_flag {
> > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > +        /* Encoder only */
> > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > +};
> > +
> > +struct virtio_video_resource_queue_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le64 timestamp;
> > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > +        le32 size;  /* Encoded size */
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > +  for synchronisation.
> > +\item[\field{flags}] mark specific buffers in the sequence.
> > +\item[\field{size}] data size in the buffer (encoder only).
> > +\end{description}
> > +
> > +The device sends a response to the queue request asynchronously when
> > +it has finished processing the buffer.
> > +
> > +The device SHOULD mark a buffer that triggered a processing error with
> > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > +
> > +The device MUST mark the last buffer with the
> > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > +sequence.
> > +
> > +In case of encoder, to denote a particular frame type the devie MUST
> > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> > +  buffers back from the input or the output queue of the device. The
> > +  device SHOULD return all of the buffers from the respective queue as
> > +  soon as possible without pushing the buffers through the processing
> > +  pipeline.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_queue_clear {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buffer type.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> > +  output of a stream.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_plane_format {
> > +        le32 plane_size;
> > +        le32 stride;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{plane_size}] size of the plane in bytes.
> > +\item[\field{stride}] stride used for the plane in bytes.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_params {
> > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> > +        le32 frame_width;
> > +        le32 frame_height;
> > +        le32 min_buffers;
> > +        le32 max_buffers;
> > +        le32 frame_rate;
> > +        struct virtio_video_crop {
> > +                le32 left;
> > +                le32 top;
> > +                le32 width;
> > +                le32 height;
> > +        } crop;
> > +        le32 num_planes;
> > +        struct virtio_video_plane_format
> > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{frame_width}] the value to get/set.
> > +\item[\field{frame_height}] the value to get/set.
> > +\item[\field{pixel_format}] the value to get/set.
> > +\item[\field{min_buffers}] minimum buffers required to handle the
> > +  format (r/o).
> > +\item[\field{max_buffers}] maximum buffers required to handle the
> > +  format (r/o).
> > +\item[\field{frame_rate}] the value to get/set.
> > +\item[\field{crop}] cropping (composing) rectangle.
> > +\item[\field{num_planes}] number of planes used to store pixel data
> > +(r/o).
> > +\item[\field{plane_formats}] description of each plane.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_get_params {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > +};
> > +
> > +struct virtio_video_get_params_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        struct virtio_video_params params;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buffer type.
> > +\item[\field{params}] parameter values.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> > +
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_set_params {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        struct virtio_video_params params;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{params}] parameters to set.
> > +\end{description}
> > +
> > +Setting stream parameters might have side effects within the device.
> > +For example, the device MAY perform alignment of width and height,
> > +change the number of planes it uses for the format, or do whatever
> > +changes that are required to continue normal operation using the
> > +updated parameters. It is up to the driver to check the parameter set
> > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > +
> > +\end{description}
> > +
> > +\subsubsection{Device Operation: eventq}
> > +
> > +The device can report events on the event queue. The driver initially
> > +populates the queue with device-writeable buffers. When the device
> > +needs to report an event, it fills a buffer and notifies the driver.
> > +The driver consumes the report and adds a new buffer to the virtqueue.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_event_type {
> > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > +        /* For all functions */
> > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > +        /* For decoder only */
> > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> > +};
> > +
> > +struct virtio_video_event {
> > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> > +        le32 stream_id;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{event_type}] type of the triggered event .
> > +\item[\field{stream_id}] id of the source stream.
> > +\end{description}
> > +
> > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > +whenever it encounters new resolution data in the stream. This
> > +includes the case of the initial device configuration after metadata
> > +has been parsed and the case of dynamic resolution change.
>
>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 0/1] VirtIO video device specification
  2020-01-03 13:05         ` Dmitry Sepp
@ 2020-01-06 10:30           ` Keiichi Watanabe
  2020-01-06 11:28             ` Dmitry Sepp
  0 siblings, 1 reply; 73+ messages in thread
From: Keiichi Watanabe @ 2020-01-06 10:30 UTC (permalink / raw)
  To: Dmitry Sepp
  Cc: Tomasz Figa, virtio-dev, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Enrico Granata, Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

Hi Dmitry, Tomasz,


On Fri, Jan 3, 2020 at 10:05 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
>
> Hi Tomasz, Keiichi,
>
> On Samstag, 21. Dezember 2019 07:19:23 CET Tomasz Figa wrote:
> > On Sat, Dec 21, 2019 at 3:18 PM Tomasz Figa <tfiga@chromium.org> wrote:
> > > On Sat, Dec 21, 2019 at 1:36 PM Keiichi Watanabe <keiichiw@chromium.org>
> wrote:
> > > > Hi Dmitry,
> > > >
> > > > On Sat, Dec 21, 2019 at 12:59 AM Dmitry Sepp
> > > >
> > > > <dmitry.sepp@opensynergy.com> wrote:
> > > > > Hi Keiichi,
> > > > >
> > > > > On Mittwoch, 18. Dezember 2019 14:02:13 CET Keiichi Watanabe wrote:
> > > > > > Hi,
> > > > > > This is the 2nd version of virtio-video patch. The PDF is available
> > > > > > in [1].
> > > > > > The first version was sent at [2].
> > > > > >
> > > > > > Any feedback would be appreciated. Thank you.
> > > > > >
> > > > > > Best,
> > > > > > Keiichi
> > > > > >
> > > > > > [1]:
> > > > > > https://drive.google.com/drive/folders/1eT5fEckBoor2iHZR4f4GLxYzFMVa
> > > > > > pOFx?us
> > > > > > p=sharing [2]: https://markmail.org/message/gc6h25acct22niut
> > > > > >
> > > > > > Change log:
> > > > > >
> > > > > > v2:
> > > > > > * Removed functionalities except encoding and decoding.
> > > > > > * Splited encoder and decoder into different devices that use the
> > > > > > same
> > > > > > protocol. * Replaced GET_FUNCS with GET_CAPABILITY.
> > > > > > * Updated structs for capabilities.
> > > > > >
> > > > > >   - Defined new structs and enums such as image formats, profiles,
> > > > > >   range
> > > > > >
> > > > > > (min, max, step), etc
> > > > > >
> > > > > >     * For virtio_video_pixel_format, chose a naming convention that
> > > > > >     is used
> > > > > >
> > > > > >       in DRM. We removed XBGR, NV21 and I422, as they are not used
> > > > > >       in the
> > > > > >       current draft implementation. https://lwn.net/Articles/806416/
> > > > > >
> > > > > >   - Removed virtio_video_control, whose usage was not documented yet
> > > > > >   and
> > > > > >
> > > > > > which is not necessary for the simplest decoding scenario.
> > > > > >
> > > > > >   - Removed virtio_video_desc, as it is no longer needed.
> > > > > >
> > > > > > * Updated struct virtio_video_config for changes around
> > > > > > capabilities.
> > > > > > * Added a way to represent supported combinations of formats.
> > > > > >
> > > > > >   - A field "mask" in virtio_video_format_desc plays this role.
> > > > > >
> > > > > > * Removed VIRTIO_VIDEO_T_STREAM_{START,STOP} because they don't play
> > > > > > any
> > > > > > meaningful roles. * Removed VIRTIO_VIDEO_T_STREAM_{ATTACH,
> > > > > > DETACH}_BACKING
> > > > > > and merged them into RESOURCE_{CREATE, DESTROY}. * Added a way to
> > > > > > notify/specify resource creation method.
> > > > > >
> > > > > >   - Added a feature flag.
> > > > > >   - Defined enum virtio_video_mem_type.
> > > > > >   - Added new fields in video_stream_create.
> > > > > >
> > > > > > * Modified fields in virtio_video_params.
> > > > > >
> > > > > >   - Added crop information.
> > > > > >
> > > > > > * Removed enum virtio_video_channel_type because we can get this
> > > > > > information by image format.
> > > > >
> > > > > Could you please explain this? How do you get the information?
> > > >
> > > > It means that if image formats are well-defined, channel information
> > > > (e.g. the order of channels) is uniquely determined.
> > > >
> > > > > Suppose you have some piece of HW on the host side that wants I420 as
> > > > > one
> > > > > contig buffer w/ some offsets. But on the driver side, say, gralloc
> > > > > gives you three separate buffers, one per channel. How do we pass
> > > > > those to the device then?
> > > >
> > > > You're talking about CrOS use case where buffers are allocated by
> > > > virtio-gpu, right?
> > > > In this case, virtio-gpu allocates one contiguous host-side buffer and
> > > > the client regards a pair of (buffer FD, offset) as one channel.
> > > > And, we can register this pair to the device when the buffer is
> > > > imported.
> > > > In the virtio-vdec spec draft, this pair corresponds to struct
> > > > virtio_vdec_plane in struct virtio_vdec_plane.
> > > >
> > > > So, I suppose we will need similar structs when we add a control to
> > > > import buffers. However, I don't think it's necessary when guest pages
> > > > are used.
> > >
> > > I think we need some way for the guest to know whether it can allocate
> > > the planes in separate buffers, even when guest pages are used. This
> > > would be equivalent to V4L2 M and non-M formats, but mixing this into
> > > FourCC in V4L2 is an acknowledged mistake, so we should add a query or
> > > something.
> > >
>
> Yes, this is what I mean. In fact, we already do face the situation when the
> device side is not happy with the sgt and wants contig. I think we'll add a
> module parameter for now.

Okay. So, I suppose we'll be able to update structs:
* Add a flag in virtio_video_format_desc that indicates whether planes
can be in separate buffers, and
* Add a flag in virtio_video_format_desc that indicates that the
device requires contiguous buffers for this format.

Does it make sense?

Best regards,
Keiichi

>
> Regards,
> Dmitry.
>
> > > For future V4L2 development we came up with the idea of a format flag
> > > which could mean that the hardware allows putting planes in separate
> > > buffers. We could have a similar per-format flag in the capabilities,
> > > as we already have a list of all the supported formats there.
> >
> > Sorry, forgot to paste the link from future V4L2 work notes from this year
> > ELCE: https://www.spinics.net/lists/linux-media/msg159789.html
> >
> > > Best regards,
> > > Tomasz
> > >
> > > > Best regards,
> > > > Keiichi
> > > >
> > > > > Best regards,
> > > > > Dmitry.
> > > > >
> > > > > > * Renamed virtio_video_pin to virtio_video_buf_type.
> > > > > >
> > > > > >   - It's similar to V4L2_BUF_TYPE_VIDEO_{OUTPUT, CAPTURE}.
> > > > > >
> > > > > > * Added an error event.
> > > > > > * Reordered some subsections.
> > > > > > * Changed styles to make it consistent with other devices.
> > > > > >
> > > > > > Dmitry Sepp (1):
> > > > > >   virtio-video: Add virtio video device specification
> > > > > >
> > > > > >  content.tex      |   1 +
> > > > > >  virtio-video.tex | 579
> > > > > >  +++++++++++++++++++++++++++++++++++++++++++++++
> > > > > >  2 files changed, 580 insertions(+)
> > > > > >  create mode 100644 virtio-video.tex
> > > > > >
> > > > > > --
> > > > > > 2.24.1.735.g03f4e72817-goog
>
>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 0/1] VirtIO video device specification
  2020-01-06 10:30           ` Keiichi Watanabe
@ 2020-01-06 11:28             ` Dmitry Sepp
  2020-01-07 10:25               ` Keiichi Watanabe
  0 siblings, 1 reply; 73+ messages in thread
From: Dmitry Sepp @ 2020-01-06 11:28 UTC (permalink / raw)
  To: Keiichi Watanabe
  Cc: Tomasz Figa, virtio-dev, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Enrico Granata, Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

Hi,

On Montag, 6. Januar 2020 11:30:22 CET Keiichi Watanabe wrote:
> Hi Dmitry, Tomasz,
> 
> On Fri, Jan 3, 2020 at 10:05 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> 
wrote:
> > Hi Tomasz, Keiichi,
> > 
> > On Samstag, 21. Dezember 2019 07:19:23 CET Tomasz Figa wrote:
> > > On Sat, Dec 21, 2019 at 3:18 PM Tomasz Figa <tfiga@chromium.org> wrote:
> > > > On Sat, Dec 21, 2019 at 1:36 PM Keiichi Watanabe
> > > > <keiichiw@chromium.org>
> > 
> > wrote:
> > > > > Hi Dmitry,
> > > > > 
> > > > > On Sat, Dec 21, 2019 at 12:59 AM Dmitry Sepp
> > > > > 
> > > > > <dmitry.sepp@opensynergy.com> wrote:
> > > > > > Hi Keiichi,
> > > > > > 
> > > > > > On Mittwoch, 18. Dezember 2019 14:02:13 CET Keiichi Watanabe 
wrote:
> > > > > > > Hi,
> > > > > > > This is the 2nd version of virtio-video patch. The PDF is
> > > > > > > available
> > > > > > > in [1].
> > > > > > > The first version was sent at [2].
> > > > > > > 
> > > > > > > Any feedback would be appreciated. Thank you.
> > > > > > > 
> > > > > > > Best,
> > > > > > > Keiichi
> > > > > > > 
> > > > > > > [1]:
> > > > > > > https://drive.google.com/drive/folders/1eT5fEckBoor2iHZR4f4GLxYz
> > > > > > > FMVa
> > > > > > > pOFx?us
> > > > > > > p=sharing [2]: https://markmail.org/message/gc6h25acct22niut
> > > > > > > 
> > > > > > > Change log:
> > > > > > > 
> > > > > > > v2:
> > > > > > > * Removed functionalities except encoding and decoding.
> > > > > > > * Splited encoder and decoder into different devices that use
> > > > > > > the
> > > > > > > same
> > > > > > > protocol. * Replaced GET_FUNCS with GET_CAPABILITY.
> > > > > > > * Updated structs for capabilities.
> > > > > > > 
> > > > > > >   - Defined new structs and enums such as image formats,
> > > > > > >   profiles,
> > > > > > >   range
> > > > > > > 
> > > > > > > (min, max, step), etc
> > > > > > > 
> > > > > > >     * For virtio_video_pixel_format, chose a naming convention
> > > > > > >     that
> > > > > > >     is used
> > > > > > >     
> > > > > > >       in DRM. We removed XBGR, NV21 and I422, as they are not
> > > > > > >       used
> > > > > > >       in the
> > > > > > >       current draft implementation.
> > > > > > >       https://lwn.net/Articles/806416/
> > > > > > >   
> > > > > > >   - Removed virtio_video_control, whose usage was not documented
> > > > > > >   yet
> > > > > > >   and
> > > > > > > 
> > > > > > > which is not necessary for the simplest decoding scenario.
> > > > > > > 
> > > > > > >   - Removed virtio_video_desc, as it is no longer needed.
> > > > > > > 
> > > > > > > * Updated struct virtio_video_config for changes around
> > > > > > > capabilities.
> > > > > > > * Added a way to represent supported combinations of formats.
> > > > > > > 
> > > > > > >   - A field "mask" in virtio_video_format_desc plays this role.
> > > > > > > 
> > > > > > > * Removed VIRTIO_VIDEO_T_STREAM_{START,STOP} because they don't
> > > > > > > play
> > > > > > > any
> > > > > > > meaningful roles. * Removed VIRTIO_VIDEO_T_STREAM_{ATTACH,
> > > > > > > DETACH}_BACKING
> > > > > > > and merged them into RESOURCE_{CREATE, DESTROY}. * Added a way
> > > > > > > to
> > > > > > > notify/specify resource creation method.
> > > > > > > 
> > > > > > >   - Added a feature flag.
> > > > > > >   - Defined enum virtio_video_mem_type.
> > > > > > >   - Added new fields in video_stream_create.
> > > > > > > 
> > > > > > > * Modified fields in virtio_video_params.
> > > > > > > 
> > > > > > >   - Added crop information.
> > > > > > > 
> > > > > > > * Removed enum virtio_video_channel_type because we can get this
> > > > > > > information by image format.
> > > > > > 
> > > > > > Could you please explain this? How do you get the information?
> > > > > 
> > > > > It means that if image formats are well-defined, channel information
> > > > > (e.g. the order of channels) is uniquely determined.
> > > > > 
> > > > > > Suppose you have some piece of HW on the host side that wants I420
> > > > > > as
> > > > > > one
> > > > > > contig buffer w/ some offsets. But on the driver side, say,
> > > > > > gralloc
> > > > > > gives you three separate buffers, one per channel. How do we pass
> > > > > > those to the device then?
> > > > > 
> > > > > You're talking about CrOS use case where buffers are allocated by
> > > > > virtio-gpu, right?
> > > > > In this case, virtio-gpu allocates one contiguous host-side buffer
> > > > > and
> > > > > the client regards a pair of (buffer FD, offset) as one channel.
> > > > > And, we can register this pair to the device when the buffer is
> > > > > imported.
> > > > > In the virtio-vdec spec draft, this pair corresponds to struct
> > > > > virtio_vdec_plane in struct virtio_vdec_plane.
> > > > > 
> > > > > So, I suppose we will need similar structs when we add a control to
> > > > > import buffers. However, I don't think it's necessary when guest
> > > > > pages
> > > > > are used.
> > > > 
> > > > I think we need some way for the guest to know whether it can allocate
> > > > the planes in separate buffers, even when guest pages are used. This
> > > > would be equivalent to V4L2 M and non-M formats, but mixing this into
> > > > FourCC in V4L2 is an acknowledged mistake, so we should add a query or
> > > > something.
> > 
> > Yes, this is what I mean. In fact, we already do face the situation when
> > the device side is not happy with the sgt and wants contig. I think we'll
> > add a module parameter for now.
> 
> Okay. So, I suppose we'll be able to update structs:
> * Add a flag in virtio_video_format_desc that indicates whether planes
> can be in separate buffers, and
> * Add a flag in virtio_video_format_desc that indicates that the
> device requires contiguous buffers for this format.
> 
> Does it make sense?
> 
Sorry, I don't understand the difference between the two above: isn't the first 
case is just when the flag is not set?

Regards,
Dmitry.

> Best regards,
> Keiichi
> 
> > Regards,
> > Dmitry.
> > 
> > > > For future V4L2 development we came up with the idea of a format flag
> > > > which could mean that the hardware allows putting planes in separate
> > > > buffers. We could have a similar per-format flag in the capabilities,
> > > > as we already have a list of all the supported formats there.
> > > 
> > > Sorry, forgot to paste the link from future V4L2 work notes from this
> > > year
> > > ELCE: https://www.spinics.net/lists/linux-media/msg159789.html
> > > 
> > > > Best regards,
> > > > Tomasz
> > > > 
> > > > > Best regards,
> > > > > Keiichi
> > > > > 
> > > > > > Best regards,
> > > > > > Dmitry.
> > > > > > 
> > > > > > > * Renamed virtio_video_pin to virtio_video_buf_type.
> > > > > > > 
> > > > > > >   - It's similar to V4L2_BUF_TYPE_VIDEO_{OUTPUT, CAPTURE}.
> > > > > > > 
> > > > > > > * Added an error event.
> > > > > > > * Reordered some subsections.
> > > > > > > * Changed styles to make it consistent with other devices.
> > > > > > > 
> > > > > > > Dmitry Sepp (1):
> > > > > > >   virtio-video: Add virtio video device specification
> > > > > > >  
> > > > > > >  content.tex      |   1 +
> > > > > > >  virtio-video.tex | 579
> > > > > > >  +++++++++++++++++++++++++++++++++++++++++++++++
> > > > > > >  2 files changed, 580 insertions(+)
> > > > > > >  create mode 100644 virtio-video.tex
> > > > > > > 
> > > > > > > --
> > > > > > > 2.24.1.735.g03f4e72817-goog



^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-18 13:02 ` [PATCH v2 1/1] virtio-video: Add virtio " Keiichi Watanabe
                     ` (3 preceding siblings ...)
  2020-01-03 15:47   ` Dmitry Sepp
@ 2020-01-06 14:59   ` Dmitry Sepp
  2020-01-07 13:24     ` Keiichi Watanabe
  2020-01-08 12:23   ` Keiichi Watanabe
  5 siblings, 1 reply; 73+ messages in thread
From: Dmitry Sepp @ 2020-01-06 14:59 UTC (permalink / raw)
  To: Keiichi Watanabe
  Cc: virtio-dev, linux-media, acourbot, alexlau, daniel, dgreid,
	egranata, fziglio, hverkuil, kraxel, marcheu, posciak,
	spice-devel, stevensd, tfiga, uril

Hi,

a couple of new comments:

On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> 
> The virtio video encoder device and decoder device provide functionalities
> to encode and decode video stream respectively.
> Though video encoder and decoder are provided as different devices, they use
> a same protocol.
> 
> Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> ---
>  content.tex      |   1 +
>  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 580 insertions(+)
>  create mode 100644 virtio-video.tex
> 
> diff --git a/content.tex b/content.tex
> index 556b373..9e56839 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> Requirements}\label{sec:Device \input{virtio-vsock.tex}
>  \input{virtio-fs.tex}
>  \input{virtio-rpmb.tex}
> +\input{virtio-video.tex}
> 
>  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> 
> diff --git a/virtio-video.tex b/virtio-video.tex
> new file mode 100644
> index 0000000..30e728d
> --- /dev/null
> +++ b/virtio-video.tex
> @@ -0,0 +1,579 @@
> +\section{Video Device}\label{sec:Device Types / Video Device}
> +
> +The virtio video encoder device and decoder device are virtual devices that
> +supports encoding and decoding respectively. Though the encoder and the
> decoder +are different devices, they use the same protocol.
> +
> +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
> +
> +\begin{description}
> +\item[30] encoder device
> +\item[31] decoder device
> +\end{description}
> +
> +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues}
> +
> +\begin{description}
> +\item[0] controlq - queue for sending control commands.
> +\item[1] eventq - queue for sending events happened in the device.
> +\end{description}
> +
> +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature
> bits} +
> +\begin{description}
> +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for
> video +  buffers.
> +\end{description}
> +
> +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device
> / Feature bits} +
> +The device MUST offer at least one of feature bits.
> +
> +\subsection{Device configuration layout}\label{sec:Device Types / Video
> Device / Device configuration layout} +
> +Video device configuration uses the following layout structure:
> +
> +\begin{lstlisting}
> +struct virtio_video_config {
> +        le32 max_cap_len;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> +  MUST set this value.
> +\end{description}
> +
> +\subsection{Device Initialization}\label{sec:Device Types / Video Device /
> Device Initialization} +
> +\devicenormative{\subsubsection}{Device Initialization}{Device Types /
> Video Device / Device Initialization} +
> +The driver SHOULD query device capability by using the
> +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> +setup.
> +
> +\subsection{Device Operation}\label{sec:Device Types / Video Device /
> Device Operation} +
> +The driver allocates input and output buffers and queues the buffers
> +to the device. The device performs operations on the buffers according
> +to the function in question.
> +
> +\subsubsection{Device Operation: Create stream}
> +
> +To process buffers, the device needs to associate them with a certain
> +video stream (essentially, a context). Streams are created by
> +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> +determined by the device.
> +
> +\subsubsection{Device Operation: Create buffers}
> +
> +Buffers are used to store the actual data as well as the relevant
> +metadata. Scatter lists are supported, so the buffer doesn't need to
> +be contiguous in guest physical memory.
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> +  resource that is backed by a buffer allocated from the driver's
> +  memory.
> +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> +  is no longer needed.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Stream parameter control}
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters
> for +  input and output streams from the device.
> +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the
> +  device.
> +\item After setting stream parameters, the driver may issue
> +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can
> be +  changed implicitly by the device during the set operation.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Process buffers}
> +
> +\begin{itemize*}
> +\item If the function and the buffer type require so, write data to
> +the buffer memory.
> +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> +processing in the device.
> +\item The request completes asynchronously when the device has
> +finished with the buffer.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Buffer processing control}
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> +  return all of the already queued buffers.
> +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> +  already queued buffers from the input or the output queue. This also
> +  includes input or output buffers that can be currently owned by the
> +  device's processing pipeline.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Asynchronous events}
> +
> +While processing buffers, the device can send asynchronous event
> +notifications to the driver. The behaviour depends on the exact
> +stream. For example, the decoder device sends a resolution change
> +event when it encounters new resolution metadata in the stream.
> +
> +\subsubsection{Device Operation: Request header}
> +
> +All requests and responses on the control virt queue have a fixed
> +header using the following layout structure and definitions:
> +
> +\begin{lstlisting}
> +enum virtio_video_ctrl_type {
> +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> +
> +        /* request */
> +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> +        VIRTIO_VIDEO_T_STREAM_CREATE,
> +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> +        VIRTIO_VIDEO_T_SET_PARAMS,
> +        VIRTIO_VIDEO_T_GET_PARAMS,
> +
> +        /* response */
> +        VIRTIO_VIDEO_S_OK = 0x0200,
> +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> +
> +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> +};
> +
> +struct virtio_video_ctrl_hdr {
> +        le32 type;
> +        le32 stream_id;
> +        le32 len; /* Length of the structure in bytes. */
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{type}] is the type of the driver request or the device
> +response.
> +\item[\field{stream_id}] specifies a target stream.
> +\item[\field{len}] is the length of data in bytes, which includes
> +length of the header.
> +\end{description}
> +
> +\subsubsection{Device Operation: controlq}
> +
> +\begin{description}
> +
> +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> +supported formats.
> +
> +The driver uses \field{struct virtio_video_get_capability} to send a
> +query request.
> +
> +\begin{lstlisting}
> +enum virtio_video_buf_type {
> +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> +};
> +
> +struct virtio_video_get_capability {
> +        struct virtio_video_ctrl_hdr hdr;
> +        enum virtio_video_buf_type buf_type;
> +};
> +\end{lstlisting}
> +\begin{description}
> +\item[\field{buf_type}] is the buffer type that the driver asks
> +information about. The driver MUST set either
> +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> +
> +The device responds a capability by using \field{struct
> +virtio_video_get_capability_resp}.
> +\begin{lstlisting}
> +enum virtio_video_format {
> +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> +        /* Raw formats */
> +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> +        VIRTIO_VIDEO_FORMAT_YUV420,
> +        VIRTIO_VIDEO_FORMAT_YVU420,
> +
> +        /* Compressed formats */
> +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> +};
> +
> +enum virtio_video_profile {
> +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> +
> +        /* H.264 */
> +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> VIRTIO_VIDEO_PROFILE_H264_BASELINE, +       
> VIRTIO_VIDEO_PROFILE_H264_MAIN,
> +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> +        /* VP8 */
> +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> +
> +        /* VP9 */
> +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> +        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> +};
> +
> +struct virtio_video_format_range {
> +        le32 min;
> +        le32 max;
> +        le32 step;
> +        u8 paddings[4];
> +};
> +
> +struct virtio_video_format_desc {
> +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> +        le64 mask;
> +        struct virtio_video_format_range width;
> +        struct virtio_video_format_range height;
> +        le32 num_rates;
> +        u8 padding[4];
> +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> +};
> +
> +struct virtio_video_get_capability_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 num_descs;
> +        /* Followed by struct virtio_video_format_desc desc[] */
> +};
> +\end{lstlisting}
> +
> +The format description \field{struct virtio_video_format_desc}
> +includes the following fields:
> +\begin{description}
> +\item[\field{format}] specifies an image format. The device MUST set one
> +  of \field{enum virtio_video_format}.
> +\item[\field{profile}] specifies a profile of the compressed image format
> +  specified in \field{format}. The driver SHOULD ignore this value if
> +  \field{format} is a raw format.

So how should this be used? The spec does not define any way to set profile for 
the device. It is very important for encoder.

Also, shouldn't the profile come together with level? Would make sense for 
encoders.

> +\item[\field{mask}] is a bitset that represents the supported
> +  combination of input and output format. If \textit{i}-th bit is set
> +  in \field{mask} of \textit{j}-th \field{struct
> +  virtio_video_format_desc} for input, the device supports encoding or
> +  decoding from the \textit{j}-th input format to \textit{i}-th output
> +  format.
> +\item[\field{width, height}] represents a range of resolutions
> +  supported by the device. If its \field{step} is not applicable, its
> +  \field{min} is equal to its \field{max}.
> +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In
> case of decoder, the driver SHOULD ignore this value.
> +\item[\field{frame_rates}] is an array of supported frame rates.
> +\end{description}
> +

I'd guess frame rates depend on the resolution as well. This dependency was 
clear in the v1 spec, but in the v2 there is no dependency anymore. I think we 
need to update this.

Best regards,
Dmitry.

> +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> +  within the device.
> +
> +\begin{lstlisting}
> +enum virtio_video_mem_type {
> +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> +};
> +
> +struct virtio_video_stream_create {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> +        char debug_name[64];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{in_mem_type}] is a type of buffer management for input
> +buffers. The driver MUST set a value in \field{enum
> +virtio_video_mem_type}.
> +\item[\field{out_mem_type}] is a type of buffer management for output
> +buffers. The driver MUST set a value in \field{enum
> +virtio_video_mem_type}.
> +\item[\field{debug_name}] is a text string for a debug purpose.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_stream_destroy {
> +        struct virtio_video_ctrl_hdr hdr;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> +  queued buffers through the pipeline.
> +
> +\begin{lstlisting}
> +struct virtio_video_stream_drain {
> +        struct virtio_video_ctrl_hdr hdr;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_mem_entry {
> +        le64 addr;
> +        le32 length;
> +        u8 padding[4];
> +};
> +
> +struct virtio_video_resource_create {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 resource_id;
> +        le32 nr_entries;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{resource_id}] internal id of the resource.
> +\item[\field{nr_entries}] number of \field{struct
> +  virtio_video_mem_entry} memory entries.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_resource_destroy {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 resource_id;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{resource_id}] internal id of the resource.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> +queue.
> +
> +\begin{lstlisting}
> +#define VIRTIO_VIDEO_MAX_PLANES 8
> +
> +struct virtio_video_resource_queue {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type;
> +        le32 resource_id;
> +        le64 timestamp;
> +        le32 nr_data_size;
> +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buf_type of the .
> +\item[\field{resource_id}] internal id of the resource.
> +\item[\field{timestamp}] an abstract sequence counter that can be used
> +  for synchronisation.
> +\item[\field{nr_data_size}] number of \field{data_size} entries.
> +\item[\field{data_size}] number of data bytes within a plane.
> +\end{description}
> +
> +\begin{lstlisting}
> +enum virtio_video_buffer_flag {
> +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> +        /* Encoder only */
> +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> +};
> +
> +struct virtio_video_resource_queue_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le64 timestamp;
> +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> +        le32 size;  /* Encoded size */
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{timestamp}] an abstract sequence counter that can be used
> +  for synchronisation.
> +\item[\field{flags}] mark specific buffers in the sequence.
> +\item[\field{size}] data size in the buffer (encoder only).
> +\end{description}
> +
> +The device sends a response to the queue request asynchronously when
> +it has finished processing the buffer.
> +
> +The device SHOULD mark a buffer that triggered a processing error with
> +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> +
> +The device MUST mark the last buffer with the
> +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> +sequence.
> +
> +In case of encoder, to denote a particular frame type the devie MUST
> +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> +  buffers back from the input or the output queue of the device. The
> +  device SHOULD return all of the buffers from the respective queue as
> +  soon as possible without pushing the buffers through the processing
> +  pipeline.
> +
> +\begin{lstlisting}
> +struct virtio_video_queue_clear {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buffer type.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> +  output of a stream.
> +
> +\begin{lstlisting}
> +struct virtio_video_plane_format {
> +        le32 plane_size;
> +        le32 stride;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{plane_size}] size of the plane in bytes.
> +\item[\field{stride}] stride used for the plane in bytes.
> +\end{description}
> +
> +\begin{lstlisting}
> +struct virtio_video_params {
> +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> +        le32 frame_width;
> +        le32 frame_height;
> +        le32 min_buffers;
> +        le32 max_buffers;
> +        le32 frame_rate;
> +        struct virtio_video_crop {
> +                le32 left;
> +                le32 top;
> +                le32 width;
> +                le32 height;
> +        } crop;
> +        le32 num_planes;
> +        struct virtio_video_plane_format
> plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{frame_width}] the value to get/set.
> +\item[\field{frame_height}] the value to get/set.
> +\item[\field{pixel_format}] the value to get/set.
> +\item[\field{min_buffers}] minimum buffers required to handle the
> +  format (r/o).
> +\item[\field{max_buffers}] maximum buffers required to handle the
> +  format (r/o).
> +\item[\field{frame_rate}] the value to get/set.
> +\item[\field{crop}] cropping (composing) rectangle.
> +\item[\field{num_planes}] number of planes used to store pixel data
> +(r/o).
> +\item[\field{plane_formats}] description of each plane.
> +\end{description}
> +
> +\begin{lstlisting}
> +struct virtio_video_get_params {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> +};
> +
> +struct virtio_video_get_params_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        struct virtio_video_params params;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buffer type.
> +\item[\field{params}] parameter values.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> +
> +
> +\begin{lstlisting}
> +struct virtio_video_set_params {
> +        struct virtio_video_ctrl_hdr hdr;
> +        struct virtio_video_params params;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{params}] parameters to set.
> +\end{description}
> +
> +Setting stream parameters might have side effects within the device.
> +For example, the device MAY perform alignment of width and height,
> +change the number of planes it uses for the format, or do whatever
> +changes that are required to continue normal operation using the
> +updated parameters. It is up to the driver to check the parameter set
> +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> +
> +\end{description}
> +
> +\subsubsection{Device Operation: eventq}
> +
> +The device can report events on the event queue. The driver initially
> +populates the queue with device-writeable buffers. When the device
> +needs to report an event, it fills a buffer and notifies the driver.
> +The driver consumes the report and adds a new buffer to the virtqueue.
> +
> +\begin{lstlisting}
> +enum virtio_video_event_type {
> +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> +        /* For all functions */
> +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> +        /* For decoder only */
> +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> +};
> +
> +struct virtio_video_event {
> +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> +        le32 stream_id;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{event_type}] type of the triggered event .
> +\item[\field{stream_id}] id of the source stream.
> +\end{description}
> +
> +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> +whenever it encounters new resolution data in the stream. This
> +includes the case of the initial device configuration after metadata
> +has been parsed and the case of dynamic resolution change.



^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 0/1] VirtIO video device specification
  2020-01-06 11:28             ` Dmitry Sepp
@ 2020-01-07 10:25               ` Keiichi Watanabe
  2020-01-09 14:56                 ` Dmitry Sepp
  0 siblings, 1 reply; 73+ messages in thread
From: Keiichi Watanabe @ 2020-01-07 10:25 UTC (permalink / raw)
  To: Dmitry Sepp
  Cc: Tomasz Figa, virtio-dev, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Enrico Granata, Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

Hi Dmitry,

On Mon, Jan 6, 2020 at 8:28 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
>
> Hi,
>
> On Montag, 6. Januar 2020 11:30:22 CET Keiichi Watanabe wrote:
> > Hi Dmitry, Tomasz,
> >
> > On Fri, Jan 3, 2020 at 10:05 PM Dmitry Sepp <dmitry.sepp@opensynergy.com>
> wrote:
> > > Hi Tomasz, Keiichi,
> > >
> > > On Samstag, 21. Dezember 2019 07:19:23 CET Tomasz Figa wrote:
> > > > On Sat, Dec 21, 2019 at 3:18 PM Tomasz Figa <tfiga@chromium.org> wrote:
> > > > > On Sat, Dec 21, 2019 at 1:36 PM Keiichi Watanabe
> > > > > <keiichiw@chromium.org>
> > >
> > > wrote:
> > > > > > Hi Dmitry,
> > > > > >
> > > > > > On Sat, Dec 21, 2019 at 12:59 AM Dmitry Sepp
> > > > > >
> > > > > > <dmitry.sepp@opensynergy.com> wrote:
> > > > > > > Hi Keiichi,
> > > > > > >
> > > > > > > On Mittwoch, 18. Dezember 2019 14:02:13 CET Keiichi Watanabe
> wrote:
> > > > > > > > Hi,
> > > > > > > > This is the 2nd version of virtio-video patch. The PDF is
> > > > > > > > available
> > > > > > > > in [1].
> > > > > > > > The first version was sent at [2].
> > > > > > > >
> > > > > > > > Any feedback would be appreciated. Thank you.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Keiichi
> > > > > > > >
> > > > > > > > [1]:
> > > > > > > > https://drive.google.com/drive/folders/1eT5fEckBoor2iHZR4f4GLxYz
> > > > > > > > FMVa
> > > > > > > > pOFx?us
> > > > > > > > p=sharing [2]: https://markmail.org/message/gc6h25acct22niut
> > > > > > > >
> > > > > > > > Change log:
> > > > > > > >
> > > > > > > > v2:
> > > > > > > > * Removed functionalities except encoding and decoding.
> > > > > > > > * Splited encoder and decoder into different devices that use
> > > > > > > > the
> > > > > > > > same
> > > > > > > > protocol. * Replaced GET_FUNCS with GET_CAPABILITY.
> > > > > > > > * Updated structs for capabilities.
> > > > > > > >
> > > > > > > >   - Defined new structs and enums such as image formats,
> > > > > > > >   profiles,
> > > > > > > >   range
> > > > > > > >
> > > > > > > > (min, max, step), etc
> > > > > > > >
> > > > > > > >     * For virtio_video_pixel_format, chose a naming convention
> > > > > > > >     that
> > > > > > > >     is used
> > > > > > > >
> > > > > > > >       in DRM. We removed XBGR, NV21 and I422, as they are not
> > > > > > > >       used
> > > > > > > >       in the
> > > > > > > >       current draft implementation.
> > > > > > > >       https://lwn.net/Articles/806416/
> > > > > > > >
> > > > > > > >   - Removed virtio_video_control, whose usage was not documented
> > > > > > > >   yet
> > > > > > > >   and
> > > > > > > >
> > > > > > > > which is not necessary for the simplest decoding scenario.
> > > > > > > >
> > > > > > > >   - Removed virtio_video_desc, as it is no longer needed.
> > > > > > > >
> > > > > > > > * Updated struct virtio_video_config for changes around
> > > > > > > > capabilities.
> > > > > > > > * Added a way to represent supported combinations of formats.
> > > > > > > >
> > > > > > > >   - A field "mask" in virtio_video_format_desc plays this role.
> > > > > > > >
> > > > > > > > * Removed VIRTIO_VIDEO_T_STREAM_{START,STOP} because they don't
> > > > > > > > play
> > > > > > > > any
> > > > > > > > meaningful roles. * Removed VIRTIO_VIDEO_T_STREAM_{ATTACH,
> > > > > > > > DETACH}_BACKING
> > > > > > > > and merged them into RESOURCE_{CREATE, DESTROY}. * Added a way
> > > > > > > > to
> > > > > > > > notify/specify resource creation method.
> > > > > > > >
> > > > > > > >   - Added a feature flag.
> > > > > > > >   - Defined enum virtio_video_mem_type.
> > > > > > > >   - Added new fields in video_stream_create.
> > > > > > > >
> > > > > > > > * Modified fields in virtio_video_params.
> > > > > > > >
> > > > > > > >   - Added crop information.
> > > > > > > >
> > > > > > > > * Removed enum virtio_video_channel_type because we can get this
> > > > > > > > information by image format.
> > > > > > >
> > > > > > > Could you please explain this? How do you get the information?
> > > > > >
> > > > > > It means that if image formats are well-defined, channel information
> > > > > > (e.g. the order of channels) is uniquely determined.
> > > > > >
> > > > > > > Suppose you have some piece of HW on the host side that wants I420
> > > > > > > as
> > > > > > > one
> > > > > > > contig buffer w/ some offsets. But on the driver side, say,
> > > > > > > gralloc
> > > > > > > gives you three separate buffers, one per channel. How do we pass
> > > > > > > those to the device then?
> > > > > >
> > > > > > You're talking about CrOS use case where buffers are allocated by
> > > > > > virtio-gpu, right?
> > > > > > In this case, virtio-gpu allocates one contiguous host-side buffer
> > > > > > and
> > > > > > the client regards a pair of (buffer FD, offset) as one channel.
> > > > > > And, we can register this pair to the device when the buffer is
> > > > > > imported.
> > > > > > In the virtio-vdec spec draft, this pair corresponds to struct
> > > > > > virtio_vdec_plane in struct virtio_vdec_plane.
> > > > > >
> > > > > > So, I suppose we will need similar structs when we add a control to
> > > > > > import buffers. However, I don't think it's necessary when guest
> > > > > > pages
> > > > > > are used.
> > > > >
> > > > > I think we need some way for the guest to know whether it can allocate
> > > > > the planes in separate buffers, even when guest pages are used. This
> > > > > would be equivalent to V4L2 M and non-M formats, but mixing this into
> > > > > FourCC in V4L2 is an acknowledged mistake, so we should add a query or
> > > > > something.
> > >
> > > Yes, this is what I mean. In fact, we already do face the situation when
> > > the device side is not happy with the sgt and wants contig. I think we'll
> > > add a module parameter for now.
> >
> > Okay. So, I suppose we'll be able to update structs:
> > * Add a flag in virtio_video_format_desc that indicates whether planes
> > can be in separate buffers, and
> > * Add a flag in virtio_video_format_desc that indicates that the
> > device requires contiguous buffers for this format.
> >
> > Does it make sense?
> >
> Sorry, I don't understand the difference between the two above: isn't the first
> case is just when the flag is not set?

Ah, I was confused and wrote something strange. Yeah,  these two are the same.
Sorry for that.

So, the suggestion is to add a field "planes_layout" in
virtio_video_format_desc, which is one of the following enums:

enum virtio_video_planes_layout {
    VIRTIO_VIDEO_PLANES_LAYOUT_UNSPEC = 0,  /* no special requirement */
    VIRTIO_VIDEO_PLANES_LAYOUT_CONTIGUOUS,
};

If we have a better idea or naming, please let me know.

Best regards,
Keiichi

>
> Regards,
> Dmitry.
>
> > Best regards,
> > Keiichi
> >
> > > Regards,
> > > Dmitry.
> > >
> > > > > For future V4L2 development we came up with the idea of a format flag
> > > > > which could mean that the hardware allows putting planes in separate
> > > > > buffers. We could have a similar per-format flag in the capabilities,
> > > > > as we already have a list of all the supported formats there.
> > > >
> > > > Sorry, forgot to paste the link from future V4L2 work notes from this
> > > > year
> > > > ELCE: https://www.spinics.net/lists/linux-media/msg159789.html
> > > >
> > > > > Best regards,
> > > > > Tomasz
> > > > >
> > > > > > Best regards,
> > > > > > Keiichi
> > > > > >
> > > > > > > Best regards,
> > > > > > > Dmitry.
> > > > > > >
> > > > > > > > * Renamed virtio_video_pin to virtio_video_buf_type.
> > > > > > > >
> > > > > > > >   - It's similar to V4L2_BUF_TYPE_VIDEO_{OUTPUT, CAPTURE}.
> > > > > > > >
> > > > > > > > * Added an error event.
> > > > > > > > * Reordered some subsections.
> > > > > > > > * Changed styles to make it consistent with other devices.
> > > > > > > >
> > > > > > > > Dmitry Sepp (1):
> > > > > > > >   virtio-video: Add virtio video device specification
> > > > > > > >
> > > > > > > >  content.tex      |   1 +
> > > > > > > >  virtio-video.tex | 579
> > > > > > > >  +++++++++++++++++++++++++++++++++++++++++++++++
> > > > > > > >  2 files changed, 580 insertions(+)
> > > > > > > >  create mode 100644 virtio-video.tex
> > > > > > > >
> > > > > > > > --
> > > > > > > > 2.24.1.735.g03f4e72817-goog
>
>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2020-01-06 14:59   ` Dmitry Sepp
@ 2020-01-07 13:24     ` Keiichi Watanabe
  2020-01-07 16:50       ` Dmitry Sepp
  0 siblings, 1 reply; 73+ messages in thread
From: Keiichi Watanabe @ 2020-01-07 13:24 UTC (permalink / raw)
  To: Dmitry Sepp
  Cc: virtio-dev, Linux Media Mailing List, Alexandre Courbot,
	Alex Lau, Daniel Vetter, Dylan Reid, Enrico Granata,
	Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, Tomasz Figa, uril

Hi Dmitry,

On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
>
> Hi,
>
> a couple of new comments:
>
> On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> >
> > The virtio video encoder device and decoder device provide functionalities
> > to encode and decode video stream respectively.
> > Though video encoder and decoder are provided as different devices, they use
> > a same protocol.
> >
> > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > ---
> >  content.tex      |   1 +
> >  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 580 insertions(+)
> >  create mode 100644 virtio-video.tex
> >
> > diff --git a/content.tex b/content.tex
> > index 556b373..9e56839 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> >  \input{virtio-fs.tex}
> >  \input{virtio-rpmb.tex}
> > +\input{virtio-video.tex}
> >
> >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> >
> > diff --git a/virtio-video.tex b/virtio-video.tex
> > new file mode 100644
> > index 0000000..30e728d
> > --- /dev/null
> > +++ b/virtio-video.tex
> > @@ -0,0 +1,579 @@
> > +\section{Video Device}\label{sec:Device Types / Video Device}
> > +
> > +The virtio video encoder device and decoder device are virtual devices that
> > +supports encoding and decoding respectively. Though the encoder and the
> > decoder +are different devices, they use the same protocol.
> > +
> > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
> > +
> > +\begin{description}
> > +\item[30] encoder device
> > +\item[31] decoder device
> > +\end{description}
> > +
> > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues}
> > +
> > +\begin{description}
> > +\item[0] controlq - queue for sending control commands.
> > +\item[1] eventq - queue for sending events happened in the device.
> > +\end{description}
> > +
> > +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature
> > bits} +
> > +\begin{description}
> > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for
> > video +  buffers.
> > +\end{description}
> > +
> > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device
> > / Feature bits} +
> > +The device MUST offer at least one of feature bits.
> > +
> > +\subsection{Device configuration layout}\label{sec:Device Types / Video
> > Device / Device configuration layout} +
> > +Video device configuration uses the following layout structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_config {
> > +        le32 max_cap_len;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > +  MUST set this value.
> > +\end{description}
> > +
> > +\subsection{Device Initialization}\label{sec:Device Types / Video Device /
> > Device Initialization} +
> > +\devicenormative{\subsubsection}{Device Initialization}{Device Types /
> > Video Device / Device Initialization} +
> > +The driver SHOULD query device capability by using the
> > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> > +setup.
> > +
> > +\subsection{Device Operation}\label{sec:Device Types / Video Device /
> > Device Operation} +
> > +The driver allocates input and output buffers and queues the buffers
> > +to the device. The device performs operations on the buffers according
> > +to the function in question.
> > +
> > +\subsubsection{Device Operation: Create stream}
> > +
> > +To process buffers, the device needs to associate them with a certain
> > +video stream (essentially, a context). Streams are created by
> > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > +determined by the device.
> > +
> > +\subsubsection{Device Operation: Create buffers}
> > +
> > +Buffers are used to store the actual data as well as the relevant
> > +metadata. Scatter lists are supported, so the buffer doesn't need to
> > +be contiguous in guest physical memory.
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > +  resource that is backed by a buffer allocated from the driver's
> > +  memory.
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> > +  is no longer needed.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Stream parameter control}
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters
> > for +  input and output streams from the device.
> > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the
> > +  device.
> > +\item After setting stream parameters, the driver may issue
> > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can
> > be +  changed implicitly by the device during the set operation.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Process buffers}
> > +
> > +\begin{itemize*}
> > +\item If the function and the buffer type require so, write data to
> > +the buffer memory.
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > +processing in the device.
> > +\item The request completes asynchronously when the device has
> > +finished with the buffer.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Buffer processing control}
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> > +  return all of the already queued buffers.
> > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> > +  already queued buffers from the input or the output queue. This also
> > +  includes input or output buffers that can be currently owned by the
> > +  device's processing pipeline.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Asynchronous events}
> > +
> > +While processing buffers, the device can send asynchronous event
> > +notifications to the driver. The behaviour depends on the exact
> > +stream. For example, the decoder device sends a resolution change
> > +event when it encounters new resolution metadata in the stream.
> > +
> > +\subsubsection{Device Operation: Request header}
> > +
> > +All requests and responses on the control virt queue have a fixed
> > +header using the following layout structure and definitions:
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_ctrl_type {
> > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > +
> > +        /* request */
> > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > +
> > +        /* response */
> > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > +
> > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > +};
> > +
> > +struct virtio_video_ctrl_hdr {
> > +        le32 type;
> > +        le32 stream_id;
> > +        le32 len; /* Length of the structure in bytes. */
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{type}] is the type of the driver request or the device
> > +response.
> > +\item[\field{stream_id}] specifies a target stream.
> > +\item[\field{len}] is the length of data in bytes, which includes
> > +length of the header.
> > +\end{description}
> > +
> > +\subsubsection{Device Operation: controlq}
> > +
> > +\begin{description}
> > +
> > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > +supported formats.
> > +
> > +The driver uses \field{struct virtio_video_get_capability} to send a
> > +query request.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_buf_type {
> > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > +};
> > +
> > +struct virtio_video_get_capability {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        enum virtio_video_buf_type buf_type;
> > +};
> > +\end{lstlisting}
> > +\begin{description}
> > +\item[\field{buf_type}] is the buffer type that the driver asks
> > +information about. The driver MUST set either
> > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > +
> > +The device responds a capability by using \field{struct
> > +virtio_video_get_capability_resp}.
> > +\begin{lstlisting}
> > +enum virtio_video_format {
> > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > +        /* Raw formats */
> > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > +        VIRTIO_VIDEO_FORMAT_YVU420,
> > +
> > +        /* Compressed formats */
> > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> > +};
> > +
> > +enum virtio_video_profile {
> > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > +
> > +        /* H.264 */
> > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > +        /* VP8 */
> > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > +
> > +        /* VP9 */
> > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > +        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > +};
> > +
> > +struct virtio_video_format_range {
> > +        le32 min;
> > +        le32 max;
> > +        le32 step;
> > +        u8 paddings[4];
> > +};
> > +
> > +struct virtio_video_format_desc {
> > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > +        le64 mask;
> > +        struct virtio_video_format_range width;
> > +        struct virtio_video_format_range height;
> > +        le32 num_rates;
> > +        u8 padding[4];
> > +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> > +};
> > +
> > +struct virtio_video_get_capability_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 num_descs;
> > +        /* Followed by struct virtio_video_format_desc desc[] */
> > +};
> > +\end{lstlisting}
> > +
> > +The format description \field{struct virtio_video_format_desc}
> > +includes the following fields:
> > +\begin{description}
> > +\item[\field{format}] specifies an image format. The device MUST set one
> > +  of \field{enum virtio_video_format}.
> > +\item[\field{profile}] specifies a profile of the compressed image format
> > +  specified in \field{format}. The driver SHOULD ignore this value if
> > +  \field{format} is a raw format.
>
> So how should this be used? The spec does not define any way to set profile for
> the device. It is very important for encoder.
>

Thank you for pointing this.
These points are overlooked, as I didn't care about encoder usage enough.

After thinking it again, I think it's not a very good idea to include
supported profiles and levels in a struct for capability.
This is because these values are available only for limited number of formats.
Also, it's true that we need to have a way to set these values as
Dmitry pointed.

Instead, it would make more sense to have additional three types of
controls for profiles, levels, and bitrates:
(1) QUERY_CONTROL: Query values supported by the device
(2) GET_CONTROL: Read a value that is set in the device
(3) SET_CONTROL: Set a value in the device

These operations are similar to V4L2 controls.
(1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL, and
G_CTRL in V4L2, respectively.
Also, (3) would be similar to enum virtio_video_control_type in the
virtio-video v1 driver implementation in
https://markmail.org/message/dwghwdqsbl3gsjxu .

For QUERY_CONTROL, my idea is like this:

enum virtio_video_control_type {
  VIRTIO_VIDEO_CONTROL_UNDEFINED = 0,

  VIRTIO_VIDEO_CONTROL_BITRATE = 0x100,
  VIRTIO_VIDEO_CONTROL_PROFILE,
  VIRTIO_VIDEO_CONTROL_LEVEL,
};

struct virtio_video_query_control {
  struct virtio_video_ctrl_hdr hdr;
  le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
  le32 length;
  /* Followed by additional data.
   * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE,
   * the device must pass a codec format like H264 or VP9.
   * The requred data must be defined in the specification.
   */
};

struct virtio_video_query_control_resp {
  struct virtio_video_ctrl_hdr hdr;
  le32 length;
  u8 padding[4];
  /* Followed by data corresponds to the specified control.
   * The type of data must be defined in the spec.
   * For example, if the driver queries profiles, this part should be
   * an array of supported profiles of a given format.
   */
};

WDYT?

> Also, shouldn't the profile come together with level? Would make sense for
> encoders.

Yeah. So, in the above idea of QUERY_CONTROL, profile should be
required when querying supported levels.

>
> > +\item[\field{mask}] is a bitset that represents the supported
> > +  combination of input and output format. If \textit{i}-th bit is set
> > +  in \field{mask} of \textit{j}-th \field{struct
> > +  virtio_video_format_desc} for input, the device supports encoding or
> > +  decoding from the \textit{j}-th input format to \textit{i}-th output
> > +  format.
> > +\item[\field{width, height}] represents a range of resolutions
> > +  supported by the device. If its \field{step} is not applicable, its
> > +  \field{min} is equal to its \field{max}.
> > +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In
> > case of decoder, the driver SHOULD ignore this value.
> > +\item[\field{frame_rates}] is an array of supported frame rates.
> > +\end{description}
> > +
>
> I'd guess frame rates depend on the resolution as well. This dependency was
> clear in the v1 spec, but in the v2 there is no dependency anymore. I think we
> need to update this.

That's a good point. I missed that dependency when updating the structures.
So, let me update the structs like the following:

struct virtio_video_format_frame {
        struct virtio_video_format_range width;
        struct virtio_video_format_range height;
        le32 num_rates;
        u8 padding[4];
        /* Followed by struct virtio_video_format_range frame_rates[] */
};

struct virtio_video_format_desc {
        le64 mask;
        le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
        le32 planes_layout; /* See the thread [v2 0/1] */
        le32 num_frames;
        u8 padding[4];
        /* Followed by struct virtio_video_format_frame frames[] */
};

Best regards,
Keiichi.


>
> Best regards,
> Dmitry.
>
> > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_mem_type {
> > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > +};
> > +
> > +struct virtio_video_stream_create {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > +        char debug_name[64];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{in_mem_type}] is a type of buffer management for input
> > +buffers. The driver MUST set a value in \field{enum
> > +virtio_video_mem_type}.
> > +\item[\field{out_mem_type}] is a type of buffer management for output
> > +buffers. The driver MUST set a value in \field{enum
> > +virtio_video_mem_type}.
> > +\item[\field{debug_name}] is a text string for a debug purpose.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_stream_destroy {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > +  queued buffers through the pipeline.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_stream_drain {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_mem_entry {
> > +        le64 addr;
> > +        le32 length;
> > +        u8 padding[4];
> > +};
> > +
> > +struct virtio_video_resource_create {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 resource_id;
> > +        le32 nr_entries;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{resource_id}] internal id of the resource.
> > +\item[\field{nr_entries}] number of \field{struct
> > +  virtio_video_mem_entry} memory entries.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_resource_destroy {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 resource_id;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{resource_id}] internal id of the resource.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > +queue.
> > +
> > +\begin{lstlisting}
> > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > +
> > +struct virtio_video_resource_queue {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type;
> > +        le32 resource_id;
> > +        le64 timestamp;
> > +        le32 nr_data_size;
> > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buf_type of the .
> > +\item[\field{resource_id}] internal id of the resource.
> > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > +  for synchronisation.
> > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > +\item[\field{data_size}] number of data bytes within a plane.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_buffer_flag {
> > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > +        /* Encoder only */
> > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > +};
> > +
> > +struct virtio_video_resource_queue_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le64 timestamp;
> > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > +        le32 size;  /* Encoded size */
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > +  for synchronisation.
> > +\item[\field{flags}] mark specific buffers in the sequence.
> > +\item[\field{size}] data size in the buffer (encoder only).
> > +\end{description}
> > +
> > +The device sends a response to the queue request asynchronously when
> > +it has finished processing the buffer.
> > +
> > +The device SHOULD mark a buffer that triggered a processing error with
> > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > +
> > +The device MUST mark the last buffer with the
> > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > +sequence.
> > +
> > +In case of encoder, to denote a particular frame type the devie MUST
> > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> > +  buffers back from the input or the output queue of the device. The
> > +  device SHOULD return all of the buffers from the respective queue as
> > +  soon as possible without pushing the buffers through the processing
> > +  pipeline.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_queue_clear {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buffer type.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> > +  output of a stream.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_plane_format {
> > +        le32 plane_size;
> > +        le32 stride;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{plane_size}] size of the plane in bytes.
> > +\item[\field{stride}] stride used for the plane in bytes.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_params {
> > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> > +        le32 frame_width;
> > +        le32 frame_height;
> > +        le32 min_buffers;
> > +        le32 max_buffers;
> > +        le32 frame_rate;
> > +        struct virtio_video_crop {
> > +                le32 left;
> > +                le32 top;
> > +                le32 width;
> > +                le32 height;
> > +        } crop;
> > +        le32 num_planes;
> > +        struct virtio_video_plane_format
> > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{frame_width}] the value to get/set.
> > +\item[\field{frame_height}] the value to get/set.
> > +\item[\field{pixel_format}] the value to get/set.
> > +\item[\field{min_buffers}] minimum buffers required to handle the
> > +  format (r/o).
> > +\item[\field{max_buffers}] maximum buffers required to handle the
> > +  format (r/o).
> > +\item[\field{frame_rate}] the value to get/set.
> > +\item[\field{crop}] cropping (composing) rectangle.
> > +\item[\field{num_planes}] number of planes used to store pixel data
> > +(r/o).
> > +\item[\field{plane_formats}] description of each plane.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_get_params {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > +};
> > +
> > +struct virtio_video_get_params_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        struct virtio_video_params params;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buffer type.
> > +\item[\field{params}] parameter values.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> > +
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_set_params {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        struct virtio_video_params params;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{params}] parameters to set.
> > +\end{description}
> > +
> > +Setting stream parameters might have side effects within the device.
> > +For example, the device MAY perform alignment of width and height,
> > +change the number of planes it uses for the format, or do whatever
> > +changes that are required to continue normal operation using the
> > +updated parameters. It is up to the driver to check the parameter set
> > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > +
> > +\end{description}
> > +
> > +\subsubsection{Device Operation: eventq}
> > +
> > +The device can report events on the event queue. The driver initially
> > +populates the queue with device-writeable buffers. When the device
> > +needs to report an event, it fills a buffer and notifies the driver.
> > +The driver consumes the report and adds a new buffer to the virtqueue.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_event_type {
> > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > +        /* For all functions */
> > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > +        /* For decoder only */
> > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> > +};
> > +
> > +struct virtio_video_event {
> > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> > +        le32 stream_id;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{event_type}] type of the triggered event .
> > +\item[\field{stream_id}] id of the source stream.
> > +\end{description}
> > +
> > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > +whenever it encounters new resolution data in the stream. This
> > +includes the case of the initial device configuration after metadata
> > +has been parsed and the case of dynamic resolution change.
>
>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2020-01-07 13:24     ` Keiichi Watanabe
@ 2020-01-07 16:50       ` Dmitry Sepp
  2020-01-08  6:59         ` Keiichi Watanabe
  0 siblings, 1 reply; 73+ messages in thread
From: Dmitry Sepp @ 2020-01-07 16:50 UTC (permalink / raw)
  To: Keiichi Watanabe, uril
  Cc: virtio-dev, Linux Media Mailing List, Alexandre Courbot,
	Alex Lau, Daniel Vetter, Dylan Reid, Enrico Granata,
	Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, Tomasz Figa

Hi Keiichi,

thanks for the updates, please see my comments below.

On Dienstag, 7. Januar 2020 14:24:31 CET Keiichi Watanabe wrote:
> Hi Dmitry,
> 
> On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> 
wrote:
> > Hi,
> > 
> > a couple of new comments:
> > 
> > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > 
> > > The virtio video encoder device and decoder device provide
> > > functionalities
> > > to encode and decode video stream respectively.
> > > Though video encoder and decoder are provided as different devices, they
> > > use a same protocol.
> > > 
> > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > > ---
> > > 
> > >  content.tex      |   1 +
> > >  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
> > >  2 files changed, 580 insertions(+)
> > >  create mode 100644 virtio-video.tex
> > > 
> > > diff --git a/content.tex b/content.tex
> > > index 556b373..9e56839 100644
> > > --- a/content.tex
> > > +++ b/content.tex
> > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> > > 
> > >  \input{virtio-fs.tex}
> > >  \input{virtio-rpmb.tex}
> > > 
> > > +\input{virtio-video.tex}
> > > 
> > >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> > > 
> > > diff --git a/virtio-video.tex b/virtio-video.tex
> > > new file mode 100644
> > > index 0000000..30e728d
> > > --- /dev/null
> > > +++ b/virtio-video.tex
> > > @@ -0,0 +1,579 @@
> > > +\section{Video Device}\label{sec:Device Types / Video Device}
> > > +
> > > +The virtio video encoder device and decoder device are virtual devices
> > > that +supports encoding and decoding respectively. Though the encoder
> > > and the decoder +are different devices, they use the same protocol.
> > > +
> > > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device
> > > ID}
> > > +
> > > +\begin{description}
> > > +\item[30] encoder device
> > > +\item[31] decoder device
> > > +\end{description}
> > > +
> > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device /
> > > Virtqueues} +
> > > +\begin{description}
> > > +\item[0] controlq - queue for sending control commands.
> > > +\item[1] eventq - queue for sending events happened in the device.
> > > +\end{description}
> > > +
> > > +\subsection{Feature bits}\label{sec:Device Types / Video Device /
> > > Feature
> > > bits} +
> > > +\begin{description}
> > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used
> > > for
> > > video +  buffers.
> > > +\end{description}
> > > +
> > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video
> > > Device
> > > / Feature bits} +
> > > +The device MUST offer at least one of feature bits.
> > > +
> > > +\subsection{Device configuration layout}\label{sec:Device Types / Video
> > > Device / Device configuration layout} +
> > > +Video device configuration uses the following layout structure:
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_config {
> > > +        le32 max_cap_len;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> > > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > > +  MUST set this value.
> > > +\end{description}
> > > +
> > > +\subsection{Device Initialization}\label{sec:Device Types / Video
> > > Device /
> > > Device Initialization} +
> > > +\devicenormative{\subsubsection}{Device Initialization}{Device Types /
> > > Video Device / Device Initialization} +
> > > +The driver SHOULD query device capability by using the
> > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> > > +setup.
> > > +
> > > +\subsection{Device Operation}\label{sec:Device Types / Video Device /
> > > Device Operation} +
> > > +The driver allocates input and output buffers and queues the buffers
> > > +to the device. The device performs operations on the buffers according
> > > +to the function in question.
> > > +
> > > +\subsubsection{Device Operation: Create stream}
> > > +
> > > +To process buffers, the device needs to associate them with a certain
> > > +video stream (essentially, a context). Streams are created by
> > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > > +determined by the device.
> > > +
> > > +\subsubsection{Device Operation: Create buffers}
> > > +
> > > +Buffers are used to store the actual data as well as the relevant
> > > +metadata. Scatter lists are supported, so the buffer doesn't need to
> > > +be contiguous in guest physical memory.
> > > +
> > > +\begin{itemize*}
> > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > > +  resource that is backed by a buffer allocated from the driver's
> > > +  memory.
> > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> > > +  is no longer needed.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Stream parameter control}
> > > +
> > > +\begin{itemize*}
> > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream
> > > parameters
> > > for +  input and output streams from the device.
> > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to
> > > the +  device.
> > > +\item After setting stream parameters, the driver may issue
> > > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output
> > > can be +  changed implicitly by the device during the set operation.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Process buffers}
> > > +
> > > +\begin{itemize*}
> > > +\item If the function and the buffer type require so, write data to
> > > +the buffer memory.
> > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > > +processing in the device.
> > > +\item The request completes asynchronously when the device has
> > > +finished with the buffer.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Buffer processing control}
> > > +
> > > +\begin{itemize*}
> > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> > > +  return all of the already queued buffers.
> > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> > > +  already queued buffers from the input or the output queue. This also
> > > +  includes input or output buffers that can be currently owned by the
> > > +  device's processing pipeline.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Asynchronous events}
> > > +
> > > +While processing buffers, the device can send asynchronous event
> > > +notifications to the driver. The behaviour depends on the exact
> > > +stream. For example, the decoder device sends a resolution change
> > > +event when it encounters new resolution metadata in the stream.
> > > +
> > > +\subsubsection{Device Operation: Request header}
> > > +
> > > +All requests and responses on the control virt queue have a fixed
> > > +header using the following layout structure and definitions:
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_ctrl_type {
> > > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > > +
> > > +        /* request */
> > > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > > +
> > > +        /* response */
> > > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > > +
> > > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > > +};
> > > +
> > > +struct virtio_video_ctrl_hdr {
> > > +        le32 type;
> > > +        le32 stream_id;
> > > +        le32 len; /* Length of the structure in bytes. */
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{type}] is the type of the driver request or the device
> > > +response.
> > > +\item[\field{stream_id}] specifies a target stream.
> > > +\item[\field{len}] is the length of data in bytes, which includes
> > > +length of the header.
> > > +\end{description}
> > > +
> > > +\subsubsection{Device Operation: controlq}
> > > +
> > > +\begin{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > > +supported formats.
> > > +
> > > +The driver uses \field{struct virtio_video_get_capability} to send a
> > > +query request.
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_buf_type {
> > > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > > +};
> > > +
> > > +struct virtio_video_get_capability {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        enum virtio_video_buf_type buf_type;
> > > +};
> > > +\end{lstlisting}
> > > +\begin{description}
> > > +\item[\field{buf_type}] is the buffer type that the driver asks
> > > +information about. The driver MUST set either
> > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > > +
> > > +The device responds a capability by using \field{struct
> > > +virtio_video_get_capability_resp}.
> > > +\begin{lstlisting}
> > > +enum virtio_video_format {
> > > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > > +        /* Raw formats */
> > > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > > +        VIRTIO_VIDEO_FORMAT_YVU420,
> > > +
> > > +        /* Compressed formats */
> > > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> > > +};
> > > +
> > > +enum virtio_video_profile {
> > > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > > +
> > > +        /* H.264 */
> > > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > > +        /* VP8 */
> > > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > > +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > > +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > > +
> > > +        /* VP9 */
> > > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 =
> > > VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_MAX =
> > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > +};
> > > +
> > > +struct virtio_video_format_range {
> > > +        le32 min;
> > > +        le32 max;
> > > +        le32 step;
> > > +        u8 paddings[4];
> > > +};
> > > +
> > > +struct virtio_video_format_desc {
> > > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > > +        le64 mask;
> > > +        struct virtio_video_format_range width;
> > > +        struct virtio_video_format_range height;
> > > +        le32 num_rates;
> > > +        u8 padding[4];
> > > +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> > > +};
> > > +
> > > +struct virtio_video_get_capability_resp {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 num_descs;
> > > +        /* Followed by struct virtio_video_format_desc desc[] */
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +The format description \field{struct virtio_video_format_desc}
> > > +includes the following fields:
> > > +\begin{description}
> > > +\item[\field{format}] specifies an image format. The device MUST set
> > > one
> > > +  of \field{enum virtio_video_format}.
> > > +\item[\field{profile}] specifies a profile of the compressed image
> > > format
> > > +  specified in \field{format}. The driver SHOULD ignore this value if
> > > +  \field{format} is a raw format.
> > 
> > So how should this be used? The spec does not define any way to set
> > profile for the device. It is very important for encoder.
> 
> Thank you for pointing this.
> These points are overlooked, as I didn't care about encoder usage enough.
> 
> After thinking it again, I think it's not a very good idea to include
> supported profiles and levels in a struct for capability.
> This is because these values are available only for limited number of
> formats. Also, it's true that we need to have a way to set these values as
> Dmitry pointed.
> 

Yes, you are right. In fact, the approach of the v1 spec to keep controls 
separately was not correct.

> Instead, it would make more sense to have additional three types of
> controls for profiles, levels, and bitrates:
> (1) QUERY_CONTROL: Query values supported by the device
> (2) GET_CONTROL: Read a value that is set in the device
> (3) SET_CONTROL: Set a value in the device
> 
> These operations are similar to V4L2 controls.
> (1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL, and
> G_CTRL in V4L2, respectively.
> Also, (3) would be similar to enum virtio_video_control_type in the
> virtio-video v1 driver implementation in
> https://markmail.org/message/dwghwdqsbl3gsjxu .
> 
> For QUERY_CONTROL, my idea is like this:
> 
> enum virtio_video_control_type {
>   VIRTIO_VIDEO_CONTROL_UNDEFINED = 0,
> 
>   VIRTIO_VIDEO_CONTROL_BITRATE = 0x100,
>   VIRTIO_VIDEO_CONTROL_PROFILE,
>   VIRTIO_VIDEO_CONTROL_LEVEL,
> };
> 
> struct virtio_video_query_control {
>   struct virtio_video_ctrl_hdr hdr;
>   le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
>   le32 length;
>   /* Followed by additional data.
>    * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE,
>    * the device must pass a codec format like H264 or VP9.
>    * The requred data must be defined in the specification.
>    */
> };
> 
> struct virtio_video_query_control_resp {
>   struct virtio_video_ctrl_hdr hdr;
>   le32 length;
>   u8 padding[4];
>   /* Followed by data corresponds to the specified control.
>    * The type of data must be defined in the spec.
>    * For example, if the driver queries profiles, this part should be
>    * an array of supported profiles of a given format.
>    */
> };
> 
> WDYT?
> 

I think virtio_video_control_type should make sense. But I would disagree with 
the need to have new QUERY_CONTROL and GET_CONTROL.

I assume the set of supported controls is fixed for some particular format on a 
given IP. So we'd propose to include controls into format descriptors, so we 
don't need to QUERY_CONTROL. This way (with 'virtio_video_format_list') we can 
define not-contiguos ranges, e.g. for profiles.

struct virtio_video_format_frame {
        /* As proposed in Keiichi's prev email */
};

virtio_video_format_list {
        le32 num_entries;
        u8 padding[4];
        /* Followed by le64 entries[] */
};

struct virtio_video_format_control {
        le32 type;
        u8 padding[4];
        struct virtio_video_format_list values;
};

struct virtio_video_format_desc {
        le64 mask;
        le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
        le32 planes_layout; /* See the thread [v2 0/1] */
        le32 num_frames;
        le32 num_controls;
        /* Followed by struct virtio_video_format_frame frames[] */
        /* Followed by struct virtio_video_format_control controls[] */
};

SET_CONTROL seems to be mandatory. If it succeeds, we can store the current 
value locally, so there is no need to have GET_CONTROL.

The only exception is the initial (default) control value in the device. But 
with the removal of 'function' and with addition of 'caps' instead, the way to 
provide defaults is gone. So I suppose for formats we'll be just using GET on 
driver start to get the 'defaults'. But the thing is that for formats there 
are other uses for GET, but for controls GET apparently does not make to much 
sense at runtime.

> > Also, shouldn't the profile come together with level? Would make sense for
> > encoders.
> 
> Yeah. So, in the above idea of QUERY_CONTROL, profile should be
> required when querying supported levels.
> 

So probably should be enumerated together, not queried, as per the comment 
above.

> > > +\item[\field{mask}] is a bitset that represents the supported
> > > +  combination of input and output format. If \textit{i}-th bit is set
> > > +  in \field{mask} of \textit{j}-th \field{struct
> > > +  virtio_video_format_desc} for input, the device supports encoding or
> > > +  decoding from the \textit{j}-th input format to \textit{i}-th output
> > > +  format.
> > > +\item[\field{width, height}] represents a range of resolutions
> > > +  supported by the device. If its \field{step} is not applicable, its
> > > +  \field{min} is equal to its \field{max}.
> > > +\item[\field{num_rates}] is the length of an array \field{frame_rates}.
> > > In
> > > case of decoder, the driver SHOULD ignore this value.
> > > +\item[\field{frame_rates}] is an array of supported frame rates.
> > > +\end{description}
> > > +
> > 
> > I'd guess frame rates depend on the resolution as well. This dependency
> > was
> > clear in the v1 spec, but in the v2 there is no dependency anymore. I
> > think we need to update this.
> 
> That's a good point. I missed that dependency when updating the structures.
> So, let me update the structs like the following:
> 
> struct virtio_video_format_frame {
>         struct virtio_video_format_range width;
>         struct virtio_video_format_range height;
>         le32 num_rates;
>         u8 padding[4];
>         /* Followed by struct virtio_video_format_range frame_rates[] */
> };
> 
> struct virtio_video_format_desc {
>         le64 mask;
>         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
>         le32 planes_layout; /* See the thread [v2 0/1] */
>         le32 num_frames;
>         u8 padding[4];
>         /* Followed by struct virtio_video_format_frame frames[] */
> };
>

Yes, I do agree with this approach.

Best regards,
Dmitry.
 
> Best regards,
> Keiichi.
> 
> > Best regards,
> > Dmitry.
> > 
> > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_mem_type {
> > > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > > +};
> > > +
> > > +struct virtio_video_stream_create {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > > +        char debug_name[64];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{in_mem_type}] is a type of buffer management for input
> > > +buffers. The driver MUST set a value in \field{enum
> > > +virtio_video_mem_type}.
> > > +\item[\field{out_mem_type}] is a type of buffer management for output
> > > +buffers. The driver MUST set a value in \field{enum
> > > +virtio_video_mem_type}.
> > > +\item[\field{debug_name}] is a text string for a debug purpose.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_stream_destroy {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > > +  queued buffers through the pipeline.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_stream_drain {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_mem_entry {
> > > +        le64 addr;
> > > +        le32 length;
> > > +        u8 padding[4];
> > > +};
> > > +
> > > +struct virtio_video_resource_create {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 resource_id;
> > > +        le32 nr_entries;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{resource_id}] internal id of the resource.
> > > +\item[\field{nr_entries}] number of \field{struct
> > > +  virtio_video_mem_entry} memory entries.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_resource_destroy {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 resource_id;
> > > +        u8 padding[4];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{resource_id}] internal id of the resource.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > > +queue.
> > > +
> > > +\begin{lstlisting}
> > > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > > +
> > > +struct virtio_video_resource_queue {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 buf_type;
> > > +        le32 resource_id;
> > > +        le64 timestamp;
> > > +        le32 nr_data_size;
> > > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{buf_type}] buf_type of the .
> > > +\item[\field{resource_id}] internal id of the resource.
> > > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > > +  for synchronisation.
> > > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > > +\item[\field{data_size}] number of data bytes within a plane.
> > > +\end{description}
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_buffer_flag {
> > > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > > +        /* Encoder only */
> > > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > > +};
> > > +
> > > +struct virtio_video_resource_queue_resp {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le64 timestamp;
> > > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > > +        le32 size;  /* Encoded size */
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > > +  for synchronisation.
> > > +\item[\field{flags}] mark specific buffers in the sequence.
> > > +\item[\field{size}] data size in the buffer (encoder only).
> > > +\end{description}
> > > +
> > > +The device sends a response to the queue request asynchronously when
> > > +it has finished processing the buffer.
> > > +
> > > +The device SHOULD mark a buffer that triggered a processing error with
> > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > > +
> > > +The device MUST mark the last buffer with the
> > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > +sequence.
> > > +
> > > +In case of encoder, to denote a particular frame type the devie MUST
> > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > > +
> > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> > > +  buffers back from the input or the output queue of the device. The
> > > +  device SHOULD return all of the buffers from the respective queue as
> > > +  soon as possible without pushing the buffers through the processing
> > > +  pipeline.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_queue_clear {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 buf_type;
> > > +        u8 padding[4];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{buf_type}] buffer type.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> > > +  output of a stream.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_plane_format {
> > > +        le32 plane_size;
> > > +        le32 stride;
> > > +        u8 padding[4];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{plane_size}] size of the plane in bytes.
> > > +\item[\field{stride}] stride used for the plane in bytes.
> > > +\end{description}
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_params {
> > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> > > +        le32 frame_width;
> > > +        le32 frame_height;
> > > +        le32 min_buffers;
> > > +        le32 max_buffers;
> > > +        le32 frame_rate;
> > > +        struct virtio_video_crop {
> > > +                le32 left;
> > > +                le32 top;
> > > +                le32 width;
> > > +                le32 height;
> > > +        } crop;
> > > +        le32 num_planes;
> > > +        struct virtio_video_plane_format
> > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{frame_width}] the value to get/set.
> > > +\item[\field{frame_height}] the value to get/set.
> > > +\item[\field{pixel_format}] the value to get/set.
> > > +\item[\field{min_buffers}] minimum buffers required to handle the
> > > +  format (r/o).
> > > +\item[\field{max_buffers}] maximum buffers required to handle the
> > > +  format (r/o).
> > > +\item[\field{frame_rate}] the value to get/set.
> > > +\item[\field{crop}] cropping (composing) rectangle.
> > > +\item[\field{num_planes}] number of planes used to store pixel data
> > > +(r/o).
> > > +\item[\field{plane_formats}] description of each plane.
> > > +\end{description}
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_get_params {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > +};
> > > +
> > > +struct virtio_video_get_params_resp {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        struct virtio_video_params params;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{buf_type}] buffer type.
> > > +\item[\field{params}] parameter values.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> > > +
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_set_params {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        struct virtio_video_params params;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{params}] parameters to set.
> > > +\end{description}
> > > +
> > > +Setting stream parameters might have side effects within the device.
> > > +For example, the device MAY perform alignment of width and height,
> > > +change the number of planes it uses for the format, or do whatever
> > > +changes that are required to continue normal operation using the
> > > +updated parameters. It is up to the driver to check the parameter set
> > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > > +
> > > +\end{description}
> > > +
> > > +\subsubsection{Device Operation: eventq}
> > > +
> > > +The device can report events on the event queue. The driver initially
> > > +populates the queue with device-writeable buffers. When the device
> > > +needs to report an event, it fills a buffer and notifies the driver.
> > > +The driver consumes the report and adds a new buffer to the virtqueue.
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_event_type {
> > > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > > +        /* For all functions */
> > > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > > +        /* For decoder only */
> > > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> > > +};
> > > +
> > > +struct virtio_video_event {
> > > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> > > +        le32 stream_id;
> > > +        u8 padding[4];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{event_type}] type of the triggered event .
> > > +\item[\field{stream_id}] id of the source stream.
> > > +\end{description}
> > > +
> > > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > > +whenever it encounters new resolution data in the stream. This
> > > +includes the case of the initial device configuration after metadata
> > > +has been parsed and the case of dynamic resolution change.



^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2020-01-07 16:50       ` Dmitry Sepp
@ 2020-01-08  6:59         ` Keiichi Watanabe
  2020-01-08 10:00           ` Dmitry Sepp
  0 siblings, 1 reply; 73+ messages in thread
From: Keiichi Watanabe @ 2020-01-08  6:59 UTC (permalink / raw)
  To: Dmitry Sepp
  Cc: uril, virtio-dev, Linux Media Mailing List, Alexandre Courbot,
	Alex Lau, Daniel Vetter, Dylan Reid, Enrico Granata,
	Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, Tomasz Figa

Hi Dmitry,

On Wed, Jan 8, 2020 at 1:50 AM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
>
> Hi Keiichi,
>
> thanks for the updates, please see my comments below.
>
> On Dienstag, 7. Januar 2020 14:24:31 CET Keiichi Watanabe wrote:
> > Hi Dmitry,
> >
> > On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp <dmitry.sepp@opensynergy.com>
> wrote:
> > > Hi,
> > >
> > > a couple of new comments:
> > >
> > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > >
> > > > The virtio video encoder device and decoder device provide
> > > > functionalities
> > > > to encode and decode video stream respectively.
> > > > Though video encoder and decoder are provided as different devices, they
> > > > use a same protocol.
> > > >
> > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > > > ---
> > > >
> > > >  content.tex      |   1 +
> > > >  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
> > > >  2 files changed, 580 insertions(+)
> > > >  create mode 100644 virtio-video.tex
> > > >
> > > > diff --git a/content.tex b/content.tex
> > > > index 556b373..9e56839 100644
> > > > --- a/content.tex
> > > > +++ b/content.tex
> > > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > > > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> > > >
> > > >  \input{virtio-fs.tex}
> > > >  \input{virtio-rpmb.tex}
> > > >
> > > > +\input{virtio-video.tex}
> > > >
> > > >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> > > >
> > > > diff --git a/virtio-video.tex b/virtio-video.tex
> > > > new file mode 100644
> > > > index 0000000..30e728d
> > > > --- /dev/null
> > > > +++ b/virtio-video.tex
> > > > @@ -0,0 +1,579 @@
> > > > +\section{Video Device}\label{sec:Device Types / Video Device}
> > > > +
> > > > +The virtio video encoder device and decoder device are virtual devices
> > > > that +supports encoding and decoding respectively. Though the encoder
> > > > and the decoder +are different devices, they use the same protocol.
> > > > +
> > > > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device
> > > > ID}
> > > > +
> > > > +\begin{description}
> > > > +\item[30] encoder device
> > > > +\item[31] decoder device
> > > > +\end{description}
> > > > +
> > > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device /
> > > > Virtqueues} +
> > > > +\begin{description}
> > > > +\item[0] controlq - queue for sending control commands.
> > > > +\item[1] eventq - queue for sending events happened in the device.
> > > > +\end{description}
> > > > +
> > > > +\subsection{Feature bits}\label{sec:Device Types / Video Device /
> > > > Feature
> > > > bits} +
> > > > +\begin{description}
> > > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used
> > > > for
> > > > video +  buffers.
> > > > +\end{description}
> > > > +
> > > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video
> > > > Device
> > > > / Feature bits} +
> > > > +The device MUST offer at least one of feature bits.
> > > > +
> > > > +\subsection{Device configuration layout}\label{sec:Device Types / Video
> > > > Device / Device configuration layout} +
> > > > +Video device configuration uses the following layout structure:
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_video_config {
> > > > +        le32 max_cap_len;
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> > > > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > > > +  MUST set this value.
> > > > +\end{description}
> > > > +
> > > > +\subsection{Device Initialization}\label{sec:Device Types / Video
> > > > Device /
> > > > Device Initialization} +
> > > > +\devicenormative{\subsubsection}{Device Initialization}{Device Types /
> > > > Video Device / Device Initialization} +
> > > > +The driver SHOULD query device capability by using the
> > > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> > > > +setup.
> > > > +
> > > > +\subsection{Device Operation}\label{sec:Device Types / Video Device /
> > > > Device Operation} +
> > > > +The driver allocates input and output buffers and queues the buffers
> > > > +to the device. The device performs operations on the buffers according
> > > > +to the function in question.
> > > > +
> > > > +\subsubsection{Device Operation: Create stream}
> > > > +
> > > > +To process buffers, the device needs to associate them with a certain
> > > > +video stream (essentially, a context). Streams are created by
> > > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > > > +determined by the device.
> > > > +
> > > > +\subsubsection{Device Operation: Create buffers}
> > > > +
> > > > +Buffers are used to store the actual data as well as the relevant
> > > > +metadata. Scatter lists are supported, so the buffer doesn't need to
> > > > +be contiguous in guest physical memory.
> > > > +
> > > > +\begin{itemize*}
> > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > > > +  resource that is backed by a buffer allocated from the driver's
> > > > +  memory.
> > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> > > > +  is no longer needed.
> > > > +\end{itemize*}
> > > > +
> > > > +\subsubsection{Device Operation: Stream parameter control}
> > > > +
> > > > +\begin{itemize*}
> > > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream
> > > > parameters
> > > > for +  input and output streams from the device.
> > > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to
> > > > the +  device.
> > > > +\item After setting stream parameters, the driver may issue
> > > > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output
> > > > can be +  changed implicitly by the device during the set operation.
> > > > +\end{itemize*}
> > > > +
> > > > +\subsubsection{Device Operation: Process buffers}
> > > > +
> > > > +\begin{itemize*}
> > > > +\item If the function and the buffer type require so, write data to
> > > > +the buffer memory.
> > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > > > +processing in the device.
> > > > +\item The request completes asynchronously when the device has
> > > > +finished with the buffer.
> > > > +\end{itemize*}
> > > > +
> > > > +\subsubsection{Device Operation: Buffer processing control}
> > > > +
> > > > +\begin{itemize*}
> > > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> > > > +  return all of the already queued buffers.
> > > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> > > > +  already queued buffers from the input or the output queue. This also
> > > > +  includes input or output buffers that can be currently owned by the
> > > > +  device's processing pipeline.
> > > > +\end{itemize*}
> > > > +
> > > > +\subsubsection{Device Operation: Asynchronous events}
> > > > +
> > > > +While processing buffers, the device can send asynchronous event
> > > > +notifications to the driver. The behaviour depends on the exact
> > > > +stream. For example, the decoder device sends a resolution change
> > > > +event when it encounters new resolution metadata in the stream.
> > > > +
> > > > +\subsubsection{Device Operation: Request header}
> > > > +
> > > > +All requests and responses on the control virt queue have a fixed
> > > > +header using the following layout structure and definitions:
> > > > +
> > > > +\begin{lstlisting}
> > > > +enum virtio_video_ctrl_type {
> > > > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > > > +
> > > > +        /* request */
> > > > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > > > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > > > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > > > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > > > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > > > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > > > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > > > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > > > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > > > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > > > +
> > > > +        /* response */
> > > > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > > > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > > > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > > > +
> > > > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > > > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > > > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > > > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > > > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > > > +};
> > > > +
> > > > +struct virtio_video_ctrl_hdr {
> > > > +        le32 type;
> > > > +        le32 stream_id;
> > > > +        le32 len; /* Length of the structure in bytes. */
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{type}] is the type of the driver request or the device
> > > > +response.
> > > > +\item[\field{stream_id}] specifies a target stream.
> > > > +\item[\field{len}] is the length of data in bytes, which includes
> > > > +length of the header.
> > > > +\end{description}
> > > > +
> > > > +\subsubsection{Device Operation: controlq}
> > > > +
> > > > +\begin{description}
> > > > +
> > > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > > > +supported formats.
> > > > +
> > > > +The driver uses \field{struct virtio_video_get_capability} to send a
> > > > +query request.
> > > > +
> > > > +\begin{lstlisting}
> > > > +enum virtio_video_buf_type {
> > > > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > > > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > > > +};
> > > > +
> > > > +struct virtio_video_get_capability {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +        enum virtio_video_buf_type buf_type;
> > > > +};
> > > > +\end{lstlisting}
> > > > +\begin{description}
> > > > +\item[\field{buf_type}] is the buffer type that the driver asks
> > > > +information about. The driver MUST set either
> > > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > > > +
> > > > +The device responds a capability by using \field{struct
> > > > +virtio_video_get_capability_resp}.
> > > > +\begin{lstlisting}
> > > > +enum virtio_video_format {
> > > > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > > > +        /* Raw formats */
> > > > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > > > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > > > +        VIRTIO_VIDEO_FORMAT_YVU420,
> > > > +
> > > > +        /* Compressed formats */
> > > > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > > > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > > > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> > > > +};
> > > > +
> > > > +enum virtio_video_profile {
> > > > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > > > +
> > > > +        /* H.264 */
> > > > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > > > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > > > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > > > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > > > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > > > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > > > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > > > +        /* VP8 */
> > > > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > > > +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > > > +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > > > +
> > > > +        /* VP9 */
> > > > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 =
> > > > VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > +        VIRTIO_VIDEO_PROFILE_VP9_MAX =
> > > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > +};
> > > > +
> > > > +struct virtio_video_format_range {
> > > > +        le32 min;
> > > > +        le32 max;
> > > > +        le32 step;
> > > > +        u8 paddings[4];
> > > > +};
> > > > +
> > > > +struct virtio_video_format_desc {
> > > > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > > > +        le64 mask;
> > > > +        struct virtio_video_format_range width;
> > > > +        struct virtio_video_format_range height;
> > > > +        le32 num_rates;
> > > > +        u8 padding[4];
> > > > +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> > > > +};
> > > > +
> > > > +struct virtio_video_get_capability_resp {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +        le32 num_descs;
> > > > +        /* Followed by struct virtio_video_format_desc desc[] */
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +The format description \field{struct virtio_video_format_desc}
> > > > +includes the following fields:
> > > > +\begin{description}
> > > > +\item[\field{format}] specifies an image format. The device MUST set
> > > > one
> > > > +  of \field{enum virtio_video_format}.
> > > > +\item[\field{profile}] specifies a profile of the compressed image
> > > > format
> > > > +  specified in \field{format}. The driver SHOULD ignore this value if
> > > > +  \field{format} is a raw format.
> > >
> > > So how should this be used? The spec does not define any way to set
> > > profile for the device. It is very important for encoder.
> >
> > Thank you for pointing this.
> > These points are overlooked, as I didn't care about encoder usage enough.
> >
> > After thinking it again, I think it's not a very good idea to include
> > supported profiles and levels in a struct for capability.
> > This is because these values are available only for limited number of
> > formats. Also, it's true that we need to have a way to set these values as
> > Dmitry pointed.
> >
>
> Yes, you are right. In fact, the approach of the v1 spec to keep controls
> separately was not correct.
>
> > Instead, it would make more sense to have additional three types of
> > controls for profiles, levels, and bitrates:
> > (1) QUERY_CONTROL: Query values supported by the device
> > (2) GET_CONTROL: Read a value that is set in the device
> > (3) SET_CONTROL: Set a value in the device
> >
> > These operations are similar to V4L2 controls.
> > (1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL, and
> > G_CTRL in V4L2, respectively.
> > Also, (3) would be similar to enum virtio_video_control_type in the
> > virtio-video v1 driver implementation in
> > https://markmail.org/message/dwghwdqsbl3gsjxu .
> >
> > For QUERY_CONTROL, my idea is like this:
> >
> > enum virtio_video_control_type {
> >   VIRTIO_VIDEO_CONTROL_UNDEFINED = 0,
> >
> >   VIRTIO_VIDEO_CONTROL_BITRATE = 0x100,
> >   VIRTIO_VIDEO_CONTROL_PROFILE,
> >   VIRTIO_VIDEO_CONTROL_LEVEL,
> > };
> >
> > struct virtio_video_query_control {
> >   struct virtio_video_ctrl_hdr hdr;
> >   le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
> >   le32 length;
> >   /* Followed by additional data.
> >    * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE,
> >    * the device must pass a codec format like H264 or VP9.
> >    * The requred data must be defined in the specification.
> >    */
> > };
> >
> > struct virtio_video_query_control_resp {
> >   struct virtio_video_ctrl_hdr hdr;
> >   le32 length;
> >   u8 padding[4];
> >   /* Followed by data corresponds to the specified control.
> >    * The type of data must be defined in the spec.
> >    * For example, if the driver queries profiles, this part should be
> >    * an array of supported profiles of a given format.
> >    */
> > };
> >
> > WDYT?
> >
>
> I think virtio_video_control_type should make sense. But I would disagree with
> the need to have new QUERY_CONTROL and GET_CONTROL.
>
> I assume the set of supported controls is fixed for some particular format on a
> given IP. So we'd propose to include controls into format descriptors, so we
> don't need to QUERY_CONTROL. This way (with 'virtio_video_format_list') we can
> define not-contiguos ranges, e.g. for profiles.
>
> struct virtio_video_format_frame {
>         /* As proposed in Keiichi's prev email */
> };
>
> virtio_video_format_list {
>         le32 num_entries;
>         u8 padding[4];
>         /* Followed by le64 entries[] */
> };
>
> struct virtio_video_format_control {
>         le32 type;
>         u8 padding[4];
>         struct virtio_video_format_list values;
> };
>
> struct virtio_video_format_desc {
>         le64 mask;
>         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
>         le32 planes_layout; /* See the thread [v2 0/1] */
>         le32 num_frames;
>         le32 num_controls;
>         /* Followed by struct virtio_video_format_frame frames[] */
>         /* Followed by struct virtio_video_format_control controls[] */
> };

In my understanding, a set of supported levels depends on a profile.
H.264 spec says that levels are specified within each profile.
cf. "0.5 Profiles and levels" in https://www.itu.int/rec/T-REC-H.264-201906-I/en
(V4L2's QUERYMENU doesn't seem to provide a way to query levels for
each profile properly, though)

So, if we want to have supported profiles and levels in format_desc as
your idea, each supported profile should have a list of supported
levels.
I suppose it makes the structure of video_format_desc too complicated.
I'd like to avoid this complexity caused by some specific formats.

Instead, I'd like to keep virtio_video_format_desc minimal and have
QUERY_CONTROL to query format-specific values like profiles and
levels.
This design would make it easy to extend controls when we want to
support new types of formats. We will just need to define new
VIRTIO_VIDEO_CONTROL_*.
What do you think?

Best regards,
Keiichi

>
> SET_CONTROL seems to be mandatory. If it succeeds, we can store the current
> value locally, so there is no need to have GET_CONTROL.
>
> The only exception is the initial (default) control value in the device. But
> with the removal of 'function' and with addition of 'caps' instead, the way to
> provide defaults is gone. So I suppose for formats we'll be just using GET on
> driver start to get the 'defaults'. But the thing is that for formats there
> are other uses for GET, but for controls GET apparently does not make to much
> sense at runtime.
>
> > > Also, shouldn't the profile come together with level? Would make sense for
> > > encoders.
> >
> > Yeah. So, in the above idea of QUERY_CONTROL, profile should be
> > required when querying supported levels.
> >
>
> So probably should be enumerated together, not queried, as per the comment
> above.
>
> > > > +\item[\field{mask}] is a bitset that represents the supported
> > > > +  combination of input and output format. If \textit{i}-th bit is set
> > > > +  in \field{mask} of \textit{j}-th \field{struct
> > > > +  virtio_video_format_desc} for input, the device supports encoding or
> > > > +  decoding from the \textit{j}-th input format to \textit{i}-th output
> > > > +  format.
> > > > +\item[\field{width, height}] represents a range of resolutions
> > > > +  supported by the device. If its \field{step} is not applicable, its
> > > > +  \field{min} is equal to its \field{max}.
> > > > +\item[\field{num_rates}] is the length of an array \field{frame_rates}.
> > > > In
> > > > case of decoder, the driver SHOULD ignore this value.
> > > > +\item[\field{frame_rates}] is an array of supported frame rates.
> > > > +\end{description}
> > > > +
> > >
> > > I'd guess frame rates depend on the resolution as well. This dependency
> > > was
> > > clear in the v1 spec, but in the v2 there is no dependency anymore. I
> > > think we need to update this.
> >
> > That's a good point. I missed that dependency when updating the structures.
> > So, let me update the structs like the following:
> >
> > struct virtio_video_format_frame {
> >         struct virtio_video_format_range width;
> >         struct virtio_video_format_range height;
> >         le32 num_rates;
> >         u8 padding[4];
> >         /* Followed by struct virtio_video_format_range frame_rates[] */
> > };
> >
> > struct virtio_video_format_desc {
> >         le64 mask;
> >         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> >         le32 planes_layout; /* See the thread [v2 0/1] */
> >         le32 num_frames;
> >         u8 padding[4];
> >         /* Followed by struct virtio_video_format_frame frames[] */
> > };
> >
>
> Yes, I do agree with this approach.
>
> Best regards,
> Dmitry.
>
> > Best regards,
> > Keiichi.
> >
> > > Best regards,
> > > Dmitry.
> > >
> > > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > > > +  within the device.
> > > > +
> > > > +\begin{lstlisting}
> > > > +enum virtio_video_mem_type {
> > > > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > > > +};
> > > > +
> > > > +struct virtio_video_stream_create {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > > > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > > > +        char debug_name[64];
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{in_mem_type}] is a type of buffer management for input
> > > > +buffers. The driver MUST set a value in \field{enum
> > > > +virtio_video_mem_type}.
> > > > +\item[\field{out_mem_type}] is a type of buffer management for output
> > > > +buffers. The driver MUST set a value in \field{enum
> > > > +virtio_video_mem_type}.
> > > > +\item[\field{debug_name}] is a text string for a debug purpose.
> > > > +\end{description}
> > > > +
> > > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> > > > +  within the device.
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_video_stream_destroy {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > > > +  queued buffers through the pipeline.
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_video_stream_drain {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > > > +  within the device.
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_video_mem_entry {
> > > > +        le64 addr;
> > > > +        le32 length;
> > > > +        u8 padding[4];
> > > > +};
> > > > +
> > > > +struct virtio_video_resource_create {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +        le32 resource_id;
> > > > +        le32 nr_entries;
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{resource_id}] internal id of the resource.
> > > > +\item[\field{nr_entries}] number of \field{struct
> > > > +  virtio_video_mem_entry} memory entries.
> > > > +\end{description}
> > > > +
> > > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> > > > +  within the device.
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_video_resource_destroy {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +        le32 resource_id;
> > > > +        u8 padding[4];
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{resource_id}] internal id of the resource.
> > > > +\end{description}
> > > > +
> > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > > > +queue.
> > > > +
> > > > +\begin{lstlisting}
> > > > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > > > +
> > > > +struct virtio_video_resource_queue {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +        le32 buf_type;
> > > > +        le32 resource_id;
> > > > +        le64 timestamp;
> > > > +        le32 nr_data_size;
> > > > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{buf_type}] buf_type of the .
> > > > +\item[\field{resource_id}] internal id of the resource.
> > > > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > > > +  for synchronisation.
> > > > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > > > +\item[\field{data_size}] number of data bytes within a plane.
> > > > +\end{description}
> > > > +
> > > > +\begin{lstlisting}
> > > > +enum virtio_video_buffer_flag {
> > > > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > > > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > > > +        /* Encoder only */
> > > > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > > > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > > > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > > > +};
> > > > +
> > > > +struct virtio_video_resource_queue_resp {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +        le64 timestamp;
> > > > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > > > +        le32 size;  /* Encoded size */
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > > > +  for synchronisation.
> > > > +\item[\field{flags}] mark specific buffers in the sequence.
> > > > +\item[\field{size}] data size in the buffer (encoder only).
> > > > +\end{description}
> > > > +
> > > > +The device sends a response to the queue request asynchronously when
> > > > +it has finished processing the buffer.
> > > > +
> > > > +The device SHOULD mark a buffer that triggered a processing error with
> > > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > > > +
> > > > +The device MUST mark the last buffer with the
> > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > > +sequence.
> > > > +
> > > > +In case of encoder, to denote a particular frame type the devie MUST
> > > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > > > +
> > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> > > > +  buffers back from the input or the output queue of the device. The
> > > > +  device SHOULD return all of the buffers from the respective queue as
> > > > +  soon as possible without pushing the buffers through the processing
> > > > +  pipeline.
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_video_queue_clear {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +        le32 buf_type;
> > > > +        u8 padding[4];
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{buf_type}] buffer type.
> > > > +\end{description}
> > > > +
> > > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> > > > +  output of a stream.
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_video_plane_format {
> > > > +        le32 plane_size;
> > > > +        le32 stride;
> > > > +        u8 padding[4];
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{plane_size}] size of the plane in bytes.
> > > > +\item[\field{stride}] stride used for the plane in bytes.
> > > > +\end{description}
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_video_params {
> > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> > > > +        le32 frame_width;
> > > > +        le32 frame_height;
> > > > +        le32 min_buffers;
> > > > +        le32 max_buffers;
> > > > +        le32 frame_rate;
> > > > +        struct virtio_video_crop {
> > > > +                le32 left;
> > > > +                le32 top;
> > > > +                le32 width;
> > > > +                le32 height;
> > > > +        } crop;
> > > > +        le32 num_planes;
> > > > +        struct virtio_video_plane_format
> > > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{frame_width}] the value to get/set.
> > > > +\item[\field{frame_height}] the value to get/set.
> > > > +\item[\field{pixel_format}] the value to get/set.
> > > > +\item[\field{min_buffers}] minimum buffers required to handle the
> > > > +  format (r/o).
> > > > +\item[\field{max_buffers}] maximum buffers required to handle the
> > > > +  format (r/o).
> > > > +\item[\field{frame_rate}] the value to get/set.
> > > > +\item[\field{crop}] cropping (composing) rectangle.
> > > > +\item[\field{num_planes}] number of planes used to store pixel data
> > > > +(r/o).
> > > > +\item[\field{plane_formats}] description of each plane.
> > > > +\end{description}
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_video_get_params {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > > +};
> > > > +
> > > > +struct virtio_video_get_params_resp {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +        struct virtio_video_params params;
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{buf_type}] buffer type.
> > > > +\item[\field{params}] parameter values.
> > > > +\end{description}
> > > > +
> > > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> > > > +
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_video_set_params {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +        struct virtio_video_params params;
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{params}] parameters to set.
> > > > +\end{description}
> > > > +
> > > > +Setting stream parameters might have side effects within the device.
> > > > +For example, the device MAY perform alignment of width and height,
> > > > +change the number of planes it uses for the format, or do whatever
> > > > +changes that are required to continue normal operation using the
> > > > +updated parameters. It is up to the driver to check the parameter set
> > > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > > > +
> > > > +\end{description}
> > > > +
> > > > +\subsubsection{Device Operation: eventq}
> > > > +
> > > > +The device can report events on the event queue. The driver initially
> > > > +populates the queue with device-writeable buffers. When the device
> > > > +needs to report an event, it fills a buffer and notifies the driver.
> > > > +The driver consumes the report and adds a new buffer to the virtqueue.
> > > > +
> > > > +\begin{lstlisting}
> > > > +enum virtio_video_event_type {
> > > > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > > > +        /* For all functions */
> > > > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > > > +        /* For decoder only */
> > > > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> > > > +};
> > > > +
> > > > +struct virtio_video_event {
> > > > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> > > > +        le32 stream_id;
> > > > +        u8 padding[4];
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{event_type}] type of the triggered event .
> > > > +\item[\field{stream_id}] id of the source stream.
> > > > +\end{description}
> > > > +
> > > > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > > > +whenever it encounters new resolution data in the stream. This
> > > > +includes the case of the initial device configuration after metadata
> > > > +has been parsed and the case of dynamic resolution change.
>
>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2020-01-08  6:59         ` Keiichi Watanabe
@ 2020-01-08 10:00           ` Dmitry Sepp
  2020-01-08 12:14             ` Keiichi Watanabe
  0 siblings, 1 reply; 73+ messages in thread
From: Dmitry Sepp @ 2020-01-08 10:00 UTC (permalink / raw)
  To: Keiichi Watanabe
  Cc: uril, virtio-dev, Linux Media Mailing List, Alexandre Courbot,
	Alex Lau, Daniel Vetter, Dylan Reid, Enrico Granata,
	Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, Tomasz Figa

Hi Keiichi,

On Mittwoch, 8. Januar 2020 07:59:22 CET Keiichi Watanabe wrote:
> Hi Dmitry,
> 
> On Wed, Jan 8, 2020 at 1:50 AM Dmitry Sepp <dmitry.sepp@opensynergy.com> 
wrote:
> > Hi Keiichi,
> > 
> > thanks for the updates, please see my comments below.
> > 
> > On Dienstag, 7. Januar 2020 14:24:31 CET Keiichi Watanabe wrote:
> > > Hi Dmitry,
> > > 
> > > On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp
> > > <dmitry.sepp@opensynergy.com>
> > 
> > wrote:
> > > > Hi,
> > > > 
> > > > a couple of new comments:
> > > > 
> > > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> > > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > > 
> > > > > The virtio video encoder device and decoder device provide
> > > > > functionalities
> > > > > to encode and decode video stream respectively.
> > > > > Though video encoder and decoder are provided as different devices,
> > > > > they
> > > > > use a same protocol.
> > > > > 
> > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > > > > ---
> > > > > 
> > > > >  content.tex      |   1 +
> > > > >  virtio-video.tex | 579
> > > > >  +++++++++++++++++++++++++++++++++++++++++++++++
> > > > >  2 files changed, 580 insertions(+)
> > > > >  create mode 100644 virtio-video.tex
> > > > > 
> > > > > diff --git a/content.tex b/content.tex
> > > > > index 556b373..9e56839 100644
> > > > > --- a/content.tex
> > > > > +++ b/content.tex
> > > > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > > > > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> > > > > 
> > > > >  \input{virtio-fs.tex}
> > > > >  \input{virtio-rpmb.tex}
> > > > > 
> > > > > +\input{virtio-video.tex}
> > > > > 
> > > > >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> > > > > 
> > > > > diff --git a/virtio-video.tex b/virtio-video.tex
> > > > > new file mode 100644
> > > > > index 0000000..30e728d
> > > > > --- /dev/null
> > > > > +++ b/virtio-video.tex
> > > > > @@ -0,0 +1,579 @@
> > > > > +\section{Video Device}\label{sec:Device Types / Video Device}
> > > > > +
> > > > > +The virtio video encoder device and decoder device are virtual
> > > > > devices
> > > > > that +supports encoding and decoding respectively. Though the
> > > > > encoder
> > > > > and the decoder +are different devices, they use the same protocol.
> > > > > +
> > > > > +\subsection{Device ID}\label{sec:Device Types / Video Device /
> > > > > Device
> > > > > ID}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[30] encoder device
> > > > > +\item[31] decoder device
> > > > > +\end{description}
> > > > > +
> > > > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device /
> > > > > Virtqueues} +
> > > > > +\begin{description}
> > > > > +\item[0] controlq - queue for sending control commands.
> > > > > +\item[1] eventq - queue for sending events happened in the device.
> > > > > +\end{description}
> > > > > +
> > > > > +\subsection{Feature bits}\label{sec:Device Types / Video Device /
> > > > > Feature
> > > > > bits} +
> > > > > +\begin{description}
> > > > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be
> > > > > used
> > > > > for
> > > > > video +  buffers.
> > > > > +\end{description}
> > > > > +
> > > > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video
> > > > > Device
> > > > > / Feature bits} +
> > > > > +The device MUST offer at least one of feature bits.
> > > > > +
> > > > > +\subsection{Device configuration layout}\label{sec:Device Types /
> > > > > Video
> > > > > Device / Device configuration layout} +
> > > > > +Video device configuration uses the following layout structure:
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_video_config {
> > > > > +        le32 max_cap_len;
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{max_cap_len}] defines the maximum length of a
> > > > > descriptor
> > > > > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > > > > +  MUST set this value.
> > > > > +\end{description}
> > > > > +
> > > > > +\subsection{Device Initialization}\label{sec:Device Types / Video
> > > > > Device /
> > > > > Device Initialization} +
> > > > > +\devicenormative{\subsubsection}{Device Initialization}{Device
> > > > > Types /
> > > > > Video Device / Device Initialization} +
> > > > > +The driver SHOULD query device capability by using the
> > > > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the
> > > > > initial
> > > > > +setup.
> > > > > +
> > > > > +\subsection{Device Operation}\label{sec:Device Types / Video Device
> > > > > /
> > > > > Device Operation} +
> > > > > +The driver allocates input and output buffers and queues the
> > > > > buffers
> > > > > +to the device. The device performs operations on the buffers
> > > > > according
> > > > > +to the function in question.
> > > > > +
> > > > > +\subsubsection{Device Operation: Create stream}
> > > > > +
> > > > > +To process buffers, the device needs to associate them with a
> > > > > certain
> > > > > +video stream (essentially, a context). Streams are created by
> > > > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > > > > +determined by the device.
> > > > > +
> > > > > +\subsubsection{Device Operation: Create buffers}
> > > > > +
> > > > > +Buffers are used to store the actual data as well as the relevant
> > > > > +metadata. Scatter lists are supported, so the buffer doesn't need
> > > > > to
> > > > > +be contiguous in guest physical memory.
> > > > > +
> > > > > +\begin{itemize*}
> > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > > > > +  resource that is backed by a buffer allocated from the driver's
> > > > > +  memory.
> > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource
> > > > > that
> > > > > +  is no longer needed.
> > > > > +\end{itemize*}
> > > > > +
> > > > > +\subsubsection{Device Operation: Stream parameter control}
> > > > > +
> > > > > +\begin{itemize*}
> > > > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream
> > > > > parameters
> > > > > for +  input and output streams from the device.
> > > > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream
> > > > > parameters to
> > > > > the +  device.
> > > > > +\item After setting stream parameters, the driver may issue
> > > > > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and
> > > > > output
> > > > > can be +  changed implicitly by the device during the set operation.
> > > > > +\end{itemize*}
> > > > > +
> > > > > +\subsubsection{Device Operation: Process buffers}
> > > > > +
> > > > > +\begin{itemize*}
> > > > > +\item If the function and the buffer type require so, write data to
> > > > > +the buffer memory.
> > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > > > > +processing in the device.
> > > > > +\item The request completes asynchronously when the device has
> > > > > +finished with the buffer.
> > > > > +\end{itemize*}
> > > > > +
> > > > > +\subsubsection{Device Operation: Buffer processing control}
> > > > > +
> > > > > +\begin{itemize*}
> > > > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process
> > > > > and
> > > > > +  return all of the already queued buffers.
> > > > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return
> > > > > back
> > > > > +  already queued buffers from the input or the output queue. This
> > > > > also
> > > > > +  includes input or output buffers that can be currently owned by
> > > > > the
> > > > > +  device's processing pipeline.
> > > > > +\end{itemize*}
> > > > > +
> > > > > +\subsubsection{Device Operation: Asynchronous events}
> > > > > +
> > > > > +While processing buffers, the device can send asynchronous event
> > > > > +notifications to the driver. The behaviour depends on the exact
> > > > > +stream. For example, the decoder device sends a resolution change
> > > > > +event when it encounters new resolution metadata in the stream.
> > > > > +
> > > > > +\subsubsection{Device Operation: Request header}
> > > > > +
> > > > > +All requests and responses on the control virt queue have a fixed
> > > > > +header using the following layout structure and definitions:
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +enum virtio_video_ctrl_type {
> > > > > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > > > > +
> > > > > +        /* request */
> > > > > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > > > > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > > > > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > > > > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > > > > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > > > > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > > > > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > > > > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > > > > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > > > > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > > > > +
> > > > > +        /* response */
> > > > > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > > > > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > > > > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > > > > +
> > > > > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > > > > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > > > > +};
> > > > > +
> > > > > +struct virtio_video_ctrl_hdr {
> > > > > +        le32 type;
> > > > > +        le32 stream_id;
> > > > > +        le32 len; /* Length of the structure in bytes. */
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{type}] is the type of the driver request or the device
> > > > > +response.
> > > > > +\item[\field{stream_id}] specifies a target stream.
> > > > > +\item[\field{len}] is the length of data in bytes, which includes
> > > > > +length of the header.
> > > > > +\end{description}
> > > > > +
> > > > > +\subsubsection{Device Operation: controlq}
> > > > > +
> > > > > +\begin{description}
> > > > > +
> > > > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > > > > +supported formats.
> > > > > +
> > > > > +The driver uses \field{struct virtio_video_get_capability} to send
> > > > > a
> > > > > +query request.
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +enum virtio_video_buf_type {
> > > > > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > > > > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > > > > +};
> > > > > +
> > > > > +struct virtio_video_get_capability {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +        enum virtio_video_buf_type buf_type;
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +\begin{description}
> > > > > +\item[\field{buf_type}] is the buffer type that the driver asks
> > > > > +information about. The driver MUST set either
> > > > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > > > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > > > > +
> > > > > +The device responds a capability by using \field{struct
> > > > > +virtio_video_get_capability_resp}.
> > > > > +\begin{lstlisting}
> > > > > +enum virtio_video_format {
> > > > > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > > > > +        /* Raw formats */
> > > > > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > > > > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > > > > +        VIRTIO_VIDEO_FORMAT_YVU420,
> > > > > +
> > > > > +        /* Compressed formats */
> > > > > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > > > > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > > > > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> > > > > +};
> > > > > +
> > > > > +enum virtio_video_profile {
> > > > > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > > > > +
> > > > > +        /* H.264 */
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > > > > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > > > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > > > > +        /* VP8 */
> > > > > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > > > > +        VIRTIO_VIDEO_PROFILE_VP8_ANY =
> > > > > VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > > > > +        VIRTIO_VIDEO_PROFILE_VP8_MAX =
> > > > > VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > > > > +
> > > > > +        /* VP9 */
> > > > > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 =
> > > > > VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > > +        VIRTIO_VIDEO_PROFILE_VP9_MAX =
> > > > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > > +};
> > > > > +
> > > > > +struct virtio_video_format_range {
> > > > > +        le32 min;
> > > > > +        le32 max;
> > > > > +        le32 step;
> > > > > +        u8 paddings[4];
> > > > > +};
> > > > > +
> > > > > +struct virtio_video_format_desc {
> > > > > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > > > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > > > > +        le64 mask;
> > > > > +        struct virtio_video_format_range width;
> > > > > +        struct virtio_video_format_range height;
> > > > > +        le32 num_rates;
> > > > > +        u8 padding[4];
> > > > > +        /* Followed by struct virtio_video_frame_rate frame_rates[]
> > > > > */
> > > > > +};
> > > > > +
> > > > > +struct virtio_video_get_capability_resp {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +        le32 num_descs;
> > > > > +        /* Followed by struct virtio_video_format_desc desc[] */
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +The format description \field{struct virtio_video_format_desc}
> > > > > +includes the following fields:
> > > > > +\begin{description}
> > > > > +\item[\field{format}] specifies an image format. The device MUST
> > > > > set
> > > > > one
> > > > > +  of \field{enum virtio_video_format}.
> > > > > +\item[\field{profile}] specifies a profile of the compressed image
> > > > > format
> > > > > +  specified in \field{format}. The driver SHOULD ignore this value
> > > > > if
> > > > > +  \field{format} is a raw format.
> > > > 
> > > > So how should this be used? The spec does not define any way to set
> > > > profile for the device. It is very important for encoder.
> > > 
> > > Thank you for pointing this.
> > > These points are overlooked, as I didn't care about encoder usage
> > > enough.
> > > 
> > > After thinking it again, I think it's not a very good idea to include
> > > supported profiles and levels in a struct for capability.
> > > This is because these values are available only for limited number of
> > > formats. Also, it's true that we need to have a way to set these values
> > > as
> > > Dmitry pointed.
> > 
> > Yes, you are right. In fact, the approach of the v1 spec to keep controls
> > separately was not correct.
> > 
> > > Instead, it would make more sense to have additional three types of
> > > controls for profiles, levels, and bitrates:
> > > (1) QUERY_CONTROL: Query values supported by the device
> > > (2) GET_CONTROL: Read a value that is set in the device
> > > (3) SET_CONTROL: Set a value in the device
> > > 
> > > These operations are similar to V4L2 controls.
> > > (1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL, and
> > > G_CTRL in V4L2, respectively.
> > > Also, (3) would be similar to enum virtio_video_control_type in the
> > > virtio-video v1 driver implementation in
> > > https://markmail.org/message/dwghwdqsbl3gsjxu .
> > > 
> > > For QUERY_CONTROL, my idea is like this:
> > > 
> > > enum virtio_video_control_type {
> > > 
> > >   VIRTIO_VIDEO_CONTROL_UNDEFINED = 0,
> > >   
> > >   VIRTIO_VIDEO_CONTROL_BITRATE = 0x100,
> > >   VIRTIO_VIDEO_CONTROL_PROFILE,
> > >   VIRTIO_VIDEO_CONTROL_LEVEL,
> > > 
> > > };
> > > 
> > > struct virtio_video_query_control {
> > > 
> > >   struct virtio_video_ctrl_hdr hdr;
> > >   le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
> > >   le32 length;
> > >   /* Followed by additional data.
> > >   
> > >    * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE,
> > >    * the device must pass a codec format like H264 or VP9.
> > >    * The requred data must be defined in the specification.
> > >    */
> > > 
> > > };
> > > 
> > > struct virtio_video_query_control_resp {
> > > 
> > >   struct virtio_video_ctrl_hdr hdr;
> > >   le32 length;
> > >   u8 padding[4];
> > >   /* Followed by data corresponds to the specified control.
> > >   
> > >    * The type of data must be defined in the spec.
> > >    * For example, if the driver queries profiles, this part should be
> > >    * an array of supported profiles of a given format.
> > >    */
> > > 
> > > };
> > > 
> > > WDYT?
> > 
> > I think virtio_video_control_type should make sense. But I would disagree
> > with the need to have new QUERY_CONTROL and GET_CONTROL.
> > 
> > I assume the set of supported controls is fixed for some particular format
> > on a given IP. So we'd propose to include controls into format
> > descriptors, so we don't need to QUERY_CONTROL. This way (with
> > 'virtio_video_format_list') we can define not-contiguos ranges, e.g. for
> > profiles.
> > 
> > struct virtio_video_format_frame {
> > 
> >         /* As proposed in Keiichi's prev email */
> > 
> > };
> > 
> > virtio_video_format_list {
> > 
> >         le32 num_entries;
> >         u8 padding[4];
> >         /* Followed by le64 entries[] */
> > 
> > };
> > 
> > struct virtio_video_format_control {
> > 
> >         le32 type;
> >         u8 padding[4];
> >         struct virtio_video_format_list values;
> > 
> > };
> > 
> > struct virtio_video_format_desc {
> > 
> >         le64 mask;
> >         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> >         le32 planes_layout; /* See the thread [v2 0/1] */
> >         le32 num_frames;
> >         le32 num_controls;
> >         /* Followed by struct virtio_video_format_frame frames[] */
> >         /* Followed by struct virtio_video_format_control controls[] */
> > 
> > };
> 
> In my understanding, a set of supported levels depends on a profile.
> H.264 spec says that levels are specified within each profile.
> cf. "0.5 Profiles and levels" in
> https://www.itu.int/rec/T-REC-H.264-201906-I/en (V4L2's QUERYMENU doesn't
> seem to provide a way to query levels for each profile properly, though)
> 
> So, if we want to have supported profiles and levels in format_desc as
> your idea, each supported profile should have a list of supported
> levels.
> I suppose it makes the structure of video_format_desc too complicated.
> I'd like to avoid this complexity caused by some specific formats.
> 
> Instead, I'd like to keep virtio_video_format_desc minimal and have
> QUERY_CONTROL to query format-specific values like profiles and
> levels.
> This design would make it easy to extend controls when we want to
> support new types of formats. We will just need to define new
> VIRTIO_VIDEO_CONTROL_*.
> What do you think?

Yes, I think we can keep it this way, this indeed provides more flexibility and 
this way we won't need to modify the 'format' parsing logic when more controls 
are added.

Best regards,
Dmitry.

> 
> Best regards,
> Keiichi
> 
> > SET_CONTROL seems to be mandatory. If it succeeds, we can store the
> > current
> > value locally, so there is no need to have GET_CONTROL.
> > 
> > The only exception is the initial (default) control value in the device.
> > But with the removal of 'function' and with addition of 'caps' instead,
> > the way to provide defaults is gone. So I suppose for formats we'll be
> > just using GET on driver start to get the 'defaults'. But the thing is
> > that for formats there are other uses for GET, but for controls GET
> > apparently does not make to much sense at runtime.
> > 
> > > > Also, shouldn't the profile come together with level? Would make sense
> > > > for
> > > > encoders.
> > > 
> > > Yeah. So, in the above idea of QUERY_CONTROL, profile should be
> > > required when querying supported levels.
> > 
> > So probably should be enumerated together, not queried, as per the comment
> > above.
> > 
> > > > > +\item[\field{mask}] is a bitset that represents the supported
> > > > > +  combination of input and output format. If \textit{i}-th bit is
> > > > > set
> > > > > +  in \field{mask} of \textit{j}-th \field{struct
> > > > > +  virtio_video_format_desc} for input, the device supports encoding
> > > > > or
> > > > > +  decoding from the \textit{j}-th input format to \textit{i}-th
> > > > > output
> > > > > +  format.
> > > > > +\item[\field{width, height}] represents a range of resolutions
> > > > > +  supported by the device. If its \field{step} is not applicable,
> > > > > its
> > > > > +  \field{min} is equal to its \field{max}.
> > > > > +\item[\field{num_rates}] is the length of an array
> > > > > \field{frame_rates}.
> > > > > In
> > > > > case of decoder, the driver SHOULD ignore this value.
> > > > > +\item[\field{frame_rates}] is an array of supported frame rates.
> > > > > +\end{description}
> > > > > +
> > > > 
> > > > I'd guess frame rates depend on the resolution as well. This
> > > > dependency
> > > > was
> > > > clear in the v1 spec, but in the v2 there is no dependency anymore. I
> > > > think we need to update this.
> > > 
> > > That's a good point. I missed that dependency when updating the
> > > structures.
> > > So, let me update the structs like the following:
> > > 
> > > struct virtio_video_format_frame {
> > > 
> > >         struct virtio_video_format_range width;
> > >         struct virtio_video_format_range height;
> > >         le32 num_rates;
> > >         u8 padding[4];
> > >         /* Followed by struct virtio_video_format_range frame_rates[] */
> > > 
> > > };
> > > 
> > > struct virtio_video_format_desc {
> > > 
> > >         le64 mask;
> > >         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> > >         le32 planes_layout; /* See the thread [v2 0/1] */
> > >         le32 num_frames;
> > >         u8 padding[4];
> > >         /* Followed by struct virtio_video_format_frame frames[] */
> > > 
> > > };
> > 
> > Yes, I do agree with this approach.
> > 
> > Best regards,
> > Dmitry.
> > 
> > > Best regards,
> > > Keiichi.
> > > 
> > > > Best regards,
> > > > Dmitry.
> > > > 
> > > > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > > > > +  within the device.
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +enum virtio_video_mem_type {
> > > > > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > > > > +};
> > > > > +
> > > > > +struct virtio_video_stream_create {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types
> > > > > */
> > > > > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types
> > > > > */
> > > > > +        char debug_name[64];
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{in_mem_type}] is a type of buffer management for input
> > > > > +buffers. The driver MUST set a value in \field{enum
> > > > > +virtio_video_mem_type}.
> > > > > +\item[\field{out_mem_type}] is a type of buffer management for
> > > > > output
> > > > > +buffers. The driver MUST set a value in \field{enum
> > > > > +virtio_video_mem_type}.
> > > > > +\item[\field{debug_name}] is a text string for a debug purpose.
> > > > > +\end{description}
> > > > > +
> > > > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream
> > > > > (context)
> > > > > +  within the device.
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_video_stream_destroy {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > > > > +  queued buffers through the pipeline.
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_video_stream_drain {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > > > > +  within the device.
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_video_mem_entry {
> > > > > +        le64 addr;
> > > > > +        le32 length;
> > > > > +        u8 padding[4];
> > > > > +};
> > > > > +
> > > > > +struct virtio_video_resource_create {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +        le32 resource_id;
> > > > > +        le32 nr_entries;
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > +\item[\field{nr_entries}] number of \field{struct
> > > > > +  virtio_video_mem_entry} memory entries.
> > > > > +\end{description}
> > > > > +
> > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource
> > > > > descriptor
> > > > > +  within the device.
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_video_resource_destroy {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +        le32 resource_id;
> > > > > +        u8 padding[4];
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > +\end{description}
> > > > > +
> > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > > > > +queue.
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > > > > +
> > > > > +struct virtio_video_resource_queue {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +        le32 buf_type;
> > > > > +        le32 resource_id;
> > > > > +        le64 timestamp;
> > > > > +        le32 nr_data_size;
> > > > > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{buf_type}] buf_type of the .
> > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > +\item[\field{timestamp}] an abstract sequence counter that can be
> > > > > used
> > > > > +  for synchronisation.
> > > > > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > > > > +\item[\field{data_size}] number of data bytes within a plane.
> > > > > +\end{description}
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +enum virtio_video_buffer_flag {
> > > > > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > > > > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > > > > +        /* Encoder only */
> > > > > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > > > > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > > > > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > > > > +};
> > > > > +
> > > > > +struct virtio_video_resource_queue_resp {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +        le64 timestamp;
> > > > > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > > > > +        le32 size;  /* Encoded size */
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{timestamp}] an abstract sequence counter that can be
> > > > > used
> > > > > +  for synchronisation.
> > > > > +\item[\field{flags}] mark specific buffers in the sequence.
> > > > > +\item[\field{size}] data size in the buffer (encoder only).
> > > > > +\end{description}
> > > > > +
> > > > > +The device sends a response to the queue request asynchronously
> > > > > when
> > > > > +it has finished processing the buffer.
> > > > > +
> > > > > +The device SHOULD mark a buffer that triggered a processing error
> > > > > with
> > > > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > > > > +
> > > > > +The device MUST mark the last buffer with the
> > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > > > +sequence.
> > > > > +
> > > > > +In case of encoder, to denote a particular frame type the devie
> > > > > MUST
> > > > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > > > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > > > > +
> > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> > > > > +  buffers back from the input or the output queue of the device.
> > > > > The
> > > > > +  device SHOULD return all of the buffers from the respective queue
> > > > > as
> > > > > +  soon as possible without pushing the buffers through the
> > > > > processing
> > > > > +  pipeline.
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_video_queue_clear {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +        le32 buf_type;
> > > > > +        u8 padding[4];
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{buf_type}] buffer type.
> > > > > +\end{description}
> > > > > +
> > > > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> > > > > +  output of a stream.
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_video_plane_format {
> > > > > +        le32 plane_size;
> > > > > +        le32 stride;
> > > > > +        u8 padding[4];
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{plane_size}] size of the plane in bytes.
> > > > > +\item[\field{stride}] stride used for the plane in bytes.
> > > > > +\end{description}
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_video_params {
> > > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > > > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> > > > > +        le32 frame_width;
> > > > > +        le32 frame_height;
> > > > > +        le32 min_buffers;
> > > > > +        le32 max_buffers;
> > > > > +        le32 frame_rate;
> > > > > +        struct virtio_video_crop {
> > > > > +                le32 left;
> > > > > +                le32 top;
> > > > > +                le32 width;
> > > > > +                le32 height;
> > > > > +        } crop;
> > > > > +        le32 num_planes;
> > > > > +        struct virtio_video_plane_format
> > > > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{frame_width}] the value to get/set.
> > > > > +\item[\field{frame_height}] the value to get/set.
> > > > > +\item[\field{pixel_format}] the value to get/set.
> > > > > +\item[\field{min_buffers}] minimum buffers required to handle the
> > > > > +  format (r/o).
> > > > > +\item[\field{max_buffers}] maximum buffers required to handle the
> > > > > +  format (r/o).
> > > > > +\item[\field{frame_rate}] the value to get/set.
> > > > > +\item[\field{crop}] cropping (composing) rectangle.
> > > > > +\item[\field{num_planes}] number of planes used to store pixel data
> > > > > +(r/o).
> > > > > +\item[\field{plane_formats}] description of each plane.
> > > > > +\end{description}
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_video_get_params {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > > > +};
> > > > > +
> > > > > +struct virtio_video_get_params_resp {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +        struct virtio_video_params params;
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{buf_type}] buffer type.
> > > > > +\item[\field{params}] parameter values.
> > > > > +\end{description}
> > > > > +
> > > > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> > > > > +
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_video_set_params {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +        struct virtio_video_params params;
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{params}] parameters to set.
> > > > > +\end{description}
> > > > > +
> > > > > +Setting stream parameters might have side effects within the
> > > > > device.
> > > > > +For example, the device MAY perform alignment of width and height,
> > > > > +change the number of planes it uses for the format, or do whatever
> > > > > +changes that are required to continue normal operation using the
> > > > > +updated parameters. It is up to the driver to check the parameter
> > > > > set
> > > > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > > > > +
> > > > > +\end{description}
> > > > > +
> > > > > +\subsubsection{Device Operation: eventq}
> > > > > +
> > > > > +The device can report events on the event queue. The driver
> > > > > initially
> > > > > +populates the queue with device-writeable buffers. When the device
> > > > > +needs to report an event, it fills a buffer and notifies the
> > > > > driver.
> > > > > +The driver consumes the report and adds a new buffer to the
> > > > > virtqueue.
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +enum virtio_video_event_type {
> > > > > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > > > > +        /* For all functions */
> > > > > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > > > > +        /* For decoder only */
> > > > > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> > > > > +};
> > > > > +
> > > > > +struct virtio_video_event {
> > > > > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> > > > > +        le32 stream_id;
> > > > > +        u8 padding[4];
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{event_type}] type of the triggered event .
> > > > > +\item[\field{stream_id}] id of the source stream.
> > > > > +\end{description}
> > > > > +
> > > > > +The device MUST send
> > > > > VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > > > > +whenever it encounters new resolution data in the stream. This
> > > > > +includes the case of the initial device configuration after
> > > > > metadata
> > > > > +has been parsed and the case of dynamic resolution change.



^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2020-01-08 10:00           ` Dmitry Sepp
@ 2020-01-08 12:14             ` Keiichi Watanabe
  2020-01-08 12:46               ` Tomasz Figa
  0 siblings, 1 reply; 73+ messages in thread
From: Keiichi Watanabe @ 2020-01-08 12:14 UTC (permalink / raw)
  To: Dmitry Sepp
  Cc: uril, virtio-dev, Linux Media Mailing List, Alexandre Courbot,
	Alex Lau, Daniel Vetter, Dylan Reid, Enrico Granata,
	Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, Tomasz Figa

Hi Dmitry,

On Wed, Jan 8, 2020 at 7:00 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
>
> Hi Keiichi,
>
> On Mittwoch, 8. Januar 2020 07:59:22 CET Keiichi Watanabe wrote:
> > Hi Dmitry,
> >
> > On Wed, Jan 8, 2020 at 1:50 AM Dmitry Sepp <dmitry.sepp@opensynergy.com>
> wrote:
> > > Hi Keiichi,
> > >
> > > thanks for the updates, please see my comments below.
> > >
> > > On Dienstag, 7. Januar 2020 14:24:31 CET Keiichi Watanabe wrote:
> > > > Hi Dmitry,
> > > >
> > > > On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp
> > > > <dmitry.sepp@opensynergy.com>
> > >
> > > wrote:
> > > > > Hi,
> > > > >
> > > > > a couple of new comments:
> > > > >
> > > > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> > > > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > > >
> > > > > > The virtio video encoder device and decoder device provide
> > > > > > functionalities
> > > > > > to encode and decode video stream respectively.
> > > > > > Though video encoder and decoder are provided as different devices,
> > > > > > they
> > > > > > use a same protocol.
> > > > > >
> > > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > > > > > ---
> > > > > >
> > > > > >  content.tex      |   1 +
> > > > > >  virtio-video.tex | 579
> > > > > >  +++++++++++++++++++++++++++++++++++++++++++++++
> > > > > >  2 files changed, 580 insertions(+)
> > > > > >  create mode 100644 virtio-video.tex
> > > > > >
> > > > > > diff --git a/content.tex b/content.tex
> > > > > > index 556b373..9e56839 100644
> > > > > > --- a/content.tex
> > > > > > +++ b/content.tex
> > > > > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > > > > > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> > > > > >
> > > > > >  \input{virtio-fs.tex}
> > > > > >  \input{virtio-rpmb.tex}
> > > > > >
> > > > > > +\input{virtio-video.tex}
> > > > > >
> > > > > >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> > > > > >
> > > > > > diff --git a/virtio-video.tex b/virtio-video.tex
> > > > > > new file mode 100644
> > > > > > index 0000000..30e728d
> > > > > > --- /dev/null
> > > > > > +++ b/virtio-video.tex
> > > > > > @@ -0,0 +1,579 @@
> > > > > > +\section{Video Device}\label{sec:Device Types / Video Device}
> > > > > > +
> > > > > > +The virtio video encoder device and decoder device are virtual
> > > > > > devices
> > > > > > that +supports encoding and decoding respectively. Though the
> > > > > > encoder
> > > > > > and the decoder +are different devices, they use the same protocol.
> > > > > > +
> > > > > > +\subsection{Device ID}\label{sec:Device Types / Video Device /
> > > > > > Device
> > > > > > ID}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[30] encoder device
> > > > > > +\item[31] decoder device
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device /
> > > > > > Virtqueues} +
> > > > > > +\begin{description}
> > > > > > +\item[0] controlq - queue for sending control commands.
> > > > > > +\item[1] eventq - queue for sending events happened in the device.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\subsection{Feature bits}\label{sec:Device Types / Video Device /
> > > > > > Feature
> > > > > > bits} +
> > > > > > +\begin{description}
> > > > > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be
> > > > > > used
> > > > > > for
> > > > > > video +  buffers.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video
> > > > > > Device
> > > > > > / Feature bits} +
> > > > > > +The device MUST offer at least one of feature bits.
> > > > > > +
> > > > > > +\subsection{Device configuration layout}\label{sec:Device Types /
> > > > > > Video
> > > > > > Device / Device configuration layout} +
> > > > > > +Video device configuration uses the following layout structure:
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +struct virtio_video_config {
> > > > > > +        le32 max_cap_len;
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{max_cap_len}] defines the maximum length of a
> > > > > > descriptor
> > > > > > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > > > > > +  MUST set this value.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\subsection{Device Initialization}\label{sec:Device Types / Video
> > > > > > Device /
> > > > > > Device Initialization} +
> > > > > > +\devicenormative{\subsubsection}{Device Initialization}{Device
> > > > > > Types /
> > > > > > Video Device / Device Initialization} +
> > > > > > +The driver SHOULD query device capability by using the
> > > > > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the
> > > > > > initial
> > > > > > +setup.
> > > > > > +
> > > > > > +\subsection{Device Operation}\label{sec:Device Types / Video Device
> > > > > > /
> > > > > > Device Operation} +
> > > > > > +The driver allocates input and output buffers and queues the
> > > > > > buffers
> > > > > > +to the device. The device performs operations on the buffers
> > > > > > according
> > > > > > +to the function in question.
> > > > > > +
> > > > > > +\subsubsection{Device Operation: Create stream}
> > > > > > +
> > > > > > +To process buffers, the device needs to associate them with a
> > > > > > certain
> > > > > > +video stream (essentially, a context). Streams are created by
> > > > > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > > > > > +determined by the device.
> > > > > > +
> > > > > > +\subsubsection{Device Operation: Create buffers}
> > > > > > +
> > > > > > +Buffers are used to store the actual data as well as the relevant
> > > > > > +metadata. Scatter lists are supported, so the buffer doesn't need
> > > > > > to
> > > > > > +be contiguous in guest physical memory.
> > > > > > +
> > > > > > +\begin{itemize*}
> > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > > > > > +  resource that is backed by a buffer allocated from the driver's
> > > > > > +  memory.
> > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource
> > > > > > that
> > > > > > +  is no longer needed.
> > > > > > +\end{itemize*}
> > > > > > +
> > > > > > +\subsubsection{Device Operation: Stream parameter control}
> > > > > > +
> > > > > > +\begin{itemize*}
> > > > > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream
> > > > > > parameters
> > > > > > for +  input and output streams from the device.
> > > > > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream
> > > > > > parameters to
> > > > > > the +  device.
> > > > > > +\item After setting stream parameters, the driver may issue
> > > > > > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and
> > > > > > output
> > > > > > can be +  changed implicitly by the device during the set operation.
> > > > > > +\end{itemize*}
> > > > > > +
> > > > > > +\subsubsection{Device Operation: Process buffers}
> > > > > > +
> > > > > > +\begin{itemize*}
> > > > > > +\item If the function and the buffer type require so, write data to
> > > > > > +the buffer memory.
> > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > > > > > +processing in the device.
> > > > > > +\item The request completes asynchronously when the device has
> > > > > > +finished with the buffer.
> > > > > > +\end{itemize*}
> > > > > > +
> > > > > > +\subsubsection{Device Operation: Buffer processing control}
> > > > > > +
> > > > > > +\begin{itemize*}
> > > > > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process
> > > > > > and
> > > > > > +  return all of the already queued buffers.
> > > > > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return
> > > > > > back
> > > > > > +  already queued buffers from the input or the output queue. This
> > > > > > also
> > > > > > +  includes input or output buffers that can be currently owned by
> > > > > > the
> > > > > > +  device's processing pipeline.
> > > > > > +\end{itemize*}
> > > > > > +
> > > > > > +\subsubsection{Device Operation: Asynchronous events}
> > > > > > +
> > > > > > +While processing buffers, the device can send asynchronous event
> > > > > > +notifications to the driver. The behaviour depends on the exact
> > > > > > +stream. For example, the decoder device sends a resolution change
> > > > > > +event when it encounters new resolution metadata in the stream.
> > > > > > +
> > > > > > +\subsubsection{Device Operation: Request header}
> > > > > > +
> > > > > > +All requests and responses on the control virt queue have a fixed
> > > > > > +header using the following layout structure and definitions:
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +enum virtio_video_ctrl_type {
> > > > > > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > > > > > +
> > > > > > +        /* request */
> > > > > > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > > > > > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > > > > > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > > > > > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > > > > > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > > > > > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > > > > > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > > > > > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > > > > > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > > > > > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > > > > > +
> > > > > > +        /* response */
> > > > > > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > > > > > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > > > > > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > > > > > +
> > > > > > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > > > > > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > > > > > +};
> > > > > > +
> > > > > > +struct virtio_video_ctrl_hdr {
> > > > > > +        le32 type;
> > > > > > +        le32 stream_id;
> > > > > > +        le32 len; /* Length of the structure in bytes. */
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{type}] is the type of the driver request or the device
> > > > > > +response.
> > > > > > +\item[\field{stream_id}] specifies a target stream.
> > > > > > +\item[\field{len}] is the length of data in bytes, which includes
> > > > > > +length of the header.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\subsubsection{Device Operation: controlq}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +
> > > > > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > > > > > +supported formats.
> > > > > > +
> > > > > > +The driver uses \field{struct virtio_video_get_capability} to send
> > > > > > a
> > > > > > +query request.
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +enum virtio_video_buf_type {
> > > > > > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > > > > > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > > > > > +};
> > > > > > +
> > > > > > +struct virtio_video_get_capability {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +        enum virtio_video_buf_type buf_type;
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +\begin{description}
> > > > > > +\item[\field{buf_type}] is the buffer type that the driver asks
> > > > > > +information about. The driver MUST set either
> > > > > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > > > > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > > > > > +
> > > > > > +The device responds a capability by using \field{struct
> > > > > > +virtio_video_get_capability_resp}.
> > > > > > +\begin{lstlisting}
> > > > > > +enum virtio_video_format {
> > > > > > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > > > > > +        /* Raw formats */
> > > > > > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > > > > > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > > > > > +        VIRTIO_VIDEO_FORMAT_YVU420,
> > > > > > +
> > > > > > +        /* Compressed formats */
> > > > > > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > > > > > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > > > > > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> > > > > > +};
> > > > > > +
> > > > > > +enum virtio_video_profile {
> > > > > > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > > > > > +
> > > > > > +        /* H.264 */
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > > > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > > > > > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > > > > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > > > > > +        /* VP8 */
> > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_ANY =
> > > > > > VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_MAX =
> > > > > > VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > > > > > +
> > > > > > +        /* VP9 */
> > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 =
> > > > > > VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_MAX =
> > > > > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > > > +};
> > > > > > +
> > > > > > +struct virtio_video_format_range {
> > > > > > +        le32 min;
> > > > > > +        le32 max;
> > > > > > +        le32 step;
> > > > > > +        u8 paddings[4];
> > > > > > +};
> > > > > > +
> > > > > > +struct virtio_video_format_desc {
> > > > > > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > > > > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > > > > > +        le64 mask;
> > > > > > +        struct virtio_video_format_range width;
> > > > > > +        struct virtio_video_format_range height;
> > > > > > +        le32 num_rates;
> > > > > > +        u8 padding[4];
> > > > > > +        /* Followed by struct virtio_video_frame_rate frame_rates[]
> > > > > > */
> > > > > > +};
> > > > > > +
> > > > > > +struct virtio_video_get_capability_resp {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +        le32 num_descs;
> > > > > > +        /* Followed by struct virtio_video_format_desc desc[] */
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +The format description \field{struct virtio_video_format_desc}
> > > > > > +includes the following fields:
> > > > > > +\begin{description}
> > > > > > +\item[\field{format}] specifies an image format. The device MUST
> > > > > > set
> > > > > > one
> > > > > > +  of \field{enum virtio_video_format}.
> > > > > > +\item[\field{profile}] specifies a profile of the compressed image
> > > > > > format
> > > > > > +  specified in \field{format}. The driver SHOULD ignore this value
> > > > > > if
> > > > > > +  \field{format} is a raw format.
> > > > >
> > > > > So how should this be used? The spec does not define any way to set
> > > > > profile for the device. It is very important for encoder.
> > > >
> > > > Thank you for pointing this.
> > > > These points are overlooked, as I didn't care about encoder usage
> > > > enough.
> > > >
> > > > After thinking it again, I think it's not a very good idea to include
> > > > supported profiles and levels in a struct for capability.
> > > > This is because these values are available only for limited number of
> > > > formats. Also, it's true that we need to have a way to set these values
> > > > as
> > > > Dmitry pointed.
> > >
> > > Yes, you are right. In fact, the approach of the v1 spec to keep controls
> > > separately was not correct.
> > >
> > > > Instead, it would make more sense to have additional three types of
> > > > controls for profiles, levels, and bitrates:
> > > > (1) QUERY_CONTROL: Query values supported by the device
> > > > (2) GET_CONTROL: Read a value that is set in the device
> > > > (3) SET_CONTROL: Set a value in the device
> > > >
> > > > These operations are similar to V4L2 controls.
> > > > (1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL, and
> > > > G_CTRL in V4L2, respectively.
> > > > Also, (3) would be similar to enum virtio_video_control_type in the
> > > > virtio-video v1 driver implementation in
> > > > https://markmail.org/message/dwghwdqsbl3gsjxu .
> > > >
> > > > For QUERY_CONTROL, my idea is like this:
> > > >
> > > > enum virtio_video_control_type {
> > > >
> > > >   VIRTIO_VIDEO_CONTROL_UNDEFINED = 0,
> > > >
> > > >   VIRTIO_VIDEO_CONTROL_BITRATE = 0x100,
> > > >   VIRTIO_VIDEO_CONTROL_PROFILE,
> > > >   VIRTIO_VIDEO_CONTROL_LEVEL,
> > > >
> > > > };
> > > >
> > > > struct virtio_video_query_control {
> > > >
> > > >   struct virtio_video_ctrl_hdr hdr;
> > > >   le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
> > > >   le32 length;
> > > >   /* Followed by additional data.
> > > >
> > > >    * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE,
> > > >    * the device must pass a codec format like H264 or VP9.
> > > >    * The requred data must be defined in the specification.
> > > >    */
> > > >
> > > > };
> > > >
> > > > struct virtio_video_query_control_resp {
> > > >
> > > >   struct virtio_video_ctrl_hdr hdr;
> > > >   le32 length;
> > > >   u8 padding[4];
> > > >   /* Followed by data corresponds to the specified control.
> > > >
> > > >    * The type of data must be defined in the spec.
> > > >    * For example, if the driver queries profiles, this part should be
> > > >    * an array of supported profiles of a given format.
> > > >    */
> > > >
> > > > };
> > > >
> > > > WDYT?
> > >
> > > I think virtio_video_control_type should make sense. But I would disagree
> > > with the need to have new QUERY_CONTROL and GET_CONTROL.
> > >
> > > I assume the set of supported controls is fixed for some particular format
> > > on a given IP. So we'd propose to include controls into format
> > > descriptors, so we don't need to QUERY_CONTROL. This way (with
> > > 'virtio_video_format_list') we can define not-contiguos ranges, e.g. for
> > > profiles.
> > >
> > > struct virtio_video_format_frame {
> > >
> > >         /* As proposed in Keiichi's prev email */
> > >
> > > };
> > >
> > > virtio_video_format_list {
> > >
> > >         le32 num_entries;
> > >         u8 padding[4];
> > >         /* Followed by le64 entries[] */
> > >
> > > };
> > >
> > > struct virtio_video_format_control {
> > >
> > >         le32 type;
> > >         u8 padding[4];
> > >         struct virtio_video_format_list values;
> > >
> > > };
> > >
> > > struct virtio_video_format_desc {
> > >
> > >         le64 mask;
> > >         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> > >         le32 planes_layout; /* See the thread [v2 0/1] */
> > >         le32 num_frames;
> > >         le32 num_controls;
> > >         /* Followed by struct virtio_video_format_frame frames[] */
> > >         /* Followed by struct virtio_video_format_control controls[] */
> > >
> > > };
> >
> > In my understanding, a set of supported levels depends on a profile.
> > H.264 spec says that levels are specified within each profile.
> > cf. "0.5 Profiles and levels" in
> > https://www.itu.int/rec/T-REC-H.264-201906-I/en (V4L2's QUERYMENU doesn't
> > seem to provide a way to query levels for each profile properly, though)
> >
> > So, if we want to have supported profiles and levels in format_desc as
> > your idea, each supported profile should have a list of supported
> > levels.
> > I suppose it makes the structure of video_format_desc too complicated.
> > I'd like to avoid this complexity caused by some specific formats.
> >
> > Instead, I'd like to keep virtio_video_format_desc minimal and have
> > QUERY_CONTROL to query format-specific values like profiles and
> > levels.
> > This design would make it easy to extend controls when we want to
> > support new types of formats. We will just need to define new
> > VIRTIO_VIDEO_CONTROL_*.
> > What do you think?
>
> Yes, I think we can keep it this way, this indeed provides more flexibility and
> this way we won't need to modify the 'format' parsing logic when more controls
> are added.

Thanks! I'm preparing the next version of the patch.

I have one concern about terminology.
We have two different things called "controls" now:
1) Messages passed via controlq. i.e. "ctrl" in 'enum
virtio_video_ctrl_type' and 'virtio_video_ctrl_hdr'.
2) Commands for profiles, levels, and bitrates similar to V4L2
controls. e.g.., "CONTROL" in VIRTIO_VIDEO_CONTROL_*

I feel calling both "controls" is somewhat confusing and I am
wondering if we can't rename either of them.

For example, how about renaming 2) to "command"? Then, we'll have
{QUERY ,GET, SET}_COMMAND.
It may sound strange, though.
Do you have any idea?

Best regards,
Keiichi

>
> Best regards,
> Dmitry.
>
> >
> > Best regards,
> > Keiichi
> >
> > > SET_CONTROL seems to be mandatory. If it succeeds, we can store the
> > > current
> > > value locally, so there is no need to have GET_CONTROL.
> > >
> > > The only exception is the initial (default) control value in the device.
> > > But with the removal of 'function' and with addition of 'caps' instead,
> > > the way to provide defaults is gone. So I suppose for formats we'll be
> > > just using GET on driver start to get the 'defaults'. But the thing is
> > > that for formats there are other uses for GET, but for controls GET
> > > apparently does not make to much sense at runtime.
> > >
> > > > > Also, shouldn't the profile come together with level? Would make sense
> > > > > for
> > > > > encoders.
> > > >
> > > > Yeah. So, in the above idea of QUERY_CONTROL, profile should be
> > > > required when querying supported levels.
> > >
> > > So probably should be enumerated together, not queried, as per the comment
> > > above.
> > >
> > > > > > +\item[\field{mask}] is a bitset that represents the supported
> > > > > > +  combination of input and output format. If \textit{i}-th bit is
> > > > > > set
> > > > > > +  in \field{mask} of \textit{j}-th \field{struct
> > > > > > +  virtio_video_format_desc} for input, the device supports encoding
> > > > > > or
> > > > > > +  decoding from the \textit{j}-th input format to \textit{i}-th
> > > > > > output
> > > > > > +  format.
> > > > > > +\item[\field{width, height}] represents a range of resolutions
> > > > > > +  supported by the device. If its \field{step} is not applicable,
> > > > > > its
> > > > > > +  \field{min} is equal to its \field{max}.
> > > > > > +\item[\field{num_rates}] is the length of an array
> > > > > > \field{frame_rates}.
> > > > > > In
> > > > > > case of decoder, the driver SHOULD ignore this value.
> > > > > > +\item[\field{frame_rates}] is an array of supported frame rates.
> > > > > > +\end{description}
> > > > > > +
> > > > >
> > > > > I'd guess frame rates depend on the resolution as well. This
> > > > > dependency
> > > > > was
> > > > > clear in the v1 spec, but in the v2 there is no dependency anymore. I
> > > > > think we need to update this.
> > > >
> > > > That's a good point. I missed that dependency when updating the
> > > > structures.
> > > > So, let me update the structs like the following:
> > > >
> > > > struct virtio_video_format_frame {
> > > >
> > > >         struct virtio_video_format_range width;
> > > >         struct virtio_video_format_range height;
> > > >         le32 num_rates;
> > > >         u8 padding[4];
> > > >         /* Followed by struct virtio_video_format_range frame_rates[] */
> > > >
> > > > };
> > > >
> > > > struct virtio_video_format_desc {
> > > >
> > > >         le64 mask;
> > > >         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > >         le32 planes_layout; /* See the thread [v2 0/1] */
> > > >         le32 num_frames;
> > > >         u8 padding[4];
> > > >         /* Followed by struct virtio_video_format_frame frames[] */
> > > >
> > > > };
> > >
> > > Yes, I do agree with this approach.
> > >
> > > Best regards,
> > > Dmitry.
> > >
> > > > Best regards,
> > > > Keiichi.
> > > >
> > > > > Best regards,
> > > > > Dmitry.
> > > > >
> > > > > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > > > > > +  within the device.
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +enum virtio_video_mem_type {
> > > > > > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > > > > > +};
> > > > > > +
> > > > > > +struct virtio_video_stream_create {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types
> > > > > > */
> > > > > > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types
> > > > > > */
> > > > > > +        char debug_name[64];
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{in_mem_type}] is a type of buffer management for input
> > > > > > +buffers. The driver MUST set a value in \field{enum
> > > > > > +virtio_video_mem_type}.
> > > > > > +\item[\field{out_mem_type}] is a type of buffer management for
> > > > > > output
> > > > > > +buffers. The driver MUST set a value in \field{enum
> > > > > > +virtio_video_mem_type}.
> > > > > > +\item[\field{debug_name}] is a text string for a debug purpose.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream
> > > > > > (context)
> > > > > > +  within the device.
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +struct virtio_video_stream_destroy {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > > > > > +  queued buffers through the pipeline.
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +struct virtio_video_stream_drain {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > > > > > +  within the device.
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +struct virtio_video_mem_entry {
> > > > > > +        le64 addr;
> > > > > > +        le32 length;
> > > > > > +        u8 padding[4];
> > > > > > +};
> > > > > > +
> > > > > > +struct virtio_video_resource_create {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +        le32 resource_id;
> > > > > > +        le32 nr_entries;
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > +\item[\field{nr_entries}] number of \field{struct
> > > > > > +  virtio_video_mem_entry} memory entries.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource
> > > > > > descriptor
> > > > > > +  within the device.
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +struct virtio_video_resource_destroy {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +        le32 resource_id;
> > > > > > +        u8 padding[4];
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > > > > > +queue.
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > > > > > +
> > > > > > +struct virtio_video_resource_queue {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +        le32 buf_type;
> > > > > > +        le32 resource_id;
> > > > > > +        le64 timestamp;
> > > > > > +        le32 nr_data_size;
> > > > > > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{buf_type}] buf_type of the .
> > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > +\item[\field{timestamp}] an abstract sequence counter that can be
> > > > > > used
> > > > > > +  for synchronisation.
> > > > > > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > > > > > +\item[\field{data_size}] number of data bytes within a plane.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +enum virtio_video_buffer_flag {
> > > > > > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > > > > > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > > > > > +        /* Encoder only */
> > > > > > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > > > > > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > > > > > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > > > > > +};
> > > > > > +
> > > > > > +struct virtio_video_resource_queue_resp {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +        le64 timestamp;
> > > > > > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > > > > > +        le32 size;  /* Encoded size */
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{timestamp}] an abstract sequence counter that can be
> > > > > > used
> > > > > > +  for synchronisation.
> > > > > > +\item[\field{flags}] mark specific buffers in the sequence.
> > > > > > +\item[\field{size}] data size in the buffer (encoder only).
> > > > > > +\end{description}
> > > > > > +
> > > > > > +The device sends a response to the queue request asynchronously
> > > > > > when
> > > > > > +it has finished processing the buffer.
> > > > > > +
> > > > > > +The device SHOULD mark a buffer that triggered a processing error
> > > > > > with
> > > > > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > > > > > +
> > > > > > +The device MUST mark the last buffer with the
> > > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > > > > +sequence.
> > > > > > +
> > > > > > +In case of encoder, to denote a particular frame type the devie
> > > > > > MUST
> > > > > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > > > > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > > > > > +
> > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> > > > > > +  buffers back from the input or the output queue of the device.
> > > > > > The
> > > > > > +  device SHOULD return all of the buffers from the respective queue
> > > > > > as
> > > > > > +  soon as possible without pushing the buffers through the
> > > > > > processing
> > > > > > +  pipeline.
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +struct virtio_video_queue_clear {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +        le32 buf_type;
> > > > > > +        u8 padding[4];
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{buf_type}] buffer type.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> > > > > > +  output of a stream.
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +struct virtio_video_plane_format {
> > > > > > +        le32 plane_size;
> > > > > > +        le32 stride;
> > > > > > +        u8 padding[4];
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{plane_size}] size of the plane in bytes.
> > > > > > +\item[\field{stride}] stride used for the plane in bytes.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +struct virtio_video_params {
> > > > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > > > > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> > > > > > +        le32 frame_width;
> > > > > > +        le32 frame_height;
> > > > > > +        le32 min_buffers;
> > > > > > +        le32 max_buffers;
> > > > > > +        le32 frame_rate;
> > > > > > +        struct virtio_video_crop {
> > > > > > +                le32 left;
> > > > > > +                le32 top;
> > > > > > +                le32 width;
> > > > > > +                le32 height;
> > > > > > +        } crop;
> > > > > > +        le32 num_planes;
> > > > > > +        struct virtio_video_plane_format
> > > > > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{frame_width}] the value to get/set.
> > > > > > +\item[\field{frame_height}] the value to get/set.
> > > > > > +\item[\field{pixel_format}] the value to get/set.
> > > > > > +\item[\field{min_buffers}] minimum buffers required to handle the
> > > > > > +  format (r/o).
> > > > > > +\item[\field{max_buffers}] maximum buffers required to handle the
> > > > > > +  format (r/o).
> > > > > > +\item[\field{frame_rate}] the value to get/set.
> > > > > > +\item[\field{crop}] cropping (composing) rectangle.
> > > > > > +\item[\field{num_planes}] number of planes used to store pixel data
> > > > > > +(r/o).
> > > > > > +\item[\field{plane_formats}] description of each plane.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +struct virtio_video_get_params {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > > > > +};
> > > > > > +
> > > > > > +struct virtio_video_get_params_resp {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +        struct virtio_video_params params;
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{buf_type}] buffer type.
> > > > > > +\item[\field{params}] parameter values.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> > > > > > +
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +struct virtio_video_set_params {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +        struct virtio_video_params params;
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{params}] parameters to set.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +Setting stream parameters might have side effects within the
> > > > > > device.
> > > > > > +For example, the device MAY perform alignment of width and height,
> > > > > > +change the number of planes it uses for the format, or do whatever
> > > > > > +changes that are required to continue normal operation using the
> > > > > > +updated parameters. It is up to the driver to check the parameter
> > > > > > set
> > > > > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > > > > > +
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\subsubsection{Device Operation: eventq}
> > > > > > +
> > > > > > +The device can report events on the event queue. The driver
> > > > > > initially
> > > > > > +populates the queue with device-writeable buffers. When the device
> > > > > > +needs to report an event, it fills a buffer and notifies the
> > > > > > driver.
> > > > > > +The driver consumes the report and adds a new buffer to the
> > > > > > virtqueue.
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +enum virtio_video_event_type {
> > > > > > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > > > > > +        /* For all functions */
> > > > > > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > > > > > +        /* For decoder only */
> > > > > > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> > > > > > +};
> > > > > > +
> > > > > > +struct virtio_video_event {
> > > > > > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> > > > > > +        le32 stream_id;
> > > > > > +        u8 padding[4];
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{event_type}] type of the triggered event .
> > > > > > +\item[\field{stream_id}] id of the source stream.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +The device MUST send
> > > > > > VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > > > > > +whenever it encounters new resolution data in the stream. This
> > > > > > +includes the case of the initial device configuration after
> > > > > > metadata
> > > > > > +has been parsed and the case of dynamic resolution change.
>
>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-18 13:02 ` [PATCH v2 1/1] virtio-video: Add virtio " Keiichi Watanabe
                     ` (4 preceding siblings ...)
  2020-01-06 14:59   ` Dmitry Sepp
@ 2020-01-08 12:23   ` Keiichi Watanabe
  5 siblings, 0 replies; 73+ messages in thread
From: Keiichi Watanabe @ 2020-01-08 12:23 UTC (permalink / raw)
  To: Dmitry Sepp
  Cc: Linux Media Mailing List, virtio-dev, Alexandre Courbot,
	Alex Lau, Daniel Vetter, Dylan Reid, Enrico Granata,
	Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, Tomasz Figa, uril

Hi Dmitry,

On Wed, Dec 18, 2019 at 10:02 PM Keiichi Watanabe <keiichiw@chromium.org> wrote:
>
> From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
>
> The virtio video encoder device and decoder device provide functionalities to
> encode and decode video stream respectively.
> Though video encoder and decoder are provided as different devices, they use a
> same protocol.
>
> Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> ---
>  content.tex      |   1 +
>  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 580 insertions(+)
>  create mode 100644 virtio-video.tex
>
> diff --git a/content.tex b/content.tex
> index 556b373..9e56839 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
>  \input{virtio-vsock.tex}
>  \input{virtio-fs.tex}
>  \input{virtio-rpmb.tex}
> +\input{virtio-video.tex}
>
>  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>
> diff --git a/virtio-video.tex b/virtio-video.tex
> new file mode 100644
> index 0000000..30e728d
> --- /dev/null
> +++ b/virtio-video.tex
> @@ -0,0 +1,579 @@
> +\section{Video Device}\label{sec:Device Types / Video Device}
> +
> +The virtio video encoder device and decoder device are virtual devices that
> +supports encoding and decoding respectively. Though the encoder and the decoder
> +are different devices, they use the same protocol.
> +
> +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
> +
> +\begin{description}
> +\item[30] encoder device
> +\item[31] decoder device
> +\end{description}
> +
> +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues}
> +
> +\begin{description}
> +\item[0] controlq - queue for sending control commands.
> +\item[1] eventq - queue for sending events happened in the device.
> +\end{description}
> +
> +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature bits}
> +
> +\begin{description}
> +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for video
> +  buffers.
> +\end{description}
> +
> +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits}
> +
> +The device MUST offer at least one of feature bits.
> +
> +\subsection{Device configuration layout}\label{sec:Device Types / Video Device / Device configuration layout}
> +
> +Video device configuration uses the following layout structure:
> +
> +\begin{lstlisting}
> +struct virtio_video_config {
> +        le32 max_cap_len;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> +  MUST set this value.
> +\end{description}
> +
> +\subsection{Device Initialization}\label{sec:Device Types / Video Device / Device Initialization}
> +
> +\devicenormative{\subsubsection}{Device Initialization}{Device Types / Video Device / Device Initialization}
> +
> +The driver SHOULD query device capability by using the
> +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> +setup.
> +
> +\subsection{Device Operation}\label{sec:Device Types / Video Device / Device Operation}
> +
> +The driver allocates input and output buffers and queues the buffers
> +to the device. The device performs operations on the buffers according
> +to the function in question.
> +
> +\subsubsection{Device Operation: Create stream}
> +
> +To process buffers, the device needs to associate them with a certain
> +video stream (essentially, a context). Streams are created by
> +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> +determined by the device.
> +
> +\subsubsection{Device Operation: Create buffers}
> +
> +Buffers are used to store the actual data as well as the relevant
> +metadata. Scatter lists are supported, so the buffer doesn't need to
> +be contiguous in guest physical memory.
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> +  resource that is backed by a buffer allocated from the driver's
> +  memory.
> +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> +  is no longer needed.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Stream parameter control}
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters for
> +  input and output streams from the device.
> +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the
> +  device.
> +\item After setting stream parameters, the driver may issue
> +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can be
> +  changed implicitly by the device during the set operation.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Process buffers}
> +
> +\begin{itemize*}
> +\item If the function and the buffer type require so, write data to
> +the buffer memory.
> +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> +processing in the device.
> +\item The request completes asynchronously when the device has
> +finished with the buffer.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Buffer processing control}
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> +  return all of the already queued buffers.
> +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> +  already queued buffers from the input or the output queue. This also
> +  includes input or output buffers that can be currently owned by the
> +  device's processing pipeline.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Asynchronous events}
> +
> +While processing buffers, the device can send asynchronous event
> +notifications to the driver. The behaviour depends on the exact
> +stream. For example, the decoder device sends a resolution change
> +event when it encounters new resolution metadata in the stream.
> +
> +\subsubsection{Device Operation: Request header}
> +
> +All requests and responses on the control virt queue have a fixed
> +header using the following layout structure and definitions:
> +
> +\begin{lstlisting}
> +enum virtio_video_ctrl_type {
> +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> +
> +        /* request */
> +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> +        VIRTIO_VIDEO_T_STREAM_CREATE,
> +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> +        VIRTIO_VIDEO_T_SET_PARAMS,
> +        VIRTIO_VIDEO_T_GET_PARAMS,
> +
> +        /* response */
> +        VIRTIO_VIDEO_S_OK = 0x0200,
> +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> +
> +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> +};

I have a question about naming of these constants.
What's the meaning of "T" and "S" for requests and responses, respectively?
Though I copied these values from your first version, I'm wondering if
we can't call VIRTIO_VIDEO_REQ_* and VIRTIO_VIDEO_RESP_* instead.

Best regards,
Keiichi

> +
> +struct virtio_video_ctrl_hdr {
> +        le32 type;
> +        le32 stream_id;
> +        le32 len; /* Length of the structure in bytes. */
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{type}] is the type of the driver request or the device
> +response.
> +\item[\field{stream_id}] specifies a target stream.
> +\item[\field{len}] is the length of data in bytes, which includes
> +length of the header.
> +\end{description}
> +
> +\subsubsection{Device Operation: controlq}
> +
> +\begin{description}
> +
> +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> +supported formats.
> +
> +The driver uses \field{struct virtio_video_get_capability} to send a
> +query request.
> +
> +\begin{lstlisting}
> +enum virtio_video_buf_type {
> +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> +};
> +
> +struct virtio_video_get_capability {
> +        struct virtio_video_ctrl_hdr hdr;
> +        enum virtio_video_buf_type buf_type;
> +};
> +\end{lstlisting}
> +\begin{description}
> +\item[\field{buf_type}] is the buffer type that the driver asks
> +information about. The driver MUST set either
> +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}.
> +\end{description}
> +
> +The device responds a capability by using \field{struct
> +virtio_video_get_capability_resp}.
> +\begin{lstlisting}
> +enum virtio_video_format {
> +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> +        /* Raw formats */
> +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> +        VIRTIO_VIDEO_FORMAT_YUV420,
> +        VIRTIO_VIDEO_FORMAT_YVU420,
> +
> +        /* Compressed formats */
> +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> +};
> +
> +enum virtio_video_profile {
> +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> +
> +        /* H.264 */
> +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> +        VIRTIO_VIDEO_PROFILE_H264_BASELINE = VIRTIO_VIDEO_PROFILE_H264_BASELINE,
> +        VIRTIO_VIDEO_PROFILE_H264_MAIN,
> +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_MAX = VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> +
> +        /* VP8 */
> +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> +
> +        /* VP9 */
> +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> +        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> +};
> +
> +struct virtio_video_format_range {
> +        le32 min;
> +        le32 max;
> +        le32 step;
> +        u8 paddings[4];
> +};
> +
> +struct virtio_video_format_desc {
> +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> +        le64 mask;
> +        struct virtio_video_format_range width;
> +        struct virtio_video_format_range height;
> +        le32 num_rates;
> +        u8 padding[4];
> +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> +};
> +
> +struct virtio_video_get_capability_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 num_descs;
> +        /* Followed by struct virtio_video_format_desc desc[] */
> +};
> +\end{lstlisting}
> +
> +The format description \field{struct virtio_video_format_desc}
> +includes the following fields:
> +\begin{description}
> +\item[\field{format}] specifies an image format. The device MUST set one
> +  of \field{enum virtio_video_format}.
> +\item[\field{profile}] specifies a profile of the compressed image format
> +  specified in \field{format}. The driver SHOULD ignore this value if
> +  \field{format} is a raw format.
> +\item[\field{mask}] is a bitset that represents the supported
> +  combination of input and output format. If \textit{i}-th bit is set
> +  in \field{mask} of \textit{j}-th \field{struct
> +  virtio_video_format_desc} for input, the device supports encoding or
> +  decoding from the \textit{j}-th input format to \textit{i}-th output
> +  format.
> +\item[\field{width, height}] represents a range of resolutions
> +  supported by the device. If its \field{step} is not applicable, its
> +  \field{min} is equal to its \field{max}.
> +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In case of decoder, the driver SHOULD ignore this value.
> +\item[\field{frame_rates}] is an array of supported frame rates.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> +  within the device.
> +
> +\begin{lstlisting}
> +enum virtio_video_mem_type {
> +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> +};
> +
> +struct virtio_video_stream_create {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> +        char debug_name[64];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{in_mem_type}] is a type of buffer management for input
> +buffers. The driver MUST set a value in \field{enum
> +virtio_video_mem_type}.
> +\item[\field{out_mem_type}] is a type of buffer management for output
> +buffers. The driver MUST set a value in \field{enum
> +virtio_video_mem_type}.
> +\item[\field{debug_name}] is a text string for a debug purpose.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_stream_destroy {
> +        struct virtio_video_ctrl_hdr hdr;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> +  queued buffers through the pipeline.
> +
> +\begin{lstlisting}
> +struct virtio_video_stream_drain {
> +        struct virtio_video_ctrl_hdr hdr;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_mem_entry {
> +        le64 addr;
> +        le32 length;
> +        u8 padding[4];
> +};
> +
> +struct virtio_video_resource_create {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 resource_id;
> +        le32 nr_entries;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{resource_id}] internal id of the resource.
> +\item[\field{nr_entries}] number of \field{struct
> +  virtio_video_mem_entry} memory entries.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_resource_destroy {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 resource_id;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{resource_id}] internal id of the resource.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> +queue.
> +
> +\begin{lstlisting}
> +#define VIRTIO_VIDEO_MAX_PLANES 8
> +
> +struct virtio_video_resource_queue {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type;
> +        le32 resource_id;
> +        le64 timestamp;
> +        le32 nr_data_size;
> +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buf_type of the .
> +\item[\field{resource_id}] internal id of the resource.
> +\item[\field{timestamp}] an abstract sequence counter that can be used
> +  for synchronisation.
> +\item[\field{nr_data_size}] number of \field{data_size} entries.
> +\item[\field{data_size}] number of data bytes within a plane.
> +\end{description}
> +
> +\begin{lstlisting}
> +enum virtio_video_buffer_flag {
> +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> +        /* Encoder only */
> +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> +};
> +
> +struct virtio_video_resource_queue_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le64 timestamp;
> +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> +        le32 size;  /* Encoded size */
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{timestamp}] an abstract sequence counter that can be used
> +  for synchronisation.
> +\item[\field{flags}] mark specific buffers in the sequence.
> +\item[\field{size}] data size in the buffer (encoder only).
> +\end{description}
> +
> +The device sends a response to the queue request asynchronously when
> +it has finished processing the buffer.
> +
> +The device SHOULD mark a buffer that triggered a processing error with
> +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> +
> +The device MUST mark the last buffer with the
> +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> +sequence.
> +
> +In case of encoder, to denote a particular frame type the devie MUST
> +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> +  buffers back from the input or the output queue of the device. The
> +  device SHOULD return all of the buffers from the respective queue as
> +  soon as possible without pushing the buffers through the processing
> +  pipeline.
> +
> +\begin{lstlisting}
> +struct virtio_video_queue_clear {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buffer type.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> +  output of a stream.
> +
> +\begin{lstlisting}
> +struct virtio_video_plane_format {
> +        le32 plane_size;
> +        le32 stride;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{plane_size}] size of the plane in bytes.
> +\item[\field{stride}] stride used for the plane in bytes.
> +\end{description}
> +
> +\begin{lstlisting}
> +struct virtio_video_params {
> +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> +        le32 frame_width;
> +        le32 frame_height;
> +        le32 min_buffers;
> +        le32 max_buffers;
> +        le32 frame_rate;
> +        struct virtio_video_crop {
> +                le32 left;
> +                le32 top;
> +                le32 width;
> +                le32 height;
> +        } crop;
> +        le32 num_planes;
> +        struct virtio_video_plane_format plane_formats[VIRTIO_VIDEO_MAX_PLANES];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{frame_width}] the value to get/set.
> +\item[\field{frame_height}] the value to get/set.
> +\item[\field{pixel_format}] the value to get/set.
> +\item[\field{min_buffers}] minimum buffers required to handle the
> +  format (r/o).
> +\item[\field{max_buffers}] maximum buffers required to handle the
> +  format (r/o).
> +\item[\field{frame_rate}] the value to get/set.
> +\item[\field{crop}] cropping (composing) rectangle.
> +\item[\field{num_planes}] number of planes used to store pixel data
> +(r/o).
> +\item[\field{plane_formats}] description of each plane.
> +\end{description}
> +
> +\begin{lstlisting}
> +struct virtio_video_get_params {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> +};
> +
> +struct virtio_video_get_params_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        struct virtio_video_params params;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buffer type.
> +\item[\field{params}] parameter values.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> +
> +
> +\begin{lstlisting}
> +struct virtio_video_set_params {
> +        struct virtio_video_ctrl_hdr hdr;
> +        struct virtio_video_params params;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{params}] parameters to set.
> +\end{description}
> +
> +Setting stream parameters might have side effects within the device.
> +For example, the device MAY perform alignment of width and height,
> +change the number of planes it uses for the format, or do whatever
> +changes that are required to continue normal operation using the
> +updated parameters. It is up to the driver to check the parameter set
> +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> +
> +\end{description}
> +
> +\subsubsection{Device Operation: eventq}
> +
> +The device can report events on the event queue. The driver initially
> +populates the queue with device-writeable buffers. When the device
> +needs to report an event, it fills a buffer and notifies the driver.
> +The driver consumes the report and adds a new buffer to the virtqueue.
> +
> +\begin{lstlisting}
> +enum virtio_video_event_type {
> +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> +        /* For all functions */
> +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> +        /* For decoder only */
> +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> +};
> +
> +struct virtio_video_event {
> +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> +        le32 stream_id;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{event_type}] type of the triggered event .
> +\item[\field{stream_id}] id of the source stream.
> +\end{description}
> +
> +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> +whenever it encounters new resolution data in the stream. This
> +includes the case of the initial device configuration after metadata
> +has been parsed and the case of dynamic resolution change.
> --
> 2.24.1.735.g03f4e72817-goog
>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2020-01-08 12:14             ` Keiichi Watanabe
@ 2020-01-08 12:46               ` Tomasz Figa
  2020-01-08 13:05                 ` Keiichi Watanabe
  2020-01-08 13:11                 ` Dmitry Sepp
  0 siblings, 2 replies; 73+ messages in thread
From: Tomasz Figa @ 2020-01-08 12:46 UTC (permalink / raw)
  To: Keiichi Watanabe
  Cc: Dmitry Sepp, uril, virtio-dev, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Enrico Granata, Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens

On Wed, Jan 8, 2020 at 9:15 PM Keiichi Watanabe <keiichiw@chromium.org> wrote:
>
> Hi Dmitry,
>
> On Wed, Jan 8, 2020 at 7:00 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
> >
> > Hi Keiichi,
> >
> > On Mittwoch, 8. Januar 2020 07:59:22 CET Keiichi Watanabe wrote:
> > > Hi Dmitry,
> > >
> > > On Wed, Jan 8, 2020 at 1:50 AM Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > wrote:
> > > > Hi Keiichi,
> > > >
> > > > thanks for the updates, please see my comments below.
> > > >
> > > > On Dienstag, 7. Januar 2020 14:24:31 CET Keiichi Watanabe wrote:
> > > > > Hi Dmitry,
> > > > >
> > > > > On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp
> > > > > <dmitry.sepp@opensynergy.com>
> > > >
> > > > wrote:
> > > > > > Hi,
> > > > > >
> > > > > > a couple of new comments:
> > > > > >
> > > > > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> > > > > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > > > >
> > > > > > > The virtio video encoder device and decoder device provide
> > > > > > > functionalities
> > > > > > > to encode and decode video stream respectively.
> > > > > > > Though video encoder and decoder are provided as different devices,
> > > > > > > they
> > > > > > > use a same protocol.
> > > > > > >
> > > > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > > > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > > > > > > ---
> > > > > > >
> > > > > > >  content.tex      |   1 +
> > > > > > >  virtio-video.tex | 579
> > > > > > >  +++++++++++++++++++++++++++++++++++++++++++++++
> > > > > > >  2 files changed, 580 insertions(+)
> > > > > > >  create mode 100644 virtio-video.tex
> > > > > > >
> > > > > > > diff --git a/content.tex b/content.tex
> > > > > > > index 556b373..9e56839 100644
> > > > > > > --- a/content.tex
> > > > > > > +++ b/content.tex
> > > > > > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > > > > > > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> > > > > > >
> > > > > > >  \input{virtio-fs.tex}
> > > > > > >  \input{virtio-rpmb.tex}
> > > > > > >
> > > > > > > +\input{virtio-video.tex}
> > > > > > >
> > > > > > >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> > > > > > >
> > > > > > > diff --git a/virtio-video.tex b/virtio-video.tex
> > > > > > > new file mode 100644
> > > > > > > index 0000000..30e728d
> > > > > > > --- /dev/null
> > > > > > > +++ b/virtio-video.tex
> > > > > > > @@ -0,0 +1,579 @@
> > > > > > > +\section{Video Device}\label{sec:Device Types / Video Device}
> > > > > > > +
> > > > > > > +The virtio video encoder device and decoder device are virtual
> > > > > > > devices
> > > > > > > that +supports encoding and decoding respectively. Though the
> > > > > > > encoder
> > > > > > > and the decoder +are different devices, they use the same protocol.
> > > > > > > +
> > > > > > > +\subsection{Device ID}\label{sec:Device Types / Video Device /
> > > > > > > Device
> > > > > > > ID}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[30] encoder device
> > > > > > > +\item[31] decoder device
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device /
> > > > > > > Virtqueues} +
> > > > > > > +\begin{description}
> > > > > > > +\item[0] controlq - queue for sending control commands.
> > > > > > > +\item[1] eventq - queue for sending events happened in the device.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\subsection{Feature bits}\label{sec:Device Types / Video Device /
> > > > > > > Feature
> > > > > > > bits} +
> > > > > > > +\begin{description}
> > > > > > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be
> > > > > > > used
> > > > > > > for
> > > > > > > video +  buffers.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video
> > > > > > > Device
> > > > > > > / Feature bits} +
> > > > > > > +The device MUST offer at least one of feature bits.
> > > > > > > +
> > > > > > > +\subsection{Device configuration layout}\label{sec:Device Types /
> > > > > > > Video
> > > > > > > Device / Device configuration layout} +
> > > > > > > +Video device configuration uses the following layout structure:
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +struct virtio_video_config {
> > > > > > > +        le32 max_cap_len;
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{max_cap_len}] defines the maximum length of a
> > > > > > > descriptor
> > > > > > > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > > > > > > +  MUST set this value.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\subsection{Device Initialization}\label{sec:Device Types / Video
> > > > > > > Device /
> > > > > > > Device Initialization} +
> > > > > > > +\devicenormative{\subsubsection}{Device Initialization}{Device
> > > > > > > Types /
> > > > > > > Video Device / Device Initialization} +
> > > > > > > +The driver SHOULD query device capability by using the
> > > > > > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the
> > > > > > > initial
> > > > > > > +setup.
> > > > > > > +
> > > > > > > +\subsection{Device Operation}\label{sec:Device Types / Video Device
> > > > > > > /
> > > > > > > Device Operation} +
> > > > > > > +The driver allocates input and output buffers and queues the
> > > > > > > buffers
> > > > > > > +to the device. The device performs operations on the buffers
> > > > > > > according
> > > > > > > +to the function in question.
> > > > > > > +
> > > > > > > +\subsubsection{Device Operation: Create stream}
> > > > > > > +
> > > > > > > +To process buffers, the device needs to associate them with a
> > > > > > > certain
> > > > > > > +video stream (essentially, a context). Streams are created by
> > > > > > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > > > > > > +determined by the device.
> > > > > > > +
> > > > > > > +\subsubsection{Device Operation: Create buffers}
> > > > > > > +
> > > > > > > +Buffers are used to store the actual data as well as the relevant
> > > > > > > +metadata. Scatter lists are supported, so the buffer doesn't need
> > > > > > > to
> > > > > > > +be contiguous in guest physical memory.
> > > > > > > +
> > > > > > > +\begin{itemize*}
> > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > > > > > > +  resource that is backed by a buffer allocated from the driver's
> > > > > > > +  memory.
> > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource
> > > > > > > that
> > > > > > > +  is no longer needed.
> > > > > > > +\end{itemize*}
> > > > > > > +
> > > > > > > +\subsubsection{Device Operation: Stream parameter control}
> > > > > > > +
> > > > > > > +\begin{itemize*}
> > > > > > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream
> > > > > > > parameters
> > > > > > > for +  input and output streams from the device.
> > > > > > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream
> > > > > > > parameters to
> > > > > > > the +  device.
> > > > > > > +\item After setting stream parameters, the driver may issue
> > > > > > > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and
> > > > > > > output
> > > > > > > can be +  changed implicitly by the device during the set operation.
> > > > > > > +\end{itemize*}
> > > > > > > +
> > > > > > > +\subsubsection{Device Operation: Process buffers}
> > > > > > > +
> > > > > > > +\begin{itemize*}
> > > > > > > +\item If the function and the buffer type require so, write data to
> > > > > > > +the buffer memory.
> > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > > > > > > +processing in the device.
> > > > > > > +\item The request completes asynchronously when the device has
> > > > > > > +finished with the buffer.
> > > > > > > +\end{itemize*}
> > > > > > > +
> > > > > > > +\subsubsection{Device Operation: Buffer processing control}
> > > > > > > +
> > > > > > > +\begin{itemize*}
> > > > > > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process
> > > > > > > and
> > > > > > > +  return all of the already queued buffers.
> > > > > > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return
> > > > > > > back
> > > > > > > +  already queued buffers from the input or the output queue. This
> > > > > > > also
> > > > > > > +  includes input or output buffers that can be currently owned by
> > > > > > > the
> > > > > > > +  device's processing pipeline.
> > > > > > > +\end{itemize*}
> > > > > > > +
> > > > > > > +\subsubsection{Device Operation: Asynchronous events}
> > > > > > > +
> > > > > > > +While processing buffers, the device can send asynchronous event
> > > > > > > +notifications to the driver. The behaviour depends on the exact
> > > > > > > +stream. For example, the decoder device sends a resolution change
> > > > > > > +event when it encounters new resolution metadata in the stream.
> > > > > > > +
> > > > > > > +\subsubsection{Device Operation: Request header}
> > > > > > > +
> > > > > > > +All requests and responses on the control virt queue have a fixed
> > > > > > > +header using the following layout structure and definitions:
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +enum virtio_video_ctrl_type {
> > > > > > > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > > > > > > +
> > > > > > > +        /* request */
> > > > > > > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > > > > > > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > > > > > > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > > > > > > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > > > > > > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > > > > > > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > > > > > > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > > > > > > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > > > > > > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > > > > > > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > > > > > > +
> > > > > > > +        /* response */
> > > > > > > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > > > > > > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > > > > > > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > > > > > > +
> > > > > > > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > > > > > > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > > > > > > +};
> > > > > > > +
> > > > > > > +struct virtio_video_ctrl_hdr {
> > > > > > > +        le32 type;
> > > > > > > +        le32 stream_id;
> > > > > > > +        le32 len; /* Length of the structure in bytes. */
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{type}] is the type of the driver request or the device
> > > > > > > +response.
> > > > > > > +\item[\field{stream_id}] specifies a target stream.
> > > > > > > +\item[\field{len}] is the length of data in bytes, which includes
> > > > > > > +length of the header.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\subsubsection{Device Operation: controlq}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +
> > > > > > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > > > > > > +supported formats.
> > > > > > > +
> > > > > > > +The driver uses \field{struct virtio_video_get_capability} to send
> > > > > > > a
> > > > > > > +query request.
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +enum virtio_video_buf_type {
> > > > > > > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > > > > > > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > > > > > > +};
> > > > > > > +
> > > > > > > +struct virtio_video_get_capability {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +        enum virtio_video_buf_type buf_type;
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{buf_type}] is the buffer type that the driver asks
> > > > > > > +information about. The driver MUST set either
> > > > > > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > > > > > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > > > > > > +
> > > > > > > +The device responds a capability by using \field{struct
> > > > > > > +virtio_video_get_capability_resp}.
> > > > > > > +\begin{lstlisting}
> > > > > > > +enum virtio_video_format {
> > > > > > > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > > > > > > +        /* Raw formats */
> > > > > > > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > > > > > > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > > > > > > +        VIRTIO_VIDEO_FORMAT_YVU420,
> > > > > > > +
> > > > > > > +        /* Compressed formats */
> > > > > > > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > > > > > > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > > > > > > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> > > > > > > +};
> > > > > > > +
> > > > > > > +enum virtio_video_profile {
> > > > > > > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > > > > > > +
> > > > > > > +        /* H.264 */
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > > > > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > > > > > > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > > > > > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > > > > > > +        /* VP8 */
> > > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_ANY =
> > > > > > > VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_MAX =
> > > > > > > VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > > > > > > +
> > > > > > > +        /* VP9 */
> > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 =
> > > > > > > VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_MAX =
> > > > > > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > > > > +};
> > > > > > > +
> > > > > > > +struct virtio_video_format_range {
> > > > > > > +        le32 min;
> > > > > > > +        le32 max;
> > > > > > > +        le32 step;
> > > > > > > +        u8 paddings[4];
> > > > > > > +};
> > > > > > > +
> > > > > > > +struct virtio_video_format_desc {
> > > > > > > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > > > > > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > > > > > > +        le64 mask;
> > > > > > > +        struct virtio_video_format_range width;
> > > > > > > +        struct virtio_video_format_range height;
> > > > > > > +        le32 num_rates;
> > > > > > > +        u8 padding[4];
> > > > > > > +        /* Followed by struct virtio_video_frame_rate frame_rates[]
> > > > > > > */
> > > > > > > +};
> > > > > > > +
> > > > > > > +struct virtio_video_get_capability_resp {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +        le32 num_descs;
> > > > > > > +        /* Followed by struct virtio_video_format_desc desc[] */
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +The format description \field{struct virtio_video_format_desc}
> > > > > > > +includes the following fields:
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{format}] specifies an image format. The device MUST
> > > > > > > set
> > > > > > > one
> > > > > > > +  of \field{enum virtio_video_format}.
> > > > > > > +\item[\field{profile}] specifies a profile of the compressed image
> > > > > > > format
> > > > > > > +  specified in \field{format}. The driver SHOULD ignore this value
> > > > > > > if
> > > > > > > +  \field{format} is a raw format.
> > > > > >
> > > > > > So how should this be used? The spec does not define any way to set
> > > > > > profile for the device. It is very important for encoder.
> > > > >
> > > > > Thank you for pointing this.
> > > > > These points are overlooked, as I didn't care about encoder usage
> > > > > enough.
> > > > >
> > > > > After thinking it again, I think it's not a very good idea to include
> > > > > supported profiles and levels in a struct for capability.
> > > > > This is because these values are available only for limited number of
> > > > > formats. Also, it's true that we need to have a way to set these values
> > > > > as
> > > > > Dmitry pointed.
> > > >
> > > > Yes, you are right. In fact, the approach of the v1 spec to keep controls
> > > > separately was not correct.
> > > >
> > > > > Instead, it would make more sense to have additional three types of
> > > > > controls for profiles, levels, and bitrates:
> > > > > (1) QUERY_CONTROL: Query values supported by the device
> > > > > (2) GET_CONTROL: Read a value that is set in the device
> > > > > (3) SET_CONTROL: Set a value in the device
> > > > >
> > > > > These operations are similar to V4L2 controls.
> > > > > (1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL, and
> > > > > G_CTRL in V4L2, respectively.
> > > > > Also, (3) would be similar to enum virtio_video_control_type in the
> > > > > virtio-video v1 driver implementation in
> > > > > https://markmail.org/message/dwghwdqsbl3gsjxu .
> > > > >
> > > > > For QUERY_CONTROL, my idea is like this:
> > > > >
> > > > > enum virtio_video_control_type {
> > > > >
> > > > >   VIRTIO_VIDEO_CONTROL_UNDEFINED = 0,
> > > > >
> > > > >   VIRTIO_VIDEO_CONTROL_BITRATE = 0x100,
> > > > >   VIRTIO_VIDEO_CONTROL_PROFILE,
> > > > >   VIRTIO_VIDEO_CONTROL_LEVEL,
> > > > >
> > > > > };
> > > > >
> > > > > struct virtio_video_query_control {
> > > > >
> > > > >   struct virtio_video_ctrl_hdr hdr;
> > > > >   le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
> > > > >   le32 length;
> > > > >   /* Followed by additional data.
> > > > >
> > > > >    * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE,
> > > > >    * the device must pass a codec format like H264 or VP9.
> > > > >    * The requred data must be defined in the specification.
> > > > >    */
> > > > >
> > > > > };
> > > > >
> > > > > struct virtio_video_query_control_resp {
> > > > >
> > > > >   struct virtio_video_ctrl_hdr hdr;
> > > > >   le32 length;
> > > > >   u8 padding[4];
> > > > >   /* Followed by data corresponds to the specified control.
> > > > >
> > > > >    * The type of data must be defined in the spec.
> > > > >    * For example, if the driver queries profiles, this part should be
> > > > >    * an array of supported profiles of a given format.
> > > > >    */
> > > > >
> > > > > };
> > > > >
> > > > > WDYT?
> > > >
> > > > I think virtio_video_control_type should make sense. But I would disagree
> > > > with the need to have new QUERY_CONTROL and GET_CONTROL.
> > > >
> > > > I assume the set of supported controls is fixed for some particular format
> > > > on a given IP. So we'd propose to include controls into format
> > > > descriptors, so we don't need to QUERY_CONTROL. This way (with
> > > > 'virtio_video_format_list') we can define not-contiguos ranges, e.g. for
> > > > profiles.
> > > >
> > > > struct virtio_video_format_frame {
> > > >
> > > >         /* As proposed in Keiichi's prev email */
> > > >
> > > > };
> > > >
> > > > virtio_video_format_list {
> > > >
> > > >         le32 num_entries;
> > > >         u8 padding[4];
> > > >         /* Followed by le64 entries[] */
> > > >
> > > > };
> > > >
> > > > struct virtio_video_format_control {
> > > >
> > > >         le32 type;
> > > >         u8 padding[4];
> > > >         struct virtio_video_format_list values;
> > > >
> > > > };
> > > >
> > > > struct virtio_video_format_desc {
> > > >
> > > >         le64 mask;
> > > >         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > >         le32 planes_layout; /* See the thread [v2 0/1] */
> > > >         le32 num_frames;
> > > >         le32 num_controls;
> > > >         /* Followed by struct virtio_video_format_frame frames[] */
> > > >         /* Followed by struct virtio_video_format_control controls[] */
> > > >
> > > > };
> > >
> > > In my understanding, a set of supported levels depends on a profile.
> > > H.264 spec says that levels are specified within each profile.
> > > cf. "0.5 Profiles and levels" in
> > > https://www.itu.int/rec/T-REC-H.264-201906-I/en (V4L2's QUERYMENU doesn't
> > > seem to provide a way to query levels for each profile properly, though)
> > >
> > > So, if we want to have supported profiles and levels in format_desc as
> > > your idea, each supported profile should have a list of supported
> > > levels.
> > > I suppose it makes the structure of video_format_desc too complicated.
> > > I'd like to avoid this complexity caused by some specific formats.
> > >
> > > Instead, I'd like to keep virtio_video_format_desc minimal and have
> > > QUERY_CONTROL to query format-specific values like profiles and
> > > levels.
> > > This design would make it easy to extend controls when we want to
> > > support new types of formats. We will just need to define new
> > > VIRTIO_VIDEO_CONTROL_*.
> > > What do you think?
> >
> > Yes, I think we can keep it this way, this indeed provides more flexibility and
> > this way we won't need to modify the 'format' parsing logic when more controls
> > are added.
>
> Thanks! I'm preparing the next version of the patch.
>
> I have one concern about terminology.
> We have two different things called "controls" now:
> 1) Messages passed via controlq. i.e. "ctrl" in 'enum
> virtio_video_ctrl_type' and 'virtio_video_ctrl_hdr'.
> 2) Commands for profiles, levels, and bitrates similar to V4L2
> controls. e.g.., "CONTROL" in VIRTIO_VIDEO_CONTROL_*
>
> I feel calling both "controls" is somewhat confusing and I am
> wondering if we can't rename either of them.
>
> For example, how about renaming 2) to "command"? Then, we'll have
> {QUERY ,GET, SET}_COMMAND.
> It may sound strange, though.
> Do you have any idea?

"Command" sound like executing an action, but in this context we're
thinking more about parameters or, exactly, controls. The term
"control" is not limited to V4L2 and actually used in other subsystems
as well, for example the Android Camera HAL.

How about renaming 1) to "commandq" and then having the other symbols
use the abbreviated "cmd"?

Best regards,
Tomasz

>
> Best regards,
> Keiichi
>
> >
> > Best regards,
> > Dmitry.
> >
> > >
> > > Best regards,
> > > Keiichi
> > >
> > > > SET_CONTROL seems to be mandatory. If it succeeds, we can store the
> > > > current
> > > > value locally, so there is no need to have GET_CONTROL.
> > > >
> > > > The only exception is the initial (default) control value in the device.
> > > > But with the removal of 'function' and with addition of 'caps' instead,
> > > > the way to provide defaults is gone. So I suppose for formats we'll be
> > > > just using GET on driver start to get the 'defaults'. But the thing is
> > > > that for formats there are other uses for GET, but for controls GET
> > > > apparently does not make to much sense at runtime.
> > > >
> > > > > > Also, shouldn't the profile come together with level? Would make sense
> > > > > > for
> > > > > > encoders.
> > > > >
> > > > > Yeah. So, in the above idea of QUERY_CONTROL, profile should be
> > > > > required when querying supported levels.
> > > >
> > > > So probably should be enumerated together, not queried, as per the comment
> > > > above.
> > > >
> > > > > > > +\item[\field{mask}] is a bitset that represents the supported
> > > > > > > +  combination of input and output format. If \textit{i}-th bit is
> > > > > > > set
> > > > > > > +  in \field{mask} of \textit{j}-th \field{struct
> > > > > > > +  virtio_video_format_desc} for input, the device supports encoding
> > > > > > > or
> > > > > > > +  decoding from the \textit{j}-th input format to \textit{i}-th
> > > > > > > output
> > > > > > > +  format.
> > > > > > > +\item[\field{width, height}] represents a range of resolutions
> > > > > > > +  supported by the device. If its \field{step} is not applicable,
> > > > > > > its
> > > > > > > +  \field{min} is equal to its \field{max}.
> > > > > > > +\item[\field{num_rates}] is the length of an array
> > > > > > > \field{frame_rates}.
> > > > > > > In
> > > > > > > case of decoder, the driver SHOULD ignore this value.
> > > > > > > +\item[\field{frame_rates}] is an array of supported frame rates.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > >
> > > > > > I'd guess frame rates depend on the resolution as well. This
> > > > > > dependency
> > > > > > was
> > > > > > clear in the v1 spec, but in the v2 there is no dependency anymore. I
> > > > > > think we need to update this.
> > > > >
> > > > > That's a good point. I missed that dependency when updating the
> > > > > structures.
> > > > > So, let me update the structs like the following:
> > > > >
> > > > > struct virtio_video_format_frame {
> > > > >
> > > > >         struct virtio_video_format_range width;
> > > > >         struct virtio_video_format_range height;
> > > > >         le32 num_rates;
> > > > >         u8 padding[4];
> > > > >         /* Followed by struct virtio_video_format_range frame_rates[] */
> > > > >
> > > > > };
> > > > >
> > > > > struct virtio_video_format_desc {
> > > > >
> > > > >         le64 mask;
> > > > >         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > > >         le32 planes_layout; /* See the thread [v2 0/1] */
> > > > >         le32 num_frames;
> > > > >         u8 padding[4];
> > > > >         /* Followed by struct virtio_video_format_frame frames[] */
> > > > >
> > > > > };
> > > >
> > > > Yes, I do agree with this approach.
> > > >
> > > > Best regards,
> > > > Dmitry.
> > > >
> > > > > Best regards,
> > > > > Keiichi.
> > > > >
> > > > > > Best regards,
> > > > > > Dmitry.
> > > > > >
> > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > > > > > > +  within the device.
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +enum virtio_video_mem_type {
> > > > > > > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > > > > > > +};
> > > > > > > +
> > > > > > > +struct virtio_video_stream_create {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types
> > > > > > > */
> > > > > > > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types
> > > > > > > */
> > > > > > > +        char debug_name[64];
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{in_mem_type}] is a type of buffer management for input
> > > > > > > +buffers. The driver MUST set a value in \field{enum
> > > > > > > +virtio_video_mem_type}.
> > > > > > > +\item[\field{out_mem_type}] is a type of buffer management for
> > > > > > > output
> > > > > > > +buffers. The driver MUST set a value in \field{enum
> > > > > > > +virtio_video_mem_type}.
> > > > > > > +\item[\field{debug_name}] is a text string for a debug purpose.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream
> > > > > > > (context)
> > > > > > > +  within the device.
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +struct virtio_video_stream_destroy {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > > > > > > +  queued buffers through the pipeline.
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +struct virtio_video_stream_drain {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > > > > > > +  within the device.
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +struct virtio_video_mem_entry {
> > > > > > > +        le64 addr;
> > > > > > > +        le32 length;
> > > > > > > +        u8 padding[4];
> > > > > > > +};
> > > > > > > +
> > > > > > > +struct virtio_video_resource_create {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +        le32 resource_id;
> > > > > > > +        le32 nr_entries;
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > > +\item[\field{nr_entries}] number of \field{struct
> > > > > > > +  virtio_video_mem_entry} memory entries.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource
> > > > > > > descriptor
> > > > > > > +  within the device.
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +struct virtio_video_resource_destroy {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +        le32 resource_id;
> > > > > > > +        u8 padding[4];
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > > > > > > +queue.
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > > > > > > +
> > > > > > > +struct virtio_video_resource_queue {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +        le32 buf_type;
> > > > > > > +        le32 resource_id;
> > > > > > > +        le64 timestamp;
> > > > > > > +        le32 nr_data_size;
> > > > > > > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{buf_type}] buf_type of the .
> > > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > > +\item[\field{timestamp}] an abstract sequence counter that can be
> > > > > > > used
> > > > > > > +  for synchronisation.
> > > > > > > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > > > > > > +\item[\field{data_size}] number of data bytes within a plane.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +enum virtio_video_buffer_flag {
> > > > > > > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > > > > > > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > > > > > > +        /* Encoder only */
> > > > > > > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > > > > > > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > > > > > > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > > > > > > +};
> > > > > > > +
> > > > > > > +struct virtio_video_resource_queue_resp {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +        le64 timestamp;
> > > > > > > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > > > > > > +        le32 size;  /* Encoded size */
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{timestamp}] an abstract sequence counter that can be
> > > > > > > used
> > > > > > > +  for synchronisation.
> > > > > > > +\item[\field{flags}] mark specific buffers in the sequence.
> > > > > > > +\item[\field{size}] data size in the buffer (encoder only).
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +The device sends a response to the queue request asynchronously
> > > > > > > when
> > > > > > > +it has finished processing the buffer.
> > > > > > > +
> > > > > > > +The device SHOULD mark a buffer that triggered a processing error
> > > > > > > with
> > > > > > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > > > > > > +
> > > > > > > +The device MUST mark the last buffer with the
> > > > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > > > > > +sequence.
> > > > > > > +
> > > > > > > +In case of encoder, to denote a particular frame type the devie
> > > > > > > MUST
> > > > > > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > > > > > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > > > > > > +
> > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> > > > > > > +  buffers back from the input or the output queue of the device.
> > > > > > > The
> > > > > > > +  device SHOULD return all of the buffers from the respective queue
> > > > > > > as
> > > > > > > +  soon as possible without pushing the buffers through the
> > > > > > > processing
> > > > > > > +  pipeline.
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +struct virtio_video_queue_clear {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +        le32 buf_type;
> > > > > > > +        u8 padding[4];
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{buf_type}] buffer type.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> > > > > > > +  output of a stream.
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +struct virtio_video_plane_format {
> > > > > > > +        le32 plane_size;
> > > > > > > +        le32 stride;
> > > > > > > +        u8 padding[4];
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{plane_size}] size of the plane in bytes.
> > > > > > > +\item[\field{stride}] stride used for the plane in bytes.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +struct virtio_video_params {
> > > > > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > > > > > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> > > > > > > +        le32 frame_width;
> > > > > > > +        le32 frame_height;
> > > > > > > +        le32 min_buffers;
> > > > > > > +        le32 max_buffers;
> > > > > > > +        le32 frame_rate;
> > > > > > > +        struct virtio_video_crop {
> > > > > > > +                le32 left;
> > > > > > > +                le32 top;
> > > > > > > +                le32 width;
> > > > > > > +                le32 height;
> > > > > > > +        } crop;
> > > > > > > +        le32 num_planes;
> > > > > > > +        struct virtio_video_plane_format
> > > > > > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{frame_width}] the value to get/set.
> > > > > > > +\item[\field{frame_height}] the value to get/set.
> > > > > > > +\item[\field{pixel_format}] the value to get/set.
> > > > > > > +\item[\field{min_buffers}] minimum buffers required to handle the
> > > > > > > +  format (r/o).
> > > > > > > +\item[\field{max_buffers}] maximum buffers required to handle the
> > > > > > > +  format (r/o).
> > > > > > > +\item[\field{frame_rate}] the value to get/set.
> > > > > > > +\item[\field{crop}] cropping (composing) rectangle.
> > > > > > > +\item[\field{num_planes}] number of planes used to store pixel data
> > > > > > > +(r/o).
> > > > > > > +\item[\field{plane_formats}] description of each plane.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +struct virtio_video_get_params {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > > > > > +};
> > > > > > > +
> > > > > > > +struct virtio_video_get_params_resp {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +        struct virtio_video_params params;
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{buf_type}] buffer type.
> > > > > > > +\item[\field{params}] parameter values.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> > > > > > > +
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +struct virtio_video_set_params {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +        struct virtio_video_params params;
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{params}] parameters to set.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +Setting stream parameters might have side effects within the
> > > > > > > device.
> > > > > > > +For example, the device MAY perform alignment of width and height,
> > > > > > > +change the number of planes it uses for the format, or do whatever
> > > > > > > +changes that are required to continue normal operation using the
> > > > > > > +updated parameters. It is up to the driver to check the parameter
> > > > > > > set
> > > > > > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > > > > > > +
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\subsubsection{Device Operation: eventq}
> > > > > > > +
> > > > > > > +The device can report events on the event queue. The driver
> > > > > > > initially
> > > > > > > +populates the queue with device-writeable buffers. When the device
> > > > > > > +needs to report an event, it fills a buffer and notifies the
> > > > > > > driver.
> > > > > > > +The driver consumes the report and adds a new buffer to the
> > > > > > > virtqueue.
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +enum virtio_video_event_type {
> > > > > > > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > > > > > > +        /* For all functions */
> > > > > > > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > > > > > > +        /* For decoder only */
> > > > > > > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> > > > > > > +};
> > > > > > > +
> > > > > > > +struct virtio_video_event {
> > > > > > > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> > > > > > > +        le32 stream_id;
> > > > > > > +        u8 padding[4];
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{event_type}] type of the triggered event .
> > > > > > > +\item[\field{stream_id}] id of the source stream.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +The device MUST send
> > > > > > > VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > > > > > > +whenever it encounters new resolution data in the stream. This
> > > > > > > +includes the case of the initial device configuration after
> > > > > > > metadata
> > > > > > > +has been parsed and the case of dynamic resolution change.
> >
> >

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2020-01-08 12:46               ` Tomasz Figa
@ 2020-01-08 13:05                 ` Keiichi Watanabe
  2020-01-08 13:11                 ` Dmitry Sepp
  1 sibling, 0 replies; 73+ messages in thread
From: Keiichi Watanabe @ 2020-01-08 13:05 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: Dmitry Sepp, uril, virtio-dev, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Enrico Granata, Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens

On Wed, Jan 8, 2020 at 9:46 PM Tomasz Figa <tfiga@chromium.org> wrote:
>
> On Wed, Jan 8, 2020 at 9:15 PM Keiichi Watanabe <keiichiw@chromium.org> wrote:
> >
> > Hi Dmitry,
> >
> > On Wed, Jan 8, 2020 at 7:00 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
> > >
> > > Hi Keiichi,
> > >
> > > On Mittwoch, 8. Januar 2020 07:59:22 CET Keiichi Watanabe wrote:
> > > > Hi Dmitry,
> > > >
> > > > On Wed, Jan 8, 2020 at 1:50 AM Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > wrote:
> > > > > Hi Keiichi,
> > > > >
> > > > > thanks for the updates, please see my comments below.
> > > > >
> > > > > On Dienstag, 7. Januar 2020 14:24:31 CET Keiichi Watanabe wrote:
> > > > > > Hi Dmitry,
> > > > > >
> > > > > > On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp
> > > > > > <dmitry.sepp@opensynergy.com>
> > > > >
> > > > > wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > a couple of new comments:
> > > > > > >
> > > > > > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> > > > > > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > > > > >
> > > > > > > > The virtio video encoder device and decoder device provide
> > > > > > > > functionalities
> > > > > > > > to encode and decode video stream respectively.
> > > > > > > > Though video encoder and decoder are provided as different devices,
> > > > > > > > they
> > > > > > > > use a same protocol.
> > > > > > > >
> > > > > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > > > > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > > > > > > > ---
> > > > > > > >
> > > > > > > >  content.tex      |   1 +
> > > > > > > >  virtio-video.tex | 579
> > > > > > > >  +++++++++++++++++++++++++++++++++++++++++++++++
> > > > > > > >  2 files changed, 580 insertions(+)
> > > > > > > >  create mode 100644 virtio-video.tex
> > > > > > > >
> > > > > > > > diff --git a/content.tex b/content.tex
> > > > > > > > index 556b373..9e56839 100644
> > > > > > > > --- a/content.tex
> > > > > > > > +++ b/content.tex
> > > > > > > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > > > > > > > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> > > > > > > >
> > > > > > > >  \input{virtio-fs.tex}
> > > > > > > >  \input{virtio-rpmb.tex}
> > > > > > > >
> > > > > > > > +\input{virtio-video.tex}
> > > > > > > >
> > > > > > > >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> > > > > > > >
> > > > > > > > diff --git a/virtio-video.tex b/virtio-video.tex
> > > > > > > > new file mode 100644
> > > > > > > > index 0000000..30e728d
> > > > > > > > --- /dev/null
> > > > > > > > +++ b/virtio-video.tex
> > > > > > > > @@ -0,0 +1,579 @@
> > > > > > > > +\section{Video Device}\label{sec:Device Types / Video Device}
> > > > > > > > +
> > > > > > > > +The virtio video encoder device and decoder device are virtual
> > > > > > > > devices
> > > > > > > > that +supports encoding and decoding respectively. Though the
> > > > > > > > encoder
> > > > > > > > and the decoder +are different devices, they use the same protocol.
> > > > > > > > +
> > > > > > > > +\subsection{Device ID}\label{sec:Device Types / Video Device /
> > > > > > > > Device
> > > > > > > > ID}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[30] encoder device
> > > > > > > > +\item[31] decoder device
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device /
> > > > > > > > Virtqueues} +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[0] controlq - queue for sending control commands.
> > > > > > > > +\item[1] eventq - queue for sending events happened in the device.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\subsection{Feature bits}\label{sec:Device Types / Video Device /
> > > > > > > > Feature
> > > > > > > > bits} +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be
> > > > > > > > used
> > > > > > > > for
> > > > > > > > video +  buffers.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video
> > > > > > > > Device
> > > > > > > > / Feature bits} +
> > > > > > > > +The device MUST offer at least one of feature bits.
> > > > > > > > +
> > > > > > > > +\subsection{Device configuration layout}\label{sec:Device Types /
> > > > > > > > Video
> > > > > > > > Device / Device configuration layout} +
> > > > > > > > +Video device configuration uses the following layout structure:
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_config {
> > > > > > > > +        le32 max_cap_len;
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{max_cap_len}] defines the maximum length of a
> > > > > > > > descriptor
> > > > > > > > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > > > > > > > +  MUST set this value.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\subsection{Device Initialization}\label{sec:Device Types / Video
> > > > > > > > Device /
> > > > > > > > Device Initialization} +
> > > > > > > > +\devicenormative{\subsubsection}{Device Initialization}{Device
> > > > > > > > Types /
> > > > > > > > Video Device / Device Initialization} +
> > > > > > > > +The driver SHOULD query device capability by using the
> > > > > > > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the
> > > > > > > > initial
> > > > > > > > +setup.
> > > > > > > > +
> > > > > > > > +\subsection{Device Operation}\label{sec:Device Types / Video Device
> > > > > > > > /
> > > > > > > > Device Operation} +
> > > > > > > > +The driver allocates input and output buffers and queues the
> > > > > > > > buffers
> > > > > > > > +to the device. The device performs operations on the buffers
> > > > > > > > according
> > > > > > > > +to the function in question.
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Create stream}
> > > > > > > > +
> > > > > > > > +To process buffers, the device needs to associate them with a
> > > > > > > > certain
> > > > > > > > +video stream (essentially, a context). Streams are created by
> > > > > > > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > > > > > > > +determined by the device.
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Create buffers}
> > > > > > > > +
> > > > > > > > +Buffers are used to store the actual data as well as the relevant
> > > > > > > > +metadata. Scatter lists are supported, so the buffer doesn't need
> > > > > > > > to
> > > > > > > > +be contiguous in guest physical memory.
> > > > > > > > +
> > > > > > > > +\begin{itemize*}
> > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > > > > > > > +  resource that is backed by a buffer allocated from the driver's
> > > > > > > > +  memory.
> > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource
> > > > > > > > that
> > > > > > > > +  is no longer needed.
> > > > > > > > +\end{itemize*}
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Stream parameter control}
> > > > > > > > +
> > > > > > > > +\begin{itemize*}
> > > > > > > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream
> > > > > > > > parameters
> > > > > > > > for +  input and output streams from the device.
> > > > > > > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream
> > > > > > > > parameters to
> > > > > > > > the +  device.
> > > > > > > > +\item After setting stream parameters, the driver may issue
> > > > > > > > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and
> > > > > > > > output
> > > > > > > > can be +  changed implicitly by the device during the set operation.
> > > > > > > > +\end{itemize*}
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Process buffers}
> > > > > > > > +
> > > > > > > > +\begin{itemize*}
> > > > > > > > +\item If the function and the buffer type require so, write data to
> > > > > > > > +the buffer memory.
> > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > > > > > > > +processing in the device.
> > > > > > > > +\item The request completes asynchronously when the device has
> > > > > > > > +finished with the buffer.
> > > > > > > > +\end{itemize*}
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Buffer processing control}
> > > > > > > > +
> > > > > > > > +\begin{itemize*}
> > > > > > > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process
> > > > > > > > and
> > > > > > > > +  return all of the already queued buffers.
> > > > > > > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return
> > > > > > > > back
> > > > > > > > +  already queued buffers from the input or the output queue. This
> > > > > > > > also
> > > > > > > > +  includes input or output buffers that can be currently owned by
> > > > > > > > the
> > > > > > > > +  device's processing pipeline.
> > > > > > > > +\end{itemize*}
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Asynchronous events}
> > > > > > > > +
> > > > > > > > +While processing buffers, the device can send asynchronous event
> > > > > > > > +notifications to the driver. The behaviour depends on the exact
> > > > > > > > +stream. For example, the decoder device sends a resolution change
> > > > > > > > +event when it encounters new resolution metadata in the stream.
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Request header}
> > > > > > > > +
> > > > > > > > +All requests and responses on the control virt queue have a fixed
> > > > > > > > +header using the following layout structure and definitions:
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +enum virtio_video_ctrl_type {
> > > > > > > > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > > > > > > > +
> > > > > > > > +        /* request */
> > > > > > > > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > > > > > > > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > > > > > > > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > > > > > > > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > > > > > > > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > > > > > > > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > > > > > > > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > > > > > > > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > > > > > > > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > > > > > > > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > > > > > > > +
> > > > > > > > +        /* response */
> > > > > > > > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > > > > > > > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > > > > > > > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > > > > > > > +
> > > > > > > > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > > > > > > > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > > > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > > > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > > > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_ctrl_hdr {
> > > > > > > > +        le32 type;
> > > > > > > > +        le32 stream_id;
> > > > > > > > +        le32 len; /* Length of the structure in bytes. */
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{type}] is the type of the driver request or the device
> > > > > > > > +response.
> > > > > > > > +\item[\field{stream_id}] specifies a target stream.
> > > > > > > > +\item[\field{len}] is the length of data in bytes, which includes
> > > > > > > > +length of the header.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: controlq}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > > > > > > > +supported formats.
> > > > > > > > +
> > > > > > > > +The driver uses \field{struct virtio_video_get_capability} to send
> > > > > > > > a
> > > > > > > > +query request.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +enum virtio_video_buf_type {
> > > > > > > > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > > > > > > > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_get_capability {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        enum virtio_video_buf_type buf_type;
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{buf_type}] is the buffer type that the driver asks
> > > > > > > > +information about. The driver MUST set either
> > > > > > > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > > > > > > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > > > > > > > +
> > > > > > > > +The device responds a capability by using \field{struct
> > > > > > > > +virtio_video_get_capability_resp}.
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +enum virtio_video_format {
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > > > > > > > +        /* Raw formats */
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_YVU420,
> > > > > > > > +
> > > > > > > > +        /* Compressed formats */
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +enum virtio_video_profile {
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > > > > > > > +
> > > > > > > > +        /* H.264 */
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > > > > > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > > > > > > > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > > > > > > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > > > > > > > +        /* VP8 */
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_ANY =
> > > > > > > > VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_MAX =
> > > > > > > > VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > > > > > > > +
> > > > > > > > +        /* VP9 */
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 =
> > > > > > > > VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_MAX =
> > > > > > > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_format_range {
> > > > > > > > +        le32 min;
> > > > > > > > +        le32 max;
> > > > > > > > +        le32 step;
> > > > > > > > +        u8 paddings[4];
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_format_desc {
> > > > > > > > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > > > > > > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > > > > > > > +        le64 mask;
> > > > > > > > +        struct virtio_video_format_range width;
> > > > > > > > +        struct virtio_video_format_range height;
> > > > > > > > +        le32 num_rates;
> > > > > > > > +        u8 padding[4];
> > > > > > > > +        /* Followed by struct virtio_video_frame_rate frame_rates[]
> > > > > > > > */
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_get_capability_resp {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le32 num_descs;
> > > > > > > > +        /* Followed by struct virtio_video_format_desc desc[] */
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +The format description \field{struct virtio_video_format_desc}
> > > > > > > > +includes the following fields:
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{format}] specifies an image format. The device MUST
> > > > > > > > set
> > > > > > > > one
> > > > > > > > +  of \field{enum virtio_video_format}.
> > > > > > > > +\item[\field{profile}] specifies a profile of the compressed image
> > > > > > > > format
> > > > > > > > +  specified in \field{format}. The driver SHOULD ignore this value
> > > > > > > > if
> > > > > > > > +  \field{format} is a raw format.
> > > > > > >
> > > > > > > So how should this be used? The spec does not define any way to set
> > > > > > > profile for the device. It is very important for encoder.
> > > > > >
> > > > > > Thank you for pointing this.
> > > > > > These points are overlooked, as I didn't care about encoder usage
> > > > > > enough.
> > > > > >
> > > > > > After thinking it again, I think it's not a very good idea to include
> > > > > > supported profiles and levels in a struct for capability.
> > > > > > This is because these values are available only for limited number of
> > > > > > formats. Also, it's true that we need to have a way to set these values
> > > > > > as
> > > > > > Dmitry pointed.
> > > > >
> > > > > Yes, you are right. In fact, the approach of the v1 spec to keep controls
> > > > > separately was not correct.
> > > > >
> > > > > > Instead, it would make more sense to have additional three types of
> > > > > > controls for profiles, levels, and bitrates:
> > > > > > (1) QUERY_CONTROL: Query values supported by the device
> > > > > > (2) GET_CONTROL: Read a value that is set in the device
> > > > > > (3) SET_CONTROL: Set a value in the device
> > > > > >
> > > > > > These operations are similar to V4L2 controls.
> > > > > > (1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL, and
> > > > > > G_CTRL in V4L2, respectively.
> > > > > > Also, (3) would be similar to enum virtio_video_control_type in the
> > > > > > virtio-video v1 driver implementation in
> > > > > > https://markmail.org/message/dwghwdqsbl3gsjxu .
> > > > > >
> > > > > > For QUERY_CONTROL, my idea is like this:
> > > > > >
> > > > > > enum virtio_video_control_type {
> > > > > >
> > > > > >   VIRTIO_VIDEO_CONTROL_UNDEFINED = 0,
> > > > > >
> > > > > >   VIRTIO_VIDEO_CONTROL_BITRATE = 0x100,
> > > > > >   VIRTIO_VIDEO_CONTROL_PROFILE,
> > > > > >   VIRTIO_VIDEO_CONTROL_LEVEL,
> > > > > >
> > > > > > };
> > > > > >
> > > > > > struct virtio_video_query_control {
> > > > > >
> > > > > >   struct virtio_video_ctrl_hdr hdr;
> > > > > >   le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
> > > > > >   le32 length;
> > > > > >   /* Followed by additional data.
> > > > > >
> > > > > >    * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE,
> > > > > >    * the device must pass a codec format like H264 or VP9.
> > > > > >    * The requred data must be defined in the specification.
> > > > > >    */
> > > > > >
> > > > > > };
> > > > > >
> > > > > > struct virtio_video_query_control_resp {
> > > > > >
> > > > > >   struct virtio_video_ctrl_hdr hdr;
> > > > > >   le32 length;
> > > > > >   u8 padding[4];
> > > > > >   /* Followed by data corresponds to the specified control.
> > > > > >
> > > > > >    * The type of data must be defined in the spec.
> > > > > >    * For example, if the driver queries profiles, this part should be
> > > > > >    * an array of supported profiles of a given format.
> > > > > >    */
> > > > > >
> > > > > > };
> > > > > >
> > > > > > WDYT?
> > > > >
> > > > > I think virtio_video_control_type should make sense. But I would disagree
> > > > > with the need to have new QUERY_CONTROL and GET_CONTROL.
> > > > >
> > > > > I assume the set of supported controls is fixed for some particular format
> > > > > on a given IP. So we'd propose to include controls into format
> > > > > descriptors, so we don't need to QUERY_CONTROL. This way (with
> > > > > 'virtio_video_format_list') we can define not-contiguos ranges, e.g. for
> > > > > profiles.
> > > > >
> > > > > struct virtio_video_format_frame {
> > > > >
> > > > >         /* As proposed in Keiichi's prev email */
> > > > >
> > > > > };
> > > > >
> > > > > virtio_video_format_list {
> > > > >
> > > > >         le32 num_entries;
> > > > >         u8 padding[4];
> > > > >         /* Followed by le64 entries[] */
> > > > >
> > > > > };
> > > > >
> > > > > struct virtio_video_format_control {
> > > > >
> > > > >         le32 type;
> > > > >         u8 padding[4];
> > > > >         struct virtio_video_format_list values;
> > > > >
> > > > > };
> > > > >
> > > > > struct virtio_video_format_desc {
> > > > >
> > > > >         le64 mask;
> > > > >         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > > >         le32 planes_layout; /* See the thread [v2 0/1] */
> > > > >         le32 num_frames;
> > > > >         le32 num_controls;
> > > > >         /* Followed by struct virtio_video_format_frame frames[] */
> > > > >         /* Followed by struct virtio_video_format_control controls[] */
> > > > >
> > > > > };
> > > >
> > > > In my understanding, a set of supported levels depends on a profile.
> > > > H.264 spec says that levels are specified within each profile.
> > > > cf. "0.5 Profiles and levels" in
> > > > https://www.itu.int/rec/T-REC-H.264-201906-I/en (V4L2's QUERYMENU doesn't
> > > > seem to provide a way to query levels for each profile properly, though)
> > > >
> > > > So, if we want to have supported profiles and levels in format_desc as
> > > > your idea, each supported profile should have a list of supported
> > > > levels.
> > > > I suppose it makes the structure of video_format_desc too complicated.
> > > > I'd like to avoid this complexity caused by some specific formats.
> > > >
> > > > Instead, I'd like to keep virtio_video_format_desc minimal and have
> > > > QUERY_CONTROL to query format-specific values like profiles and
> > > > levels.
> > > > This design would make it easy to extend controls when we want to
> > > > support new types of formats. We will just need to define new
> > > > VIRTIO_VIDEO_CONTROL_*.
> > > > What do you think?
> > >
> > > Yes, I think we can keep it this way, this indeed provides more flexibility and
> > > this way we won't need to modify the 'format' parsing logic when more controls
> > > are added.
> >
> > Thanks! I'm preparing the next version of the patch.
> >
> > I have one concern about terminology.
> > We have two different things called "controls" now:
> > 1) Messages passed via controlq. i.e. "ctrl" in 'enum
> > virtio_video_ctrl_type' and 'virtio_video_ctrl_hdr'.
> > 2) Commands for profiles, levels, and bitrates similar to V4L2
> > controls. e.g.., "CONTROL" in VIRTIO_VIDEO_CONTROL_*
> >
> > I feel calling both "controls" is somewhat confusing and I am
> > wondering if we can't rename either of them.
> >
> > For example, how about renaming 2) to "command"? Then, we'll have
> > {QUERY ,GET, SET}_COMMAND.
> > It may sound strange, though.
> > Do you have any idea?
>
> "Command" sound like executing an action, but in this context we're
> thinking more about parameters or, exactly, controls. The term
> "control" is not limited to V4L2 and actually used in other subsystems
> as well, for example the Android Camera HAL.
>
> How about renaming 1) to "commandq" and then having the other symbols
> use the abbreviated "cmd"?

Thanks! Keep calling 2) "control" and renaming 1) sounds better now.
Actually, virtio-gpu's requests are called "command". (The queue name
is "controlq", though)

One concern is that the name "controlq" is commonly used among some
virtio devices such as virtio-net and virtio-gpu. However, it won't
make more sense to stick to using the same queue name as other virtio
protocols.

Keiichi

>
> Best regards,
> Tomasz
>
> >
> > Best regards,
> > Keiichi
> >
> > >
> > > Best regards,
> > > Dmitry.
> > >
> > > >
> > > > Best regards,
> > > > Keiichi
> > > >
> > > > > SET_CONTROL seems to be mandatory. If it succeeds, we can store the
> > > > > current
> > > > > value locally, so there is no need to have GET_CONTROL.
> > > > >
> > > > > The only exception is the initial (default) control value in the device.
> > > > > But with the removal of 'function' and with addition of 'caps' instead,
> > > > > the way to provide defaults is gone. So I suppose for formats we'll be
> > > > > just using GET on driver start to get the 'defaults'. But the thing is
> > > > > that for formats there are other uses for GET, but for controls GET
> > > > > apparently does not make to much sense at runtime.
> > > > >
> > > > > > > Also, shouldn't the profile come together with level? Would make sense
> > > > > > > for
> > > > > > > encoders.
> > > > > >
> > > > > > Yeah. So, in the above idea of QUERY_CONTROL, profile should be
> > > > > > required when querying supported levels.
> > > > >
> > > > > So probably should be enumerated together, not queried, as per the comment
> > > > > above.
> > > > >
> > > > > > > > +\item[\field{mask}] is a bitset that represents the supported
> > > > > > > > +  combination of input and output format. If \textit{i}-th bit is
> > > > > > > > set
> > > > > > > > +  in \field{mask} of \textit{j}-th \field{struct
> > > > > > > > +  virtio_video_format_desc} for input, the device supports encoding
> > > > > > > > or
> > > > > > > > +  decoding from the \textit{j}-th input format to \textit{i}-th
> > > > > > > > output
> > > > > > > > +  format.
> > > > > > > > +\item[\field{width, height}] represents a range of resolutions
> > > > > > > > +  supported by the device. If its \field{step} is not applicable,
> > > > > > > > its
> > > > > > > > +  \field{min} is equal to its \field{max}.
> > > > > > > > +\item[\field{num_rates}] is the length of an array
> > > > > > > > \field{frame_rates}.
> > > > > > > > In
> > > > > > > > case of decoder, the driver SHOULD ignore this value.
> > > > > > > > +\item[\field{frame_rates}] is an array of supported frame rates.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > >
> > > > > > > I'd guess frame rates depend on the resolution as well. This
> > > > > > > dependency
> > > > > > > was
> > > > > > > clear in the v1 spec, but in the v2 there is no dependency anymore. I
> > > > > > > think we need to update this.
> > > > > >
> > > > > > That's a good point. I missed that dependency when updating the
> > > > > > structures.
> > > > > > So, let me update the structs like the following:
> > > > > >
> > > > > > struct virtio_video_format_frame {
> > > > > >
> > > > > >         struct virtio_video_format_range width;
> > > > > >         struct virtio_video_format_range height;
> > > > > >         le32 num_rates;
> > > > > >         u8 padding[4];
> > > > > >         /* Followed by struct virtio_video_format_range frame_rates[] */
> > > > > >
> > > > > > };
> > > > > >
> > > > > > struct virtio_video_format_desc {
> > > > > >
> > > > > >         le64 mask;
> > > > > >         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > > > >         le32 planes_layout; /* See the thread [v2 0/1] */
> > > > > >         le32 num_frames;
> > > > > >         u8 padding[4];
> > > > > >         /* Followed by struct virtio_video_format_frame frames[] */
> > > > > >
> > > > > > };
> > > > >
> > > > > Yes, I do agree with this approach.
> > > > >
> > > > > Best regards,
> > > > > Dmitry.
> > > > >
> > > > > > Best regards,
> > > > > > Keiichi.
> > > > > >
> > > > > > > Best regards,
> > > > > > > Dmitry.
> > > > > > >
> > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > > > > > > > +  within the device.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +enum virtio_video_mem_type {
> > > > > > > > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_stream_create {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types
> > > > > > > > */
> > > > > > > > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types
> > > > > > > > */
> > > > > > > > +        char debug_name[64];
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{in_mem_type}] is a type of buffer management for input
> > > > > > > > +buffers. The driver MUST set a value in \field{enum
> > > > > > > > +virtio_video_mem_type}.
> > > > > > > > +\item[\field{out_mem_type}] is a type of buffer management for
> > > > > > > > output
> > > > > > > > +buffers. The driver MUST set a value in \field{enum
> > > > > > > > +virtio_video_mem_type}.
> > > > > > > > +\item[\field{debug_name}] is a text string for a debug purpose.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream
> > > > > > > > (context)
> > > > > > > > +  within the device.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_stream_destroy {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > > > > > > > +  queued buffers through the pipeline.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_stream_drain {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > > > > > > > +  within the device.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_mem_entry {
> > > > > > > > +        le64 addr;
> > > > > > > > +        le32 length;
> > > > > > > > +        u8 padding[4];
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_resource_create {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le32 resource_id;
> > > > > > > > +        le32 nr_entries;
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > > > +\item[\field{nr_entries}] number of \field{struct
> > > > > > > > +  virtio_video_mem_entry} memory entries.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource
> > > > > > > > descriptor
> > > > > > > > +  within the device.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_resource_destroy {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le32 resource_id;
> > > > > > > > +        u8 padding[4];
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > > > > > > > +queue.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > > > > > > > +
> > > > > > > > +struct virtio_video_resource_queue {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le32 buf_type;
> > > > > > > > +        le32 resource_id;
> > > > > > > > +        le64 timestamp;
> > > > > > > > +        le32 nr_data_size;
> > > > > > > > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{buf_type}] buf_type of the .
> > > > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > > > +\item[\field{timestamp}] an abstract sequence counter that can be
> > > > > > > > used
> > > > > > > > +  for synchronisation.
> > > > > > > > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > > > > > > > +\item[\field{data_size}] number of data bytes within a plane.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +enum virtio_video_buffer_flag {
> > > > > > > > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > > > > > > > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > > > > > > > +        /* Encoder only */
> > > > > > > > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > > > > > > > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > > > > > > > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_resource_queue_resp {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le64 timestamp;
> > > > > > > > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > > > > > > > +        le32 size;  /* Encoded size */
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{timestamp}] an abstract sequence counter that can be
> > > > > > > > used
> > > > > > > > +  for synchronisation.
> > > > > > > > +\item[\field{flags}] mark specific buffers in the sequence.
> > > > > > > > +\item[\field{size}] data size in the buffer (encoder only).
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +The device sends a response to the queue request asynchronously
> > > > > > > > when
> > > > > > > > +it has finished processing the buffer.
> > > > > > > > +
> > > > > > > > +The device SHOULD mark a buffer that triggered a processing error
> > > > > > > > with
> > > > > > > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > > > > > > > +
> > > > > > > > +The device MUST mark the last buffer with the
> > > > > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > > > > > > +sequence.
> > > > > > > > +
> > > > > > > > +In case of encoder, to denote a particular frame type the devie
> > > > > > > > MUST
> > > > > > > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > > > > > > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> > > > > > > > +  buffers back from the input or the output queue of the device.
> > > > > > > > The
> > > > > > > > +  device SHOULD return all of the buffers from the respective queue
> > > > > > > > as
> > > > > > > > +  soon as possible without pushing the buffers through the
> > > > > > > > processing
> > > > > > > > +  pipeline.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_queue_clear {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le32 buf_type;
> > > > > > > > +        u8 padding[4];
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{buf_type}] buffer type.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> > > > > > > > +  output of a stream.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_plane_format {
> > > > > > > > +        le32 plane_size;
> > > > > > > > +        le32 stride;
> > > > > > > > +        u8 padding[4];
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{plane_size}] size of the plane in bytes.
> > > > > > > > +\item[\field{stride}] stride used for the plane in bytes.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_params {
> > > > > > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > > > > > > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> > > > > > > > +        le32 frame_width;
> > > > > > > > +        le32 frame_height;
> > > > > > > > +        le32 min_buffers;
> > > > > > > > +        le32 max_buffers;
> > > > > > > > +        le32 frame_rate;
> > > > > > > > +        struct virtio_video_crop {
> > > > > > > > +                le32 left;
> > > > > > > > +                le32 top;
> > > > > > > > +                le32 width;
> > > > > > > > +                le32 height;
> > > > > > > > +        } crop;
> > > > > > > > +        le32 num_planes;
> > > > > > > > +        struct virtio_video_plane_format
> > > > > > > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{frame_width}] the value to get/set.
> > > > > > > > +\item[\field{frame_height}] the value to get/set.
> > > > > > > > +\item[\field{pixel_format}] the value to get/set.
> > > > > > > > +\item[\field{min_buffers}] minimum buffers required to handle the
> > > > > > > > +  format (r/o).
> > > > > > > > +\item[\field{max_buffers}] maximum buffers required to handle the
> > > > > > > > +  format (r/o).
> > > > > > > > +\item[\field{frame_rate}] the value to get/set.
> > > > > > > > +\item[\field{crop}] cropping (composing) rectangle.
> > > > > > > > +\item[\field{num_planes}] number of planes used to store pixel data
> > > > > > > > +(r/o).
> > > > > > > > +\item[\field{plane_formats}] description of each plane.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_get_params {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_get_params_resp {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        struct virtio_video_params params;
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{buf_type}] buffer type.
> > > > > > > > +\item[\field{params}] parameter values.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> > > > > > > > +
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_set_params {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        struct virtio_video_params params;
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{params}] parameters to set.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +Setting stream parameters might have side effects within the
> > > > > > > > device.
> > > > > > > > +For example, the device MAY perform alignment of width and height,
> > > > > > > > +change the number of planes it uses for the format, or do whatever
> > > > > > > > +changes that are required to continue normal operation using the
> > > > > > > > +updated parameters. It is up to the driver to check the parameter
> > > > > > > > set
> > > > > > > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > > > > > > > +
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: eventq}
> > > > > > > > +
> > > > > > > > +The device can report events on the event queue. The driver
> > > > > > > > initially
> > > > > > > > +populates the queue with device-writeable buffers. When the device
> > > > > > > > +needs to report an event, it fills a buffer and notifies the
> > > > > > > > driver.
> > > > > > > > +The driver consumes the report and adds a new buffer to the
> > > > > > > > virtqueue.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +enum virtio_video_event_type {
> > > > > > > > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > > > > > > > +        /* For all functions */
> > > > > > > > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > > > > > > > +        /* For decoder only */
> > > > > > > > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_event {
> > > > > > > > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> > > > > > > > +        le32 stream_id;
> > > > > > > > +        u8 padding[4];
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{event_type}] type of the triggered event .
> > > > > > > > +\item[\field{stream_id}] id of the source stream.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +The device MUST send
> > > > > > > > VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > > > > > > > +whenever it encounters new resolution data in the stream. This
> > > > > > > > +includes the case of the initial device configuration after
> > > > > > > > metadata
> > > > > > > > +has been parsed and the case of dynamic resolution change.
> > >
> > >

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2020-01-08 12:46               ` Tomasz Figa
  2020-01-08 13:05                 ` Keiichi Watanabe
@ 2020-01-08 13:11                 ` Dmitry Sepp
  2020-01-08 13:23                   ` [virtio-dev] " Keiichi Watanabe
  1 sibling, 1 reply; 73+ messages in thread
From: Dmitry Sepp @ 2020-01-08 13:11 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: Keiichi Watanabe, uril, virtio-dev, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Enrico Granata, Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens

Hi Tomasz, Keiichi,

On Mittwoch, 8. Januar 2020 13:46:25 CET Tomasz Figa wrote:
> On Wed, Jan 8, 2020 at 9:15 PM Keiichi Watanabe <keiichiw@chromium.org> 
wrote:
> > Hi Dmitry,
> > 
> > On Wed, Jan 8, 2020 at 7:00 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> 
wrote:
> > > Hi Keiichi,
> > > 
> > > On Mittwoch, 8. Januar 2020 07:59:22 CET Keiichi Watanabe wrote:
> > > > Hi Dmitry,
> > > > 
> > > > On Wed, Jan 8, 2020 at 1:50 AM Dmitry Sepp
> > > > <dmitry.sepp@opensynergy.com>
> > > 
> > > wrote:
> > > > > Hi Keiichi,
> > > > > 
> > > > > thanks for the updates, please see my comments below.
> > > > > 
> > > > > On Dienstag, 7. Januar 2020 14:24:31 CET Keiichi Watanabe wrote:
> > > > > > Hi Dmitry,
> > > > > > 
> > > > > > On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp
> > > > > > <dmitry.sepp@opensynergy.com>
> > > > > 
> > > > > wrote:
> > > > > > > Hi,
> > > > > > > 
> > > > > > > a couple of new comments:
> > > > > > > 
> > > > > > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe 
wrote:
> > > > > > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > > > > > 
> > > > > > > > The virtio video encoder device and decoder device provide
> > > > > > > > functionalities
> > > > > > > > to encode and decode video stream respectively.
> > > > > > > > Though video encoder and decoder are provided as different
> > > > > > > > devices,
> > > > > > > > they
> > > > > > > > use a same protocol.
> > > > > > > > 
> > > > > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > > > > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > > > > > > > ---
> > > > > > > > 
> > > > > > > >  content.tex      |   1 +
> > > > > > > >  virtio-video.tex | 579
> > > > > > > >  +++++++++++++++++++++++++++++++++++++++++++++++
> > > > > > > >  2 files changed, 580 insertions(+)
> > > > > > > >  create mode 100644 virtio-video.tex
> > > > > > > > 
> > > > > > > > diff --git a/content.tex b/content.tex
> > > > > > > > index 556b373..9e56839 100644
> > > > > > > > --- a/content.tex
> > > > > > > > +++ b/content.tex
> > > > > > > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > > > > > > > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> > > > > > > > 
> > > > > > > >  \input{virtio-fs.tex}
> > > > > > > >  \input{virtio-rpmb.tex}
> > > > > > > > 
> > > > > > > > +\input{virtio-video.tex}
> > > > > > > > 
> > > > > > > >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature
> > > > > > > >  Bits}
> > > > > > > > 
> > > > > > > > diff --git a/virtio-video.tex b/virtio-video.tex
> > > > > > > > new file mode 100644
> > > > > > > > index 0000000..30e728d
> > > > > > > > --- /dev/null
> > > > > > > > +++ b/virtio-video.tex
> > > > > > > > @@ -0,0 +1,579 @@
> > > > > > > > +\section{Video Device}\label{sec:Device Types / Video Device}
> > > > > > > > +
> > > > > > > > +The virtio video encoder device and decoder device are
> > > > > > > > virtual
> > > > > > > > devices
> > > > > > > > that +supports encoding and decoding respectively. Though the
> > > > > > > > encoder
> > > > > > > > and the decoder +are different devices, they use the same
> > > > > > > > protocol.
> > > > > > > > +
> > > > > > > > +\subsection{Device ID}\label{sec:Device Types / Video Device
> > > > > > > > /
> > > > > > > > Device
> > > > > > > > ID}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[30] encoder device
> > > > > > > > +\item[31] decoder device
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device
> > > > > > > > /
> > > > > > > > Virtqueues} +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[0] controlq - queue for sending control commands.
> > > > > > > > +\item[1] eventq - queue for sending events happened in the
> > > > > > > > device.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\subsection{Feature bits}\label{sec:Device Types / Video
> > > > > > > > Device /
> > > > > > > > Feature
> > > > > > > > bits} +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages
> > > > > > > > can be
> > > > > > > > used
> > > > > > > > for
> > > > > > > > video +  buffers.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\devicenormative{\subsubsection}{Feature bits}{Device Types /
> > > > > > > > Video
> > > > > > > > Device
> > > > > > > > / Feature bits} +
> > > > > > > > +The device MUST offer at least one of feature bits.
> > > > > > > > +
> > > > > > > > +\subsection{Device configuration layout}\label{sec:Device
> > > > > > > > Types /
> > > > > > > > Video
> > > > > > > > Device / Device configuration layout} +
> > > > > > > > +Video device configuration uses the following layout
> > > > > > > > structure:
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_config {
> > > > > > > > +        le32 max_cap_len;
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{max_cap_len}] defines the maximum length of a
> > > > > > > > descriptor
> > > > > > > > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The
> > > > > > > > device
> > > > > > > > +  MUST set this value.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\subsection{Device Initialization}\label{sec:Device Types /
> > > > > > > > Video
> > > > > > > > Device /
> > > > > > > > Device Initialization} +
> > > > > > > > +\devicenormative{\subsubsection}{Device
> > > > > > > > Initialization}{Device
> > > > > > > > Types /
> > > > > > > > Video Device / Device Initialization} +
> > > > > > > > +The driver SHOULD query device capability by using the
> > > > > > > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for
> > > > > > > > the
> > > > > > > > initial
> > > > > > > > +setup.
> > > > > > > > +
> > > > > > > > +\subsection{Device Operation}\label{sec:Device Types / Video
> > > > > > > > Device
> > > > > > > > /
> > > > > > > > Device Operation} +
> > > > > > > > +The driver allocates input and output buffers and queues the
> > > > > > > > buffers
> > > > > > > > +to the device. The device performs operations on the buffers
> > > > > > > > according
> > > > > > > > +to the function in question.
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Create stream}
> > > > > > > > +
> > > > > > > > +To process buffers, the device needs to associate them with a
> > > > > > > > certain
> > > > > > > > +video stream (essentially, a context). Streams are created by
> > > > > > > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > > > > > > > +determined by the device.
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Create buffers}
> > > > > > > > +
> > > > > > > > +Buffers are used to store the actual data as well as the
> > > > > > > > relevant
> > > > > > > > +metadata. Scatter lists are supported, so the buffer doesn't
> > > > > > > > need
> > > > > > > > to
> > > > > > > > +be contiguous in guest physical memory.
> > > > > > > > +
> > > > > > > > +\begin{itemize*}
> > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio
> > > > > > > > video
> > > > > > > > +  resource that is backed by a buffer allocated from the
> > > > > > > > driver's
> > > > > > > > +  memory.
> > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a
> > > > > > > > resource
> > > > > > > > that
> > > > > > > > +  is no longer needed.
> > > > > > > > +\end{itemize*}
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Stream parameter control}
> > > > > > > > +
> > > > > > > > +\begin{itemize*}
> > > > > > > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream
> > > > > > > > parameters
> > > > > > > > for +  input and output streams from the device.
> > > > > > > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream
> > > > > > > > parameters to
> > > > > > > > the +  device.
> > > > > > > > +\item After setting stream parameters, the driver may issue
> > > > > > > > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input
> > > > > > > > and
> > > > > > > > output
> > > > > > > > can be +  changed implicitly by the device during the set
> > > > > > > > operation.
> > > > > > > > +\end{itemize*}
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Process buffers}
> > > > > > > > +
> > > > > > > > +\begin{itemize*}
> > > > > > > > +\item If the function and the buffer type require so, write
> > > > > > > > data to
> > > > > > > > +the buffer memory.
> > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer
> > > > > > > > for
> > > > > > > > +processing in the device.
> > > > > > > > +\item The request completes asynchronously when the device
> > > > > > > > has
> > > > > > > > +finished with the buffer.
> > > > > > > > +\end{itemize*}
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Buffer processing control}
> > > > > > > > +
> > > > > > > > +\begin{itemize*}
> > > > > > > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to
> > > > > > > > process
> > > > > > > > and
> > > > > > > > +  return all of the already queued buffers.
> > > > > > > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to
> > > > > > > > return
> > > > > > > > back
> > > > > > > > +  already queued buffers from the input or the output queue.
> > > > > > > > This
> > > > > > > > also
> > > > > > > > +  includes input or output buffers that can be currently
> > > > > > > > owned by
> > > > > > > > the
> > > > > > > > +  device's processing pipeline.
> > > > > > > > +\end{itemize*}
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Asynchronous events}
> > > > > > > > +
> > > > > > > > +While processing buffers, the device can send asynchronous
> > > > > > > > event
> > > > > > > > +notifications to the driver. The behaviour depends on the
> > > > > > > > exact
> > > > > > > > +stream. For example, the decoder device sends a resolution
> > > > > > > > change
> > > > > > > > +event when it encounters new resolution metadata in the
> > > > > > > > stream.
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Request header}
> > > > > > > > +
> > > > > > > > +All requests and responses on the control virt queue have a
> > > > > > > > fixed
> > > > > > > > +header using the following layout structure and definitions:
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +enum virtio_video_ctrl_type {
> > > > > > > > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > > > > > > > +
> > > > > > > > +        /* request */
> > > > > > > > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > > > > > > > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > > > > > > > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > > > > > > > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > > > > > > > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > > > > > > > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > > > > > > > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > > > > > > > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > > > > > > > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > > > > > > > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > > > > > > > +
> > > > > > > > +        /* response */
> > > > > > > > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > > > > > > > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > > > > > > > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > > > > > > > +
> > > > > > > > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > > > > > > > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > > > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > > > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > > > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_ctrl_hdr {
> > > > > > > > +        le32 type;
> > > > > > > > +        le32 stream_id;
> > > > > > > > +        le32 len; /* Length of the structure in bytes. */
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{type}] is the type of the driver request or the
> > > > > > > > device
> > > > > > > > +response.
> > > > > > > > +\item[\field{stream_id}] specifies a target stream.
> > > > > > > > +\item[\field{len}] is the length of data in bytes, which
> > > > > > > > includes
> > > > > > > > +length of the header.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: controlq}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information
> > > > > > > > about
> > > > > > > > +supported formats.
> > > > > > > > +
> > > > > > > > +The driver uses \field{struct virtio_video_get_capability} to
> > > > > > > > send
> > > > > > > > a
> > > > > > > > +query request.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +enum virtio_video_buf_type {
> > > > > > > > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > > > > > > > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_get_capability {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        enum virtio_video_buf_type buf_type;
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{buf_type}] is the buffer type that the driver
> > > > > > > > asks
> > > > > > > > +information about. The driver MUST set either
> > > > > > > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > > > > > > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > > > > > > > +
> > > > > > > > +The device responds a capability by using \field{struct
> > > > > > > > +virtio_video_get_capability_resp}.
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +enum virtio_video_format {
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > > > > > > > +        /* Raw formats */
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_YVU420,
> > > > > > > > +
> > > > > > > > +        /* Compressed formats */
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +enum virtio_video_profile {
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > > > > > > > +
> > > > > > > > +        /* H.264 */
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > > > > > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > > > > > > > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > > > > > > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > > > > > > > +        /* VP8 */
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_ANY =
> > > > > > > > VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_MAX =
> > > > > > > > VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > > > > > > > +
> > > > > > > > +        /* VP9 */
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 =
> > > > > > > > VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_MAX =
> > > > > > > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_format_range {
> > > > > > > > +        le32 min;
> > > > > > > > +        le32 max;
> > > > > > > > +        le32 step;
> > > > > > > > +        u8 paddings[4];
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_format_desc {
> > > > > > > > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types
> > > > > > > > */
> > > > > > > > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types
> > > > > > > > */
> > > > > > > > +        le64 mask;
> > > > > > > > +        struct virtio_video_format_range width;
> > > > > > > > +        struct virtio_video_format_range height;
> > > > > > > > +        le32 num_rates;
> > > > > > > > +        u8 padding[4];
> > > > > > > > +        /* Followed by struct virtio_video_frame_rate
> > > > > > > > frame_rates[]
> > > > > > > > */
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_get_capability_resp {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le32 num_descs;
> > > > > > > > +        /* Followed by struct virtio_video_format_desc desc[]
> > > > > > > > */
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +The format description \field{struct
> > > > > > > > virtio_video_format_desc}
> > > > > > > > +includes the following fields:
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{format}] specifies an image format. The device
> > > > > > > > MUST
> > > > > > > > set
> > > > > > > > one
> > > > > > > > +  of \field{enum virtio_video_format}.
> > > > > > > > +\item[\field{profile}] specifies a profile of the compressed
> > > > > > > > image
> > > > > > > > format
> > > > > > > > +  specified in \field{format}. The driver SHOULD ignore this
> > > > > > > > value
> > > > > > > > if
> > > > > > > > +  \field{format} is a raw format.
> > > > > > > 
> > > > > > > So how should this be used? The spec does not define any way to
> > > > > > > set
> > > > > > > profile for the device. It is very important for encoder.
> > > > > > 
> > > > > > Thank you for pointing this.
> > > > > > These points are overlooked, as I didn't care about encoder usage
> > > > > > enough.
> > > > > > 
> > > > > > After thinking it again, I think it's not a very good idea to
> > > > > > include
> > > > > > supported profiles and levels in a struct for capability.
> > > > > > This is because these values are available only for limited number
> > > > > > of
> > > > > > formats. Also, it's true that we need to have a way to set these
> > > > > > values
> > > > > > as
> > > > > > Dmitry pointed.
> > > > > 
> > > > > Yes, you are right. In fact, the approach of the v1 spec to keep
> > > > > controls
> > > > > separately was not correct.
> > > > > 
> > > > > > Instead, it would make more sense to have additional three types
> > > > > > of
> > > > > > controls for profiles, levels, and bitrates:
> > > > > > (1) QUERY_CONTROL: Query values supported by the device
> > > > > > (2) GET_CONTROL: Read a value that is set in the device
> > > > > > (3) SET_CONTROL: Set a value in the device
> > > > > > 
> > > > > > These operations are similar to V4L2 controls.
> > > > > > (1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL,
> > > > > > and
> > > > > > G_CTRL in V4L2, respectively.
> > > > > > Also, (3) would be similar to enum virtio_video_control_type in
> > > > > > the
> > > > > > virtio-video v1 driver implementation in
> > > > > > https://markmail.org/message/dwghwdqsbl3gsjxu .
> > > > > > 
> > > > > > For QUERY_CONTROL, my idea is like this:
> > > > > > 
> > > > > > enum virtio_video_control_type {
> > > > > > 
> > > > > >   VIRTIO_VIDEO_CONTROL_UNDEFINED = 0,
> > > > > >   
> > > > > >   VIRTIO_VIDEO_CONTROL_BITRATE = 0x100,
> > > > > >   VIRTIO_VIDEO_CONTROL_PROFILE,
> > > > > >   VIRTIO_VIDEO_CONTROL_LEVEL,
> > > > > > 
> > > > > > };
> > > > > > 
> > > > > > struct virtio_video_query_control {
> > > > > > 
> > > > > >   struct virtio_video_ctrl_hdr hdr;
> > > > > >   le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
> > > > > >   le32 length;
> > > > > >   /* Followed by additional data.
> > > > > >   
> > > > > >    * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE,
> > > > > >    * the device must pass a codec format like H264 or VP9.
> > > > > >    * The requred data must be defined in the specification.
> > > > > >    */
> > > > > > 
> > > > > > };
> > > > > > 
> > > > > > struct virtio_video_query_control_resp {
> > > > > > 
> > > > > >   struct virtio_video_ctrl_hdr hdr;
> > > > > >   le32 length;
> > > > > >   u8 padding[4];
> > > > > >   /* Followed by data corresponds to the specified control.
> > > > > >   
> > > > > >    * The type of data must be defined in the spec.
> > > > > >    * For example, if the driver queries profiles, this part should
> > > > > >    be
> > > > > >    * an array of supported profiles of a given format.
> > > > > >    */
> > > > > > 
> > > > > > };
> > > > > > 
> > > > > > WDYT?
> > > > > 
> > > > > I think virtio_video_control_type should make sense. But I would
> > > > > disagree
> > > > > with the need to have new QUERY_CONTROL and GET_CONTROL.
> > > > > 
> > > > > I assume the set of supported controls is fixed for some particular
> > > > > format
> > > > > on a given IP. So we'd propose to include controls into format
> > > > > descriptors, so we don't need to QUERY_CONTROL. This way (with
> > > > > 'virtio_video_format_list') we can define not-contiguos ranges, e.g.
> > > > > for
> > > > > profiles.
> > > > > 
> > > > > struct virtio_video_format_frame {
> > > > > 
> > > > >         /* As proposed in Keiichi's prev email */
> > > > > 
> > > > > };
> > > > > 
> > > > > virtio_video_format_list {
> > > > > 
> > > > >         le32 num_entries;
> > > > >         u8 padding[4];
> > > > >         /* Followed by le64 entries[] */
> > > > > 
> > > > > };
> > > > > 
> > > > > struct virtio_video_format_control {
> > > > > 
> > > > >         le32 type;
> > > > >         u8 padding[4];
> > > > >         struct virtio_video_format_list values;
> > > > > 
> > > > > };
> > > > > 
> > > > > struct virtio_video_format_desc {
> > > > > 
> > > > >         le64 mask;
> > > > >         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > > >         le32 planes_layout; /* See the thread [v2 0/1] */
> > > > >         le32 num_frames;
> > > > >         le32 num_controls;
> > > > >         /* Followed by struct virtio_video_format_frame frames[] */
> > > > >         /* Followed by struct virtio_video_format_control controls[]
> > > > >         */
> > > > > 
> > > > > };
> > > > 
> > > > In my understanding, a set of supported levels depends on a profile.
> > > > H.264 spec says that levels are specified within each profile.
> > > > cf. "0.5 Profiles and levels" in
> > > > https://www.itu.int/rec/T-REC-H.264-201906-I/en (V4L2's QUERYMENU
> > > > doesn't
> > > > seem to provide a way to query levels for each profile properly,
> > > > though)
> > > > 
> > > > So, if we want to have supported profiles and levels in format_desc as
> > > > your idea, each supported profile should have a list of supported
> > > > levels.
> > > > I suppose it makes the structure of video_format_desc too complicated.
> > > > I'd like to avoid this complexity caused by some specific formats.
> > > > 
> > > > Instead, I'd like to keep virtio_video_format_desc minimal and have
> > > > QUERY_CONTROL to query format-specific values like profiles and
> > > > levels.
> > > > This design would make it easy to extend controls when we want to
> > > > support new types of formats. We will just need to define new
> > > > VIRTIO_VIDEO_CONTROL_*.
> > > > What do you think?
> > > 
> > > Yes, I think we can keep it this way, this indeed provides more
> > > flexibility and this way we won't need to modify the 'format' parsing
> > > logic when more controls are added.
> > 
> > Thanks! I'm preparing the next version of the patch.
> > 
> > I have one concern about terminology.
> > We have two different things called "controls" now:
> > 1) Messages passed via controlq. i.e. "ctrl" in 'enum
> > virtio_video_ctrl_type' and 'virtio_video_ctrl_hdr'.
> > 2) Commands for profiles, levels, and bitrates similar to V4L2
> > controls. e.g.., "CONTROL" in VIRTIO_VIDEO_CONTROL_*
> > 
> > I feel calling both "controls" is somewhat confusing and I am
> > wondering if we can't rename either of them.
> > 
> > For example, how about renaming 2) to "command"? Then, we'll have
> > {QUERY ,GET, SET}_COMMAND.
> > It may sound strange, though.
> > Do you have any idea?
> 
> "Command" sound like executing an action, but in this context we're
> thinking more about parameters or, exactly, controls. The term
> "control" is not limited to V4L2 and actually used in other subsystems
> as well, for example the Android Camera HAL.
> 
> How about renaming 1) to "commandq" and then having the other symbols
> use the abbreviated "cmd"?

First I thought about renaming 'virtio_video_ctrl_hdr' to 
'virtio_video_req_hdr', but then 'reqq' wouldn't look nice.

So I'd be ok to go with 'commandq' and 'cmd'.

T is for type, S is for status. I was looking at the virtio-iommu spec around 
that time, I think I derived it from there.

Regards,
Dmitry

> 
> Best regards,
> Tomasz
> 
> > Best regards,
> > Keiichi
> > 
> > > Best regards,
> > > Dmitry.
> > > 
> > > > Best regards,
> > > > Keiichi
> > > > 
> > > > > SET_CONTROL seems to be mandatory. If it succeeds, we can store the
> > > > > current
> > > > > value locally, so there is no need to have GET_CONTROL.
> > > > > 
> > > > > The only exception is the initial (default) control value in the
> > > > > device.
> > > > > But with the removal of 'function' and with addition of 'caps'
> > > > > instead,
> > > > > the way to provide defaults is gone. So I suppose for formats we'll
> > > > > be
> > > > > just using GET on driver start to get the 'defaults'. But the thing
> > > > > is
> > > > > that for formats there are other uses for GET, but for controls GET
> > > > > apparently does not make to much sense at runtime.
> > > > > 
> > > > > > > Also, shouldn't the profile come together with level? Would make
> > > > > > > sense
> > > > > > > for
> > > > > > > encoders.
> > > > > > 
> > > > > > Yeah. So, in the above idea of QUERY_CONTROL, profile should be
> > > > > > required when querying supported levels.
> > > > > 
> > > > > So probably should be enumerated together, not queried, as per the
> > > > > comment
> > > > > above.
> > > > > 
> > > > > > > > +\item[\field{mask}] is a bitset that represents the supported
> > > > > > > > +  combination of input and output format. If \textit{i}-th
> > > > > > > > bit is
> > > > > > > > set
> > > > > > > > +  in \field{mask} of \textit{j}-th \field{struct
> > > > > > > > +  virtio_video_format_desc} for input, the device supports
> > > > > > > > encoding
> > > > > > > > or
> > > > > > > > +  decoding from the \textit{j}-th input format to
> > > > > > > > \textit{i}-th
> > > > > > > > output
> > > > > > > > +  format.
> > > > > > > > +\item[\field{width, height}] represents a range of
> > > > > > > > resolutions
> > > > > > > > +  supported by the device. If its \field{step} is not
> > > > > > > > applicable,
> > > > > > > > its
> > > > > > > > +  \field{min} is equal to its \field{max}.
> > > > > > > > +\item[\field{num_rates}] is the length of an array
> > > > > > > > \field{frame_rates}.
> > > > > > > > In
> > > > > > > > case of decoder, the driver SHOULD ignore this value.
> > > > > > > > +\item[\field{frame_rates}] is an array of supported frame
> > > > > > > > rates.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > 
> > > > > > > I'd guess frame rates depend on the resolution as well. This
> > > > > > > dependency
> > > > > > > was
> > > > > > > clear in the v1 spec, but in the v2 there is no dependency
> > > > > > > anymore. I
> > > > > > > think we need to update this.
> > > > > > 
> > > > > > That's a good point. I missed that dependency when updating the
> > > > > > structures.
> > > > > > So, let me update the structs like the following:
> > > > > > 
> > > > > > struct virtio_video_format_frame {
> > > > > > 
> > > > > >         struct virtio_video_format_range width;
> > > > > >         struct virtio_video_format_range height;
> > > > > >         le32 num_rates;
> > > > > >         u8 padding[4];
> > > > > >         /* Followed by struct virtio_video_format_range
> > > > > >         frame_rates[] */
> > > > > > 
> > > > > > };
> > > > > > 
> > > > > > struct virtio_video_format_desc {
> > > > > > 
> > > > > >         le64 mask;
> > > > > >         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > > > >         le32 planes_layout; /* See the thread [v2 0/1] */
> > > > > >         le32 num_frames;
> > > > > >         u8 padding[4];
> > > > > >         /* Followed by struct virtio_video_format_frame frames[]
> > > > > >         */
> > > > > > 
> > > > > > };
> > > > > 
> > > > > Yes, I do agree with this approach.
> > > > > 
> > > > > Best regards,
> > > > > Dmitry.
> > > > > 
> > > > > > Best regards,
> > > > > > Keiichi.
> > > > > > 
> > > > > > > Best regards,
> > > > > > > Dmitry.
> > > > > > > 
> > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream
> > > > > > > > (context)
> > > > > > > > +  within the device.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +enum virtio_video_mem_type {
> > > > > > > > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_stream_create {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_*
> > > > > > > > types
> > > > > > > > */
> > > > > > > > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_*
> > > > > > > > types
> > > > > > > > */
> > > > > > > > +        char debug_name[64];
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{in_mem_type}] is a type of buffer management for
> > > > > > > > input
> > > > > > > > +buffers. The driver MUST set a value in \field{enum
> > > > > > > > +virtio_video_mem_type}.
> > > > > > > > +\item[\field{out_mem_type}] is a type of buffer management
> > > > > > > > for
> > > > > > > > output
> > > > > > > > +buffers. The driver MUST set a value in \field{enum
> > > > > > > > +virtio_video_mem_type}.
> > > > > > > > +\item[\field{debug_name}] is a text string for a debug
> > > > > > > > purpose.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream
> > > > > > > > (context)
> > > > > > > > +  within the device.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_stream_destroy {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all
> > > > > > > > the
> > > > > > > > +  queued buffers through the pipeline.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_stream_drain {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource
> > > > > > > > descriptor
> > > > > > > > +  within the device.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_mem_entry {
> > > > > > > > +        le64 addr;
> > > > > > > > +        le32 length;
> > > > > > > > +        u8 padding[4];
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_resource_create {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le32 resource_id;
> > > > > > > > +        le32 nr_entries;
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > > > +\item[\field{nr_entries}] number of \field{struct
> > > > > > > > +  virtio_video_mem_entry} memory entries.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource
> > > > > > > > descriptor
> > > > > > > > +  within the device.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_resource_destroy {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le32 resource_id;
> > > > > > > > +        u8 padding[4];
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the
> > > > > > > > device's
> > > > > > > > +queue.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > > > > > > > +
> > > > > > > > +struct virtio_video_resource_queue {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le32 buf_type;
> > > > > > > > +        le32 resource_id;
> > > > > > > > +        le64 timestamp;
> > > > > > > > +        le32 nr_data_size;
> > > > > > > > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{buf_type}] buf_type of the .
> > > > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > > > +\item[\field{timestamp}] an abstract sequence counter that
> > > > > > > > can be
> > > > > > > > used
> > > > > > > > +  for synchronisation.
> > > > > > > > +\item[\field{nr_data_size}] number of \field{data_size}
> > > > > > > > entries.
> > > > > > > > +\item[\field{data_size}] number of data bytes within a plane.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +enum virtio_video_buffer_flag {
> > > > > > > > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > > > > > > > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > > > > > > > +        /* Encoder only */
> > > > > > > > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > > > > > > > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > > > > > > > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_resource_queue_resp {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le64 timestamp;
> > > > > > > > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > > > > > > > +        le32 size;  /* Encoded size */
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{timestamp}] an abstract sequence counter that
> > > > > > > > can be
> > > > > > > > used
> > > > > > > > +  for synchronisation.
> > > > > > > > +\item[\field{flags}] mark specific buffers in the sequence.
> > > > > > > > +\item[\field{size}] data size in the buffer (encoder only).
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +The device sends a response to the queue request
> > > > > > > > asynchronously
> > > > > > > > when
> > > > > > > > +it has finished processing the buffer.
> > > > > > > > +
> > > > > > > > +The device SHOULD mark a buffer that triggered a processing
> > > > > > > > error
> > > > > > > > with
> > > > > > > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > > > > > > > +
> > > > > > > > +The device MUST mark the last buffer with the
> > > > > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the
> > > > > > > > drain
> > > > > > > > +sequence.
> > > > > > > > +
> > > > > > > > +In case of encoder, to denote a particular frame type the
> > > > > > > > devie
> > > > > > > > MUST
> > > > > > > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > > > > > > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already
> > > > > > > > queued
> > > > > > > > +  buffers back from the input or the output queue of the
> > > > > > > > device.
> > > > > > > > The
> > > > > > > > +  device SHOULD return all of the buffers from the respective
> > > > > > > > queue
> > > > > > > > as
> > > > > > > > +  soon as possible without pushing the buffers through the
> > > > > > > > processing
> > > > > > > > +  pipeline.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_queue_clear {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le32 buf_type;
> > > > > > > > +        u8 padding[4];
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{buf_type}] buffer type.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input
> > > > > > > > or the
> > > > > > > > +  output of a stream.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_plane_format {
> > > > > > > > +        le32 plane_size;
> > > > > > > > +        le32 stride;
> > > > > > > > +        u8 padding[4];
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{plane_size}] size of the plane in bytes.
> > > > > > > > +\item[\field{stride}] stride used for the plane in bytes.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_params {
> > > > > > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_*
> > > > > > > > types */
> > > > > > > > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types
> > > > > > > > */
> > > > > > > > +        le32 frame_width;
> > > > > > > > +        le32 frame_height;
> > > > > > > > +        le32 min_buffers;
> > > > > > > > +        le32 max_buffers;
> > > > > > > > +        le32 frame_rate;
> > > > > > > > +        struct virtio_video_crop {
> > > > > > > > +                le32 left;
> > > > > > > > +                le32 top;
> > > > > > > > +                le32 width;
> > > > > > > > +                le32 height;
> > > > > > > > +        } crop;
> > > > > > > > +        le32 num_planes;
> > > > > > > > +        struct virtio_video_plane_format
> > > > > > > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{frame_width}] the value to get/set.
> > > > > > > > +\item[\field{frame_height}] the value to get/set.
> > > > > > > > +\item[\field{pixel_format}] the value to get/set.
> > > > > > > > +\item[\field{min_buffers}] minimum buffers required to handle
> > > > > > > > the
> > > > > > > > +  format (r/o).
> > > > > > > > +\item[\field{max_buffers}] maximum buffers required to handle
> > > > > > > > the
> > > > > > > > +  format (r/o).
> > > > > > > > +\item[\field{frame_rate}] the value to get/set.
> > > > > > > > +\item[\field{crop}] cropping (composing) rectangle.
> > > > > > > > +\item[\field{num_planes}] number of planes used to store
> > > > > > > > pixel data
> > > > > > > > +(r/o).
> > > > > > > > +\item[\field{plane_formats}] description of each plane.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_get_params {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_*
> > > > > > > > types */
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_get_params_resp {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        struct virtio_video_params params;
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{buf_type}] buffer type.
> > > > > > > > +\item[\field{params}] parameter values.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a
> > > > > > > > stream.
> > > > > > > > +
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_set_params {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        struct virtio_video_params params;
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{params}] parameters to set.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +Setting stream parameters might have side effects within the
> > > > > > > > device.
> > > > > > > > +For example, the device MAY perform alignment of width and
> > > > > > > > height,
> > > > > > > > +change the number of planes it uses for the format, or do
> > > > > > > > whatever
> > > > > > > > +changes that are required to continue normal operation using
> > > > > > > > the
> > > > > > > > +updated parameters. It is up to the driver to check the
> > > > > > > > parameter
> > > > > > > > set
> > > > > > > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > > > > > > > +
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: eventq}
> > > > > > > > +
> > > > > > > > +The device can report events on the event queue. The driver
> > > > > > > > initially
> > > > > > > > +populates the queue with device-writeable buffers. When the
> > > > > > > > device
> > > > > > > > +needs to report an event, it fills a buffer and notifies the
> > > > > > > > driver.
> > > > > > > > +The driver consumes the report and adds a new buffer to the
> > > > > > > > virtqueue.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +enum virtio_video_event_type {
> > > > > > > > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > > > > > > > +        /* For all functions */
> > > > > > > > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > > > > > > > +        /* For decoder only */
> > > > > > > > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED =
> > > > > > > > 0x0200,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_event {
> > > > > > > > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_*
> > > > > > > > types */
> > > > > > > > +        le32 stream_id;
> > > > > > > > +        u8 padding[4];
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{event_type}] type of the triggered event .
> > > > > > > > +\item[\field{stream_id}] id of the source stream.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +The device MUST send
> > > > > > > > VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > > > > > > > +whenever it encounters new resolution data in the stream.
> > > > > > > > This
> > > > > > > > +includes the case of the initial device configuration after
> > > > > > > > metadata
> > > > > > > > +has been parsed and the case of dynamic resolution change.




^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [virtio-dev] Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2020-01-08 13:11                 ` Dmitry Sepp
@ 2020-01-08 13:23                   ` Keiichi Watanabe
  0 siblings, 0 replies; 73+ messages in thread
From: Keiichi Watanabe @ 2020-01-08 13:23 UTC (permalink / raw)
  To: Dmitry Sepp
  Cc: Tomasz Figa, uril, virtio-dev, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Enrico Granata, Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens

On Wed, Jan 8, 2020 at 10:11 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
>
> Hi Tomasz, Keiichi,
>
> On Mittwoch, 8. Januar 2020 13:46:25 CET Tomasz Figa wrote:
> > On Wed, Jan 8, 2020 at 9:15 PM Keiichi Watanabe <keiichiw@chromium.org>
> wrote:
> > > Hi Dmitry,
> > >
> > > On Wed, Jan 8, 2020 at 7:00 PM Dmitry Sepp <dmitry.sepp@opensynergy.com>
> wrote:
> > > > Hi Keiichi,
> > > >
> > > > On Mittwoch, 8. Januar 2020 07:59:22 CET Keiichi Watanabe wrote:
> > > > > Hi Dmitry,
> > > > >
> > > > > On Wed, Jan 8, 2020 at 1:50 AM Dmitry Sepp
> > > > > <dmitry.sepp@opensynergy.com>
> > > >
> > > > wrote:
> > > > > > Hi Keiichi,
> > > > > >
> > > > > > thanks for the updates, please see my comments below.
> > > > > >
> > > > > > On Dienstag, 7. Januar 2020 14:24:31 CET Keiichi Watanabe wrote:
> > > > > > > Hi Dmitry,
> > > > > > >
> > > > > > > On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp
> > > > > > > <dmitry.sepp@opensynergy.com>
> > > > > >
> > > > > > wrote:
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > a couple of new comments:
> > > > > > > >
> > > > > > > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe
> wrote:
> > > > > > > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > > > > > >
> > > > > > > > > The virtio video encoder device and decoder device provide
> > > > > > > > > functionalities
> > > > > > > > > to encode and decode video stream respectively.
> > > > > > > > > Though video encoder and decoder are provided as different
> > > > > > > > > devices,
> > > > > > > > > they
> > > > > > > > > use a same protocol.
> > > > > > > > >
> > > > > > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > > > > > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > > > > > > > > ---
> > > > > > > > >
> > > > > > > > >  content.tex      |   1 +
> > > > > > > > >  virtio-video.tex | 579
> > > > > > > > >  +++++++++++++++++++++++++++++++++++++++++++++++
> > > > > > > > >  2 files changed, 580 insertions(+)
> > > > > > > > >  create mode 100644 virtio-video.tex
> > > > > > > > >
> > > > > > > > > diff --git a/content.tex b/content.tex
> > > > > > > > > index 556b373..9e56839 100644
> > > > > > > > > --- a/content.tex
> > > > > > > > > +++ b/content.tex
> > > > > > > > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > > > > > > > > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> > > > > > > > >
> > > > > > > > >  \input{virtio-fs.tex}
> > > > > > > > >  \input{virtio-rpmb.tex}
> > > > > > > > >
> > > > > > > > > +\input{virtio-video.tex}
> > > > > > > > >
> > > > > > > > >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature
> > > > > > > > >  Bits}
> > > > > > > > >
> > > > > > > > > diff --git a/virtio-video.tex b/virtio-video.tex
> > > > > > > > > new file mode 100644
> > > > > > > > > index 0000000..30e728d
> > > > > > > > > --- /dev/null
> > > > > > > > > +++ b/virtio-video.tex
> > > > > > > > > @@ -0,0 +1,579 @@
> > > > > > > > > +\section{Video Device}\label{sec:Device Types / Video Device}
> > > > > > > > > +
> > > > > > > > > +The virtio video encoder device and decoder device are
> > > > > > > > > virtual
> > > > > > > > > devices
> > > > > > > > > that +supports encoding and decoding respectively. Though the
> > > > > > > > > encoder
> > > > > > > > > and the decoder +are different devices, they use the same
> > > > > > > > > protocol.
> > > > > > > > > +
> > > > > > > > > +\subsection{Device ID}\label{sec:Device Types / Video Device
> > > > > > > > > /
> > > > > > > > > Device
> > > > > > > > > ID}
> > > > > > > > > +
> > > > > > > > > +\begin{description}
> > > > > > > > > +\item[30] encoder device
> > > > > > > > > +\item[31] decoder device
> > > > > > > > > +\end{description}
> > > > > > > > > +
> > > > > > > > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device
> > > > > > > > > /
> > > > > > > > > Virtqueues} +
> > > > > > > > > +\begin{description}
> > > > > > > > > +\item[0] controlq - queue for sending control commands.
> > > > > > > > > +\item[1] eventq - queue for sending events happened in the
> > > > > > > > > device.
> > > > > > > > > +\end{description}
> > > > > > > > > +
> > > > > > > > > +\subsection{Feature bits}\label{sec:Device Types / Video
> > > > > > > > > Device /
> > > > > > > > > Feature
> > > > > > > > > bits} +
> > > > > > > > > +\begin{description}
> > > > > > > > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages
> > > > > > > > > can be
> > > > > > > > > used
> > > > > > > > > for
> > > > > > > > > video +  buffers.
> > > > > > > > > +\end{description}
> > > > > > > > > +
> > > > > > > > > +\devicenormative{\subsubsection}{Feature bits}{Device Types /
> > > > > > > > > Video
> > > > > > > > > Device
> > > > > > > > > / Feature bits} +
> > > > > > > > > +The device MUST offer at least one of feature bits.
> > > > > > > > > +
> > > > > > > > > +\subsection{Device configuration layout}\label{sec:Device
> > > > > > > > > Types /
> > > > > > > > > Video
> > > > > > > > > Device / Device configuration layout} +
> > > > > > > > > +Video device configuration uses the following layout
> > > > > > > > > structure:
> > > > > > > > > +
> > > > > > > > > +\begin{lstlisting}
> > > > > > > > > +struct virtio_video_config {
> > > > > > > > > +        le32 max_cap_len;
> > > > > > > > > +};
> > > > > > > > > +\end{lstlisting}
> > > > > > > > > +
> > > > > > > > > +\begin{description}
> > > > > > > > > +\item[\field{max_cap_len}] defines the maximum length of a
> > > > > > > > > descriptor
> > > > > > > > > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The
> > > > > > > > > device
> > > > > > > > > +  MUST set this value.
> > > > > > > > > +\end{description}
> > > > > > > > > +
> > > > > > > > > +\subsection{Device Initialization}\label{sec:Device Types /
> > > > > > > > > Video
> > > > > > > > > Device /
> > > > > > > > > Device Initialization} +
> > > > > > > > > +\devicenormative{\subsubsection}{Device
> > > > > > > > > Initialization}{Device
> > > > > > > > > Types /
> > > > > > > > > Video Device / Device Initialization} +
> > > > > > > > > +The driver SHOULD query device capability by using the
> > > > > > > > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for
> > > > > > > > > the
> > > > > > > > > initial
> > > > > > > > > +setup.
> > > > > > > > > +
> > > > > > > > > +\subsection{Device Operation}\label{sec:Device Types / Video
> > > > > > > > > Device
> > > > > > > > > /
> > > > > > > > > Device Operation} +
> > > > > > > > > +The driver allocates input and output buffers and queues the
> > > > > > > > > buffers
> > > > > > > > > +to the device. The device performs operations on the buffers
> > > > > > > > > according
> > > > > > > > > +to the function in question.
> > > > > > > > > +
> > > > > > > > > +\subsubsection{Device Operation: Create stream}
> > > > > > > > > +
> > > > > > > > > +To process buffers, the device needs to associate them with a
> > > > > > > > > certain
> > > > > > > > > +video stream (essentially, a context). Streams are created by
> > > > > > > > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > > > > > > > > +determined by the device.
> > > > > > > > > +
> > > > > > > > > +\subsubsection{Device Operation: Create buffers}
> > > > > > > > > +
> > > > > > > > > +Buffers are used to store the actual data as well as the
> > > > > > > > > relevant
> > > > > > > > > +metadata. Scatter lists are supported, so the buffer doesn't
> > > > > > > > > need
> > > > > > > > > to
> > > > > > > > > +be contiguous in guest physical memory.
> > > > > > > > > +
> > > > > > > > > +\begin{itemize*}
> > > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio
> > > > > > > > > video
> > > > > > > > > +  resource that is backed by a buffer allocated from the
> > > > > > > > > driver's
> > > > > > > > > +  memory.
> > > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a
> > > > > > > > > resource
> > > > > > > > > that
> > > > > > > > > +  is no longer needed.
> > > > > > > > > +\end{itemize*}
> > > > > > > > > +
> > > > > > > > > +\subsubsection{Device Operation: Stream parameter control}
> > > > > > > > > +
> > > > > > > > > +\begin{itemize*}
> > > > > > > > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream
> > > > > > > > > parameters
> > > > > > > > > for +  input and output streams from the device.
> > > > > > > > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream
> > > > > > > > > parameters to
> > > > > > > > > the +  device.
> > > > > > > > > +\item After setting stream parameters, the driver may issue
> > > > > > > > > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input
> > > > > > > > > and
> > > > > > > > > output
> > > > > > > > > can be +  changed implicitly by the device during the set
> > > > > > > > > operation.
> > > > > > > > > +\end{itemize*}
> > > > > > > > > +
> > > > > > > > > +\subsubsection{Device Operation: Process buffers}
> > > > > > > > > +
> > > > > > > > > +\begin{itemize*}
> > > > > > > > > +\item If the function and the buffer type require so, write
> > > > > > > > > data to
> > > > > > > > > +the buffer memory.
> > > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer
> > > > > > > > > for
> > > > > > > > > +processing in the device.
> > > > > > > > > +\item The request completes asynchronously when the device
> > > > > > > > > has
> > > > > > > > > +finished with the buffer.
> > > > > > > > > +\end{itemize*}
> > > > > > > > > +
> > > > > > > > > +\subsubsection{Device Operation: Buffer processing control}
> > > > > > > > > +
> > > > > > > > > +\begin{itemize*}
> > > > > > > > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to
> > > > > > > > > process
> > > > > > > > > and
> > > > > > > > > +  return all of the already queued buffers.
> > > > > > > > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to
> > > > > > > > > return
> > > > > > > > > back
> > > > > > > > > +  already queued buffers from the input or the output queue.
> > > > > > > > > This
> > > > > > > > > also
> > > > > > > > > +  includes input or output buffers that can be currently
> > > > > > > > > owned by
> > > > > > > > > the
> > > > > > > > > +  device's processing pipeline.
> > > > > > > > > +\end{itemize*}
> > > > > > > > > +
> > > > > > > > > +\subsubsection{Device Operation: Asynchronous events}
> > > > > > > > > +
> > > > > > > > > +While processing buffers, the device can send asynchronous
> > > > > > > > > event
> > > > > > > > > +notifications to the driver. The behaviour depends on the
> > > > > > > > > exact
> > > > > > > > > +stream. For example, the decoder device sends a resolution
> > > > > > > > > change
> > > > > > > > > +event when it encounters new resolution metadata in the
> > > > > > > > > stream.
> > > > > > > > > +
> > > > > > > > > +\subsubsection{Device Operation: Request header}
> > > > > > > > > +
> > > > > > > > > +All requests and responses on the control virt queue have a
> > > > > > > > > fixed
> > > > > > > > > +header using the following layout structure and definitions:
> > > > > > > > > +
> > > > > > > > > +\begin{lstlisting}
> > > > > > > > > +enum virtio_video_ctrl_type {
> > > > > > > > > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > > > > > > > > +
> > > > > > > > > +        /* request */
> > > > > > > > > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > > > > > > > > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > > > > > > > > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > > > > > > > > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > > > > > > > > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > > > > > > > > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > > > > > > > > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > > > > > > > > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > > > > > > > > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > > > > > > > > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > > > > > > > > +
> > > > > > > > > +        /* response */
> > > > > > > > > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > > > > > > > > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > > > > > > > > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > > > > > > > > +
> > > > > > > > > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > > > > > > > > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > > > > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > > > > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > > > > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > > > > > > > > +};
> > > > > > > > > +
> > > > > > > > > +struct virtio_video_ctrl_hdr {
> > > > > > > > > +        le32 type;
> > > > > > > > > +        le32 stream_id;
> > > > > > > > > +        le32 len; /* Length of the structure in bytes. */
> > > > > > > > > +};
> > > > > > > > > +\end{lstlisting}
> > > > > > > > > +
> > > > > > > > > +\begin{description}
> > > > > > > > > +\item[\field{type}] is the type of the driver request or the
> > > > > > > > > device
> > > > > > > > > +response.
> > > > > > > > > +\item[\field{stream_id}] specifies a target stream.
> > > > > > > > > +\item[\field{len}] is the length of data in bytes, which
> > > > > > > > > includes
> > > > > > > > > +length of the header.
> > > > > > > > > +\end{description}
> > > > > > > > > +
> > > > > > > > > +\subsubsection{Device Operation: controlq}
> > > > > > > > > +
> > > > > > > > > +\begin{description}
> > > > > > > > > +
> > > > > > > > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information
> > > > > > > > > about
> > > > > > > > > +supported formats.
> > > > > > > > > +
> > > > > > > > > +The driver uses \field{struct virtio_video_get_capability} to
> > > > > > > > > send
> > > > > > > > > a
> > > > > > > > > +query request.
> > > > > > > > > +
> > > > > > > > > +\begin{lstlisting}
> > > > > > > > > +enum virtio_video_buf_type {
> > > > > > > > > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > > > > > > > > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > > > > > > > > +};
> > > > > > > > > +
> > > > > > > > > +struct virtio_video_get_capability {
> > > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > > +        enum virtio_video_buf_type buf_type;
> > > > > > > > > +};
> > > > > > > > > +\end{lstlisting}
> > > > > > > > > +\begin{description}
> > > > > > > > > +\item[\field{buf_type}] is the buffer type that the driver
> > > > > > > > > asks
> > > > > > > > > +information about. The driver MUST set either
> > > > > > > > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > > > > > > > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > > > > > > > > +
> > > > > > > > > +The device responds a capability by using \field{struct
> > > > > > > > > +virtio_video_get_capability_resp}.
> > > > > > > > > +\begin{lstlisting}
> > > > > > > > > +enum virtio_video_format {
> > > > > > > > > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > > > > > > > > +        /* Raw formats */
> > > > > > > > > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > > > > > > > > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > > > > > > > > +        VIRTIO_VIDEO_FORMAT_YVU420,
> > > > > > > > > +
> > > > > > > > > +        /* Compressed formats */
> > > > > > > > > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > > > > > > > > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > > > > > > > > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> > > > > > > > > +};
> > > > > > > > > +
> > > > > > > > > +enum virtio_video_profile {
> > > > > > > > > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > > > > > > > > +
> > > > > > > > > +        /* H.264 */
> > > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > > > > > > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > > > > > > > > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > > > > > > > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > > > > > > > > +        /* VP8 */
> > > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_ANY =
> > > > > > > > > VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_MAX =
> > > > > > > > > VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > > > > > > > > +
> > > > > > > > > +        /* VP9 */
> > > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 =
> > > > > > > > > VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_MAX =
> > > > > > > > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > > > > > > +};
> > > > > > > > > +
> > > > > > > > > +struct virtio_video_format_range {
> > > > > > > > > +        le32 min;
> > > > > > > > > +        le32 max;
> > > > > > > > > +        le32 step;
> > > > > > > > > +        u8 paddings[4];
> > > > > > > > > +};
> > > > > > > > > +
> > > > > > > > > +struct virtio_video_format_desc {
> > > > > > > > > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types
> > > > > > > > > */
> > > > > > > > > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types
> > > > > > > > > */
> > > > > > > > > +        le64 mask;
> > > > > > > > > +        struct virtio_video_format_range width;
> > > > > > > > > +        struct virtio_video_format_range height;
> > > > > > > > > +        le32 num_rates;
> > > > > > > > > +        u8 padding[4];
> > > > > > > > > +        /* Followed by struct virtio_video_frame_rate
> > > > > > > > > frame_rates[]
> > > > > > > > > */
> > > > > > > > > +};
> > > > > > > > > +
> > > > > > > > > +struct virtio_video_get_capability_resp {
> > > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > > +        le32 num_descs;
> > > > > > > > > +        /* Followed by struct virtio_video_format_desc desc[]
> > > > > > > > > */
> > > > > > > > > +};
> > > > > > > > > +\end{lstlisting}
> > > > > > > > > +
> > > > > > > > > +The format description \field{struct
> > > > > > > > > virtio_video_format_desc}
> > > > > > > > > +includes the following fields:
> > > > > > > > > +\begin{description}
> > > > > > > > > +\item[\field{format}] specifies an image format. The device
> > > > > > > > > MUST
> > > > > > > > > set
> > > > > > > > > one
> > > > > > > > > +  of \field{enum virtio_video_format}.
> > > > > > > > > +\item[\field{profile}] specifies a profile of the compressed
> > > > > > > > > image
> > > > > > > > > format
> > > > > > > > > +  specified in \field{format}. The driver SHOULD ignore this
> > > > > > > > > value
> > > > > > > > > if
> > > > > > > > > +  \field{format} is a raw format.
> > > > > > > >
> > > > > > > > So how should this be used? The spec does not define any way to
> > > > > > > > set
> > > > > > > > profile for the device. It is very important for encoder.
> > > > > > >
> > > > > > > Thank you for pointing this.
> > > > > > > These points are overlooked, as I didn't care about encoder usage
> > > > > > > enough.
> > > > > > >
> > > > > > > After thinking it again, I think it's not a very good idea to
> > > > > > > include
> > > > > > > supported profiles and levels in a struct for capability.
> > > > > > > This is because these values are available only for limited number
> > > > > > > of
> > > > > > > formats. Also, it's true that we need to have a way to set these
> > > > > > > values
> > > > > > > as
> > > > > > > Dmitry pointed.
> > > > > >
> > > > > > Yes, you are right. In fact, the approach of the v1 spec to keep
> > > > > > controls
> > > > > > separately was not correct.
> > > > > >
> > > > > > > Instead, it would make more sense to have additional three types
> > > > > > > of
> > > > > > > controls for profiles, levels, and bitrates:
> > > > > > > (1) QUERY_CONTROL: Query values supported by the device
> > > > > > > (2) GET_CONTROL: Read a value that is set in the device
> > > > > > > (3) SET_CONTROL: Set a value in the device
> > > > > > >
> > > > > > > These operations are similar to V4L2 controls.
> > > > > > > (1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL,
> > > > > > > and
> > > > > > > G_CTRL in V4L2, respectively.
> > > > > > > Also, (3) would be similar to enum virtio_video_control_type in
> > > > > > > the
> > > > > > > virtio-video v1 driver implementation in
> > > > > > > https://markmail.org/message/dwghwdqsbl3gsjxu .
> > > > > > >
> > > > > > > For QUERY_CONTROL, my idea is like this:
> > > > > > >
> > > > > > > enum virtio_video_control_type {
> > > > > > >
> > > > > > >   VIRTIO_VIDEO_CONTROL_UNDEFINED = 0,
> > > > > > >
> > > > > > >   VIRTIO_VIDEO_CONTROL_BITRATE = 0x100,
> > > > > > >   VIRTIO_VIDEO_CONTROL_PROFILE,
> > > > > > >   VIRTIO_VIDEO_CONTROL_LEVEL,
> > > > > > >
> > > > > > > };
> > > > > > >
> > > > > > > struct virtio_video_query_control {
> > > > > > >
> > > > > > >   struct virtio_video_ctrl_hdr hdr;
> > > > > > >   le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
> > > > > > >   le32 length;
> > > > > > >   /* Followed by additional data.
> > > > > > >
> > > > > > >    * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE,
> > > > > > >    * the device must pass a codec format like H264 or VP9.
> > > > > > >    * The requred data must be defined in the specification.
> > > > > > >    */
> > > > > > >
> > > > > > > };
> > > > > > >
> > > > > > > struct virtio_video_query_control_resp {
> > > > > > >
> > > > > > >   struct virtio_video_ctrl_hdr hdr;
> > > > > > >   le32 length;
> > > > > > >   u8 padding[4];
> > > > > > >   /* Followed by data corresponds to the specified control.
> > > > > > >
> > > > > > >    * The type of data must be defined in the spec.
> > > > > > >    * For example, if the driver queries profiles, this part should
> > > > > > >    be
> > > > > > >    * an array of supported profiles of a given format.
> > > > > > >    */
> > > > > > >
> > > > > > > };
> > > > > > >
> > > > > > > WDYT?
> > > > > >
> > > > > > I think virtio_video_control_type should make sense. But I would
> > > > > > disagree
> > > > > > with the need to have new QUERY_CONTROL and GET_CONTROL.
> > > > > >
> > > > > > I assume the set of supported controls is fixed for some particular
> > > > > > format
> > > > > > on a given IP. So we'd propose to include controls into format
> > > > > > descriptors, so we don't need to QUERY_CONTROL. This way (with
> > > > > > 'virtio_video_format_list') we can define not-contiguos ranges, e.g.
> > > > > > for
> > > > > > profiles.
> > > > > >
> > > > > > struct virtio_video_format_frame {
> > > > > >
> > > > > >         /* As proposed in Keiichi's prev email */
> > > > > >
> > > > > > };
> > > > > >
> > > > > > virtio_video_format_list {
> > > > > >
> > > > > >         le32 num_entries;
> > > > > >         u8 padding[4];
> > > > > >         /* Followed by le64 entries[] */
> > > > > >
> > > > > > };
> > > > > >
> > > > > > struct virtio_video_format_control {
> > > > > >
> > > > > >         le32 type;
> > > > > >         u8 padding[4];
> > > > > >         struct virtio_video_format_list values;
> > > > > >
> > > > > > };
> > > > > >
> > > > > > struct virtio_video_format_desc {
> > > > > >
> > > > > >         le64 mask;
> > > > > >         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > > > >         le32 planes_layout; /* See the thread [v2 0/1] */
> > > > > >         le32 num_frames;
> > > > > >         le32 num_controls;
> > > > > >         /* Followed by struct virtio_video_format_frame frames[] */
> > > > > >         /* Followed by struct virtio_video_format_control controls[]
> > > > > >         */
> > > > > >
> > > > > > };
> > > > >
> > > > > In my understanding, a set of supported levels depends on a profile.
> > > > > H.264 spec says that levels are specified within each profile.
> > > > > cf. "0.5 Profiles and levels" in
> > > > > https://www.itu.int/rec/T-REC-H.264-201906-I/en (V4L2's QUERYMENU
> > > > > doesn't
> > > > > seem to provide a way to query levels for each profile properly,
> > > > > though)
> > > > >
> > > > > So, if we want to have supported profiles and levels in format_desc as
> > > > > your idea, each supported profile should have a list of supported
> > > > > levels.
> > > > > I suppose it makes the structure of video_format_desc too complicated.
> > > > > I'd like to avoid this complexity caused by some specific formats.
> > > > >
> > > > > Instead, I'd like to keep virtio_video_format_desc minimal and have
> > > > > QUERY_CONTROL to query format-specific values like profiles and
> > > > > levels.
> > > > > This design would make it easy to extend controls when we want to
> > > > > support new types of formats. We will just need to define new
> > > > > VIRTIO_VIDEO_CONTROL_*.
> > > > > What do you think?
> > > >
> > > > Yes, I think we can keep it this way, this indeed provides more
> > > > flexibility and this way we won't need to modify the 'format' parsing
> > > > logic when more controls are added.
> > >
> > > Thanks! I'm preparing the next version of the patch.
> > >
> > > I have one concern about terminology.
> > > We have two different things called "controls" now:
> > > 1) Messages passed via controlq. i.e. "ctrl" in 'enum
> > > virtio_video_ctrl_type' and 'virtio_video_ctrl_hdr'.
> > > 2) Commands for profiles, levels, and bitrates similar to V4L2
> > > controls. e.g.., "CONTROL" in VIRTIO_VIDEO_CONTROL_*
> > >
> > > I feel calling both "controls" is somewhat confusing and I am
> > > wondering if we can't rename either of them.
> > >
> > > For example, how about renaming 2) to "command"? Then, we'll have
> > > {QUERY ,GET, SET}_COMMAND.
> > > It may sound strange, though.
> > > Do you have any idea?
> >
> > "Command" sound like executing an action, but in this context we're
> > thinking more about parameters or, exactly, controls. The term
> > "control" is not limited to V4L2 and actually used in other subsystems
> > as well, for example the Android Camera HAL.
> >
> > How about renaming 1) to "commandq" and then having the other symbols
> > use the abbreviated "cmd"?
>
> First I thought about renaming 'virtio_video_ctrl_hdr' to
> 'virtio_video_req_hdr', but then 'reqq' wouldn't look nice.
>
> So I'd be ok to go with 'commandq' and 'cmd'.
>
> T is for type, S is for status. I was looking at the virtio-iommu spec around
> that time, I think I derived it from there.

Thank you for the explanation.
Since virtio-video's responses can contain information more than
status (e.g. capability), let's go with "CMD" and "RESP" instead of
"T" and "S" like virtio_gpu_ctrl_type.

Best regards,
Keiichi

>
> Regards,
> Dmitry
>
> >
> > Best regards,
> > Tomasz
> >
> > > Best regards,
> > > Keiichi
> > >
> > > > Best regards,
> > > > Dmitry.
> > > >
> > > > > Best regards,
> > > > > Keiichi
> > > > >
> > > > > > SET_CONTROL seems to be mandatory. If it succeeds, we can store the
> > > > > > current
> > > > > > value locally, so there is no need to have GET_CONTROL.
> > > > > >
> > > > > > The only exception is the initial (default) control value in the
> > > > > > device.
> > > > > > But with the removal of 'function' and with addition of 'caps'
> > > > > > instead,
> > > > > > the way to provide defaults is gone. So I suppose for formats we'll
> > > > > > be
> > > > > > just using GET on driver start to get the 'defaults'. But the thing
> > > > > > is
> > > > > > that for formats there are other uses for GET, but for controls GET
> > > > > > apparently does not make to much sense at runtime.
> > > > > >
> > > > > > > > Also, shouldn't the profile come together with level? Would make
> > > > > > > > sense
> > > > > > > > for
> > > > > > > > encoders.
> > > > > > >
> > > > > > > Yeah. So, in the above idea of QUERY_CONTROL, profile should be
> > > > > > > required when querying supported levels.
> > > > > >
> > > > > > So probably should be enumerated together, not queried, as per the
> > > > > > comment
> > > > > > above.
> > > > > >
> > > > > > > > > +\item[\field{mask}] is a bitset that represents the supported
> > > > > > > > > +  combination of input and output format. If \textit{i}-th
> > > > > > > > > bit is
> > > > > > > > > set
> > > > > > > > > +  in \field{mask} of \textit{j}-th \field{struct
> > > > > > > > > +  virtio_video_format_desc} for input, the device supports
> > > > > > > > > encoding
> > > > > > > > > or
> > > > > > > > > +  decoding from the \textit{j}-th input format to
> > > > > > > > > \textit{i}-th
> > > > > > > > > output
> > > > > > > > > +  format.
> > > > > > > > > +\item[\field{width, height}] represents a range of
> > > > > > > > > resolutions
> > > > > > > > > +  supported by the device. If its \field{step} is not
> > > > > > > > > applicable,
> > > > > > > > > its
> > > > > > > > > +  \field{min} is equal to its \field{max}.
> > > > > > > > > +\item[\field{num_rates}] is the length of an array
> > > > > > > > > \field{frame_rates}.
> > > > > > > > > In
> > > > > > > > > case of decoder, the driver SHOULD ignore this value.
> > > > > > > > > +\item[\field{frame_rates}] is an array of supported frame
> > > > > > > > > rates.
> > > > > > > > > +\end{description}
> > > > > > > > > +
> > > > > > > >
> > > > > > > > I'd guess frame rates depend on the resolution as well. This
> > > > > > > > dependency
> > > > > > > > was
> > > > > > > > clear in the v1 spec, but in the v2 there is no dependency
> > > > > > > > anymore. I
> > > > > > > > think we need to update this.
> > > > > > >
> > > > > > > That's a good point. I missed that dependency when updating the
> > > > > > > structures.
> > > > > > > So, let me update the structs like the following:
> > > > > > >
> > > > > > > struct virtio_video_format_frame {
> > > > > > >
> > > > > > >         struct virtio_video_format_range width;
> > > > > > >         struct virtio_video_format_range height;
> > > > > > >         le32 num_rates;
> > > > > > >         u8 padding[4];
> > > > > > >         /* Followed by struct virtio_video_format_range
> > > > > > >         frame_rates[] */
> > > > > > >
> > > > > > > };
> > > > > > >
> > > > > > > struct virtio_video_format_desc {
> > > > > > >
> > > > > > >         le64 mask;
> > > > > > >         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > > > > >         le32 planes_layout; /* See the thread [v2 0/1] */
> > > > > > >         le32 num_frames;
> > > > > > >         u8 padding[4];
> > > > > > >         /* Followed by struct virtio_video_format_frame frames[]
> > > > > > >         */
> > > > > > >
> > > > > > > };
> > > > > >
> > > > > > Yes, I do agree with this approach.
> > > > > >
> > > > > > Best regards,
> > > > > > Dmitry.
> > > > > >
> > > > > > > Best regards,
> > > > > > > Keiichi.
> > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Dmitry.
> > > > > > > >
> > > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream
> > > > > > > > > (context)
> > > > > > > > > +  within the device.
> > > > > > > > > +
> > > > > > > > > +\begin{lstlisting}
> > > > > > > > > +enum virtio_video_mem_type {
> > > > > > > > > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > > > > > > > > +};
> > > > > > > > > +
> > > > > > > > > +struct virtio_video_stream_create {
> > > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_*
> > > > > > > > > types
> > > > > > > > > */
> > > > > > > > > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_*
> > > > > > > > > types
> > > > > > > > > */
> > > > > > > > > +        char debug_name[64];
> > > > > > > > > +};
> > > > > > > > > +\end{lstlisting}
> > > > > > > > > +
> > > > > > > > > +\begin{description}
> > > > > > > > > +\item[\field{in_mem_type}] is a type of buffer management for
> > > > > > > > > input
> > > > > > > > > +buffers. The driver MUST set a value in \field{enum
> > > > > > > > > +virtio_video_mem_type}.
> > > > > > > > > +\item[\field{out_mem_type}] is a type of buffer management
> > > > > > > > > for
> > > > > > > > > output
> > > > > > > > > +buffers. The driver MUST set a value in \field{enum
> > > > > > > > > +virtio_video_mem_type}.
> > > > > > > > > +\item[\field{debug_name}] is a text string for a debug
> > > > > > > > > purpose.
> > > > > > > > > +\end{description}
> > > > > > > > > +
> > > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream
> > > > > > > > > (context)
> > > > > > > > > +  within the device.
> > > > > > > > > +
> > > > > > > > > +\begin{lstlisting}
> > > > > > > > > +struct virtio_video_stream_destroy {
> > > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > > +};
> > > > > > > > > +\end{lstlisting}
> > > > > > > > > +
> > > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all
> > > > > > > > > the
> > > > > > > > > +  queued buffers through the pipeline.
> > > > > > > > > +
> > > > > > > > > +\begin{lstlisting}
> > > > > > > > > +struct virtio_video_stream_drain {
> > > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > > +};
> > > > > > > > > +\end{lstlisting}
> > > > > > > > > +
> > > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource
> > > > > > > > > descriptor
> > > > > > > > > +  within the device.
> > > > > > > > > +
> > > > > > > > > +\begin{lstlisting}
> > > > > > > > > +struct virtio_video_mem_entry {
> > > > > > > > > +        le64 addr;
> > > > > > > > > +        le32 length;
> > > > > > > > > +        u8 padding[4];
> > > > > > > > > +};
> > > > > > > > > +
> > > > > > > > > +struct virtio_video_resource_create {
> > > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > > +        le32 resource_id;
> > > > > > > > > +        le32 nr_entries;
> > > > > > > > > +};
> > > > > > > > > +\end{lstlisting}
> > > > > > > > > +
> > > > > > > > > +\begin{description}
> > > > > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > > > > +\item[\field{nr_entries}] number of \field{struct
> > > > > > > > > +  virtio_video_mem_entry} memory entries.
> > > > > > > > > +\end{description}
> > > > > > > > > +
> > > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource
> > > > > > > > > descriptor
> > > > > > > > > +  within the device.
> > > > > > > > > +
> > > > > > > > > +\begin{lstlisting}
> > > > > > > > > +struct virtio_video_resource_destroy {
> > > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > > +        le32 resource_id;
> > > > > > > > > +        u8 padding[4];
> > > > > > > > > +};
> > > > > > > > > +\end{lstlisting}
> > > > > > > > > +
> > > > > > > > > +\begin{description}
> > > > > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > > > > +\end{description}
> > > > > > > > > +
> > > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the
> > > > > > > > > device's
> > > > > > > > > +queue.
> > > > > > > > > +
> > > > > > > > > +\begin{lstlisting}
> > > > > > > > > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > > > > > > > > +
> > > > > > > > > +struct virtio_video_resource_queue {
> > > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > > +        le32 buf_type;
> > > > > > > > > +        le32 resource_id;
> > > > > > > > > +        le64 timestamp;
> > > > > > > > > +        le32 nr_data_size;
> > > > > > > > > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > > > > > > > > +};
> > > > > > > > > +\end{lstlisting}
> > > > > > > > > +
> > > > > > > > > +\begin{description}
> > > > > > > > > +\item[\field{buf_type}] buf_type of the .
> > > > > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > > > > +\item[\field{timestamp}] an abstract sequence counter that
> > > > > > > > > can be
> > > > > > > > > used
> > > > > > > > > +  for synchronisation.
> > > > > > > > > +\item[\field{nr_data_size}] number of \field{data_size}
> > > > > > > > > entries.
> > > > > > > > > +\item[\field{data_size}] number of data bytes within a plane.
> > > > > > > > > +\end{description}
> > > > > > > > > +
> > > > > > > > > +\begin{lstlisting}
> > > > > > > > > +enum virtio_video_buffer_flag {
> > > > > > > > > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > > > > > > > > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > > > > > > > > +        /* Encoder only */
> > > > > > > > > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > > > > > > > > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > > > > > > > > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > > > > > > > > +};
> > > > > > > > > +
> > > > > > > > > +struct virtio_video_resource_queue_resp {
> > > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > > +        le64 timestamp;
> > > > > > > > > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > > > > > > > > +        le32 size;  /* Encoded size */
> > > > > > > > > +};
> > > > > > > > > +\end{lstlisting}
> > > > > > > > > +
> > > > > > > > > +\begin{description}
> > > > > > > > > +\item[\field{timestamp}] an abstract sequence counter that
> > > > > > > > > can be
> > > > > > > > > used
> > > > > > > > > +  for synchronisation.
> > > > > > > > > +\item[\field{flags}] mark specific buffers in the sequence.
> > > > > > > > > +\item[\field{size}] data size in the buffer (encoder only).
> > > > > > > > > +\end{description}
> > > > > > > > > +
> > > > > > > > > +The device sends a response to the queue request
> > > > > > > > > asynchronously
> > > > > > > > > when
> > > > > > > > > +it has finished processing the buffer.
> > > > > > > > > +
> > > > > > > > > +The device SHOULD mark a buffer that triggered a processing
> > > > > > > > > error
> > > > > > > > > with
> > > > > > > > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > > > > > > > > +
> > > > > > > > > +The device MUST mark the last buffer with the
> > > > > > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the
> > > > > > > > > drain
> > > > > > > > > +sequence.
> > > > > > > > > +
> > > > > > > > > +In case of encoder, to denote a particular frame type the
> > > > > > > > > devie
> > > > > > > > > MUST
> > > > > > > > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > > > > > > > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > > > > > > > > +
> > > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already
> > > > > > > > > queued
> > > > > > > > > +  buffers back from the input or the output queue of the
> > > > > > > > > device.
> > > > > > > > > The
> > > > > > > > > +  device SHOULD return all of the buffers from the respective
> > > > > > > > > queue
> > > > > > > > > as
> > > > > > > > > +  soon as possible without pushing the buffers through the
> > > > > > > > > processing
> > > > > > > > > +  pipeline.
> > > > > > > > > +
> > > > > > > > > +\begin{lstlisting}
> > > > > > > > > +struct virtio_video_queue_clear {
> > > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > > +        le32 buf_type;
> > > > > > > > > +        u8 padding[4];
> > > > > > > > > +};
> > > > > > > > > +\end{lstlisting}
> > > > > > > > > +
> > > > > > > > > +\begin{description}
> > > > > > > > > +\item[\field{buf_type}] buffer type.
> > > > > > > > > +\end{description}
> > > > > > > > > +
> > > > > > > > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input
> > > > > > > > > or the
> > > > > > > > > +  output of a stream.
> > > > > > > > > +
> > > > > > > > > +\begin{lstlisting}
> > > > > > > > > +struct virtio_video_plane_format {
> > > > > > > > > +        le32 plane_size;
> > > > > > > > > +        le32 stride;
> > > > > > > > > +        u8 padding[4];
> > > > > > > > > +};
> > > > > > > > > +\end{lstlisting}
> > > > > > > > > +
> > > > > > > > > +\begin{description}
> > > > > > > > > +\item[\field{plane_size}] size of the plane in bytes.
> > > > > > > > > +\item[\field{stride}] stride used for the plane in bytes.
> > > > > > > > > +\end{description}
> > > > > > > > > +
> > > > > > > > > +\begin{lstlisting}
> > > > > > > > > +struct virtio_video_params {
> > > > > > > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_*
> > > > > > > > > types */
> > > > > > > > > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types
> > > > > > > > > */
> > > > > > > > > +        le32 frame_width;
> > > > > > > > > +        le32 frame_height;
> > > > > > > > > +        le32 min_buffers;
> > > > > > > > > +        le32 max_buffers;
> > > > > > > > > +        le32 frame_rate;
> > > > > > > > > +        struct virtio_video_crop {
> > > > > > > > > +                le32 left;
> > > > > > > > > +                le32 top;
> > > > > > > > > +                le32 width;
> > > > > > > > > +                le32 height;
> > > > > > > > > +        } crop;
> > > > > > > > > +        le32 num_planes;
> > > > > > > > > +        struct virtio_video_plane_format
> > > > > > > > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > > > > > > > > +\end{lstlisting}
> > > > > > > > > +
> > > > > > > > > +\begin{description}
> > > > > > > > > +\item[\field{frame_width}] the value to get/set.
> > > > > > > > > +\item[\field{frame_height}] the value to get/set.
> > > > > > > > > +\item[\field{pixel_format}] the value to get/set.
> > > > > > > > > +\item[\field{min_buffers}] minimum buffers required to handle
> > > > > > > > > the
> > > > > > > > > +  format (r/o).
> > > > > > > > > +\item[\field{max_buffers}] maximum buffers required to handle
> > > > > > > > > the
> > > > > > > > > +  format (r/o).
> > > > > > > > > +\item[\field{frame_rate}] the value to get/set.
> > > > > > > > > +\item[\field{crop}] cropping (composing) rectangle.
> > > > > > > > > +\item[\field{num_planes}] number of planes used to store
> > > > > > > > > pixel data
> > > > > > > > > +(r/o).
> > > > > > > > > +\item[\field{plane_formats}] description of each plane.
> > > > > > > > > +\end{description}
> > > > > > > > > +
> > > > > > > > > +\begin{lstlisting}
> > > > > > > > > +struct virtio_video_get_params {
> > > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_*
> > > > > > > > > types */
> > > > > > > > > +};
> > > > > > > > > +
> > > > > > > > > +struct virtio_video_get_params_resp {
> > > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > > +        struct virtio_video_params params;
> > > > > > > > > +};
> > > > > > > > > +\end{lstlisting}
> > > > > > > > > +
> > > > > > > > > +\begin{description}
> > > > > > > > > +\item[\field{buf_type}] buffer type.
> > > > > > > > > +\item[\field{params}] parameter values.
> > > > > > > > > +\end{description}
> > > > > > > > > +
> > > > > > > > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a
> > > > > > > > > stream.
> > > > > > > > > +
> > > > > > > > > +
> > > > > > > > > +\begin{lstlisting}
> > > > > > > > > +struct virtio_video_set_params {
> > > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > > +        struct virtio_video_params params;
> > > > > > > > > +};
> > > > > > > > > +\end{lstlisting}
> > > > > > > > > +
> > > > > > > > > +\begin{description}
> > > > > > > > > +\item[\field{params}] parameters to set.
> > > > > > > > > +\end{description}
> > > > > > > > > +
> > > > > > > > > +Setting stream parameters might have side effects within the
> > > > > > > > > device.
> > > > > > > > > +For example, the device MAY perform alignment of width and
> > > > > > > > > height,
> > > > > > > > > +change the number of planes it uses for the format, or do
> > > > > > > > > whatever
> > > > > > > > > +changes that are required to continue normal operation using
> > > > > > > > > the
> > > > > > > > > +updated parameters. It is up to the driver to check the
> > > > > > > > > parameter
> > > > > > > > > set
> > > > > > > > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > > > > > > > > +
> > > > > > > > > +\end{description}
> > > > > > > > > +
> > > > > > > > > +\subsubsection{Device Operation: eventq}
> > > > > > > > > +
> > > > > > > > > +The device can report events on the event queue. The driver
> > > > > > > > > initially
> > > > > > > > > +populates the queue with device-writeable buffers. When the
> > > > > > > > > device
> > > > > > > > > +needs to report an event, it fills a buffer and notifies the
> > > > > > > > > driver.
> > > > > > > > > +The driver consumes the report and adds a new buffer to the
> > > > > > > > > virtqueue.
> > > > > > > > > +
> > > > > > > > > +\begin{lstlisting}
> > > > > > > > > +enum virtio_video_event_type {
> > > > > > > > > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > > > > > > > > +        /* For all functions */
> > > > > > > > > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > > > > > > > > +        /* For decoder only */
> > > > > > > > > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED =
> > > > > > > > > 0x0200,
> > > > > > > > > +};
> > > > > > > > > +
> > > > > > > > > +struct virtio_video_event {
> > > > > > > > > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_*
> > > > > > > > > types */
> > > > > > > > > +        le32 stream_id;
> > > > > > > > > +        u8 padding[4];
> > > > > > > > > +};
> > > > > > > > > +\end{lstlisting}
> > > > > > > > > +
> > > > > > > > > +\begin{description}
> > > > > > > > > +\item[\field{event_type}] type of the triggered event .
> > > > > > > > > +\item[\field{stream_id}] id of the source stream.
> > > > > > > > > +\end{description}
> > > > > > > > > +
> > > > > > > > > +The device MUST send
> > > > > > > > > VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > > > > > > > > +whenever it encounters new resolution data in the stream.
> > > > > > > > > This
> > > > > > > > > +includes the case of the initial device configuration after
> > > > > > > > > metadata
> > > > > > > > > +has been parsed and the case of dynamic resolution change.
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [virtio-dev] Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-19 13:01           ` Gerd Hoffmann
@ 2020-01-08 13:50             ` Keiichi Watanabe
  0 siblings, 0 replies; 73+ messages in thread
From: Keiichi Watanabe @ 2020-01-08 13:50 UTC (permalink / raw)
  To: Gerd Hoffmann
  Cc: Dmitry Sepp, Tomasz Figa, virtio-dev, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Enrico Granata, Frediano Ziglio, Hans Verkuil,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

Hi Gerd,
Thank you so much for the review. I'm sorry for not replying earlier.

On Thu, Dec 19, 2019 at 10:02 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
>
>   Hi,
>
> > > Not clearly defined in the spec:  When is the decoder supposed to send
> > > the response for a queue request?  When it finished decoding (i.e. frame
> > > is ready for playback), or when it doesn't need the buffer any more for
> > > decoding (i.e. buffer can be re-queued or pages can be released)?

The answer is "when it doesn't need the buffer any more for decoding".
The device can access buffer contents from when a queue request is
sent until the device responds it. So, the device must not responds a
queue request before finishing all process that requires the buffer
content.

Actually, the first one "When it finished decoding (i.e. frame is
ready for playback)" doesn't make much sense, as it's not necessary to
have a one-to-one correspondence between an input bitstream buffer and
a decoded frame.
It's okay to decode one input buffer contains bitstream data for two
frames. Also, a user can pass bitstream for one frame as two input
buffers.

I'll document it in the spec.

Best regards,
Keiichi

> > In my eyes the both statements mean almost the same and both are valid.
>
> Well, no.  When the device decoded a P-Frame it can notify the device,
> saying "here is your decoded frame".  But the device might still need
> the buffer with the decoded frame to properly decode the following B/I
> Frames which reference the P-Frame.
>
> cheers,
>   Gerd
>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [virtio-dev] Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2019-12-19 13:12               ` Gerd Hoffmann
@ 2020-01-08 13:52                 ` Keiichi Watanabe
  2020-01-09 13:40                   ` Gerd Hoffmann
  2020-01-09 14:20                   ` Tomasz Figa
  0 siblings, 2 replies; 73+ messages in thread
From: Keiichi Watanabe @ 2020-01-08 13:52 UTC (permalink / raw)
  To: Gerd Hoffmann
  Cc: Dmitry Sepp, Tomasz Figa, virtio-dev, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Enrico Granata, Frediano Ziglio, Hans Verkuil,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

Hi Gerd,

On Thu, Dec 19, 2019 at 10:12 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
>
>   Hi,
>
> > > However that still doesn't let the driver know which buffers will be
> > > dequeued when. A simple example of this scenario is when the guest is
> > > done displaying a frame and requeues the buffer back to the decoder.
> > > Then the decoder will not choose it for decoding next frames into as
> > > long as the frame in that buffer is still used as a reference frame,
> > > even if one sends the drain request.
> > It might be that I'm getting your point wrong, but do you mean some hardware
> > can mark a buffer as ready to be displayed yet still using the underlying
> > memory to decode other frames?
>
> Yes, this is how I understand Tomasz Figa.
>
> > This means, if you occasionally/intentionally
> > write to the buffer you mess up the whole decoding pipeline.
>
> And to avoid this the buffer handling aspect must be clarified in the
> specification.  Is the device allowed to continue using the buffer after
> finishing decoding and completing the queue request?  If so, how do we
> hand over buffer ownership back to the driver so it can free the pages?
> drain request?  How do we handle re-using buffers?  Can the driver
> simply re-queue them and expect the device figures by itself whenever it
> can use the buffer or whenever it is still needed as reference frame?

The device shouldn't be able to access buffers after it completes a
queue request.
The device can touch buffer contents from when a queue request is sent
until the device responds it.
In contrast, the driver must not modify buffer contents in that period.

Regarding re-using, the driver can simply re-queue buffers returned by
the device. If the device needs a buffer as reference frame, it must
not return the buffer.
I'll describe this rule in the next version of the patch.

Best regards,
Keiichi

>
> cheers,
>   Gerd
>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [virtio-dev] Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2020-01-08 13:52                 ` Keiichi Watanabe
@ 2020-01-09 13:40                   ` Gerd Hoffmann
  2020-01-09 14:20                   ` Tomasz Figa
  1 sibling, 0 replies; 73+ messages in thread
From: Gerd Hoffmann @ 2020-01-09 13:40 UTC (permalink / raw)
  To: Keiichi Watanabe
  Cc: Dmitry Sepp, Tomasz Figa, virtio-dev, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Enrico Granata, Frediano Ziglio, Hans Verkuil,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

  Hi,

> Regarding re-using, the driver can simply re-queue buffers returned by
> the device. If the device needs a buffer as reference frame, it must
> not return the buffer.

Ok, that'll work.

> I'll describe this rule in the next version of the patch.

Good.  You should also add a note about ordering.  If the device returns
the buffers as soon as they are no longer needed for decoding they
might be completed out-of-order, specifically B-Frames might complete
before the reference I/P frame.  Is the device allowed to do that or
should it complete the buffers in playback order?

cheers,
  Gerd


^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [virtio-dev] Re: [PATCH v2 1/1] virtio-video: Add virtio video device specification
  2020-01-08 13:52                 ` Keiichi Watanabe
  2020-01-09 13:40                   ` Gerd Hoffmann
@ 2020-01-09 14:20                   ` Tomasz Figa
  2020-01-14  7:18                     ` Keiichi Watanabe
  1 sibling, 1 reply; 73+ messages in thread
From: Tomasz Figa @ 2020-01-09 14:20 UTC (permalink / raw)
  To: Keiichi Watanabe
  Cc: Gerd Hoffmann, Dmitry Sepp, virtio-dev, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Enrico Granata, Frediano Ziglio, Hans Verkuil,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

On Wed, Jan 8, 2020 at 10:52 PM Keiichi Watanabe <keiichiw@chromium.org> wrote:
>
> Hi Gerd,
>
> On Thu, Dec 19, 2019 at 10:12 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
> >
> >   Hi,
> >
> > > > However that still doesn't let the driver know which buffers will be
> > > > dequeued when. A simple example of this scenario is when the guest is
> > > > done displaying a frame and requeues the buffer back to the decoder.
> > > > Then the decoder will not choose it for decoding next frames into as
> > > > long as the frame in that buffer is still used as a reference frame,
> > > > even if one sends the drain request.
> > > It might be that I'm getting your point wrong, but do you mean some hardware
> > > can mark a buffer as ready to be displayed yet still using the underlying
> > > memory to decode other frames?
> >
> > Yes, this is how I understand Tomasz Figa.
> >
> > > This means, if you occasionally/intentionally
> > > write to the buffer you mess up the whole decoding pipeline.
> >
> > And to avoid this the buffer handling aspect must be clarified in the
> > specification.  Is the device allowed to continue using the buffer after
> > finishing decoding and completing the queue request?  If so, how do we
> > hand over buffer ownership back to the driver so it can free the pages?
> > drain request?  How do we handle re-using buffers?  Can the driver
> > simply re-queue them and expect the device figures by itself whenever it
> > can use the buffer or whenever it is still needed as reference frame?
>
> The device shouldn't be able to access buffers after it completes a
> queue request.
> The device can touch buffer contents from when a queue request is sent
> until the device responds it.
> In contrast, the driver must not modify buffer contents in that period.
>
> Regarding re-using, the driver can simply re-queue buffers returned by
> the device. If the device needs a buffer as reference frame, it must
> not return the buffer.

I think that might not be what we expect. We want the decoder to
return a decoded frame as soon as possible, but that decoded frame may
be also needed for decoding next frames. In V4L2 stateful decoder, the
API is defined that the client must not modify the decoded
framebuffer, otherwise decoding next frames may not be correct. We may
need something similar, with an explicit operation that makes the
buffers not accessible to the host anymore. I think the queue flush
operation could work as such.

> I'll describe this rule in the next version of the patch.
>
> Best regards,
> Keiichi
>
> >
> > cheers,
> >   Gerd
> >

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v2 0/1] VirtIO video device specification
  2020-01-07 10:25               ` Keiichi Watanabe
@ 2020-01-09 14:56                 ` Dmitry Sepp
  2020-01-10 10:16                   ` [virtio-dev] " Dmitry Sepp
  0 siblings, 1 reply; 73+ messages in thread
From: Dmitry Sepp @ 2020-01-09 14:56 UTC (permalink / raw)
  To: Keiichi Watanabe
  Cc: Tomasz Figa, virtio-dev, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Enrico Granata, Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

Hi,

On Dienstag, 7. Januar 2020 11:25:56 CET Keiichi Watanabe wrote:
> Hi Dmitry,
> 
> On Mon, Jan 6, 2020 at 8:28 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> 
wrote:
> > Hi,
> > 
> > On Montag, 6. Januar 2020 11:30:22 CET Keiichi Watanabe wrote:
> > > Hi Dmitry, Tomasz,
> > > 
> > > On Fri, Jan 3, 2020 at 10:05 PM Dmitry Sepp
> > > <dmitry.sepp@opensynergy.com>
> > 
> > wrote:
> > > > Hi Tomasz, Keiichi,
> > > > 
> > > > On Samstag, 21. Dezember 2019 07:19:23 CET Tomasz Figa wrote:
> > > > > On Sat, Dec 21, 2019 at 3:18 PM Tomasz Figa <tfiga@chromium.org> 
wrote:
> > > > > > On Sat, Dec 21, 2019 at 1:36 PM Keiichi Watanabe
> > > > > > <keiichiw@chromium.org>
> > > > 
> > > > wrote:
> > > > > > > Hi Dmitry,
> > > > > > > 
> > > > > > > On Sat, Dec 21, 2019 at 12:59 AM Dmitry Sepp
> > > > > > > 
> > > > > > > <dmitry.sepp@opensynergy.com> wrote:
> > > > > > > > Hi Keiichi,
> > > > > > > > 
> > > > > > > > On Mittwoch, 18. Dezember 2019 14:02:13 CET Keiichi Watanabe
> > 
> > wrote:
> > > > > > > > > Hi,
> > > > > > > > > This is the 2nd version of virtio-video patch. The PDF is
> > > > > > > > > available
> > > > > > > > > in [1].
> > > > > > > > > The first version was sent at [2].
> > > > > > > > > 
> > > > > > > > > Any feedback would be appreciated. Thank you.
> > > > > > > > > 
> > > > > > > > > Best,
> > > > > > > > > Keiichi
> > > > > > > > > 
> > > > > > > > > [1]:
> > > > > > > > > https://drive.google.com/drive/folders/1eT5fEckBoor2iHZR4f4G
> > > > > > > > > LxYz
> > > > > > > > > FMVa
> > > > > > > > > pOFx?us
> > > > > > > > > p=sharing [2]: https://markmail.org/message/gc6h25acct22niut
> > > > > > > > > 
> > > > > > > > > Change log:
> > > > > > > > > 
> > > > > > > > > v2:
> > > > > > > > > * Removed functionalities except encoding and decoding.
> > > > > > > > > * Splited encoder and decoder into different devices that
> > > > > > > > > use
> > > > > > > > > the
> > > > > > > > > same
> > > > > > > > > protocol. * Replaced GET_FUNCS with GET_CAPABILITY.
> > > > > > > > > * Updated structs for capabilities.
> > > > > > > > > 
> > > > > > > > >   - Defined new structs and enums such as image formats,
> > > > > > > > >   profiles,
> > > > > > > > >   range
> > > > > > > > > 
> > > > > > > > > (min, max, step), etc
> > > > > > > > > 
> > > > > > > > >     * For virtio_video_pixel_format, chose a naming
> > > > > > > > >     convention
> > > > > > > > >     that
> > > > > > > > >     is used
> > > > > > > > >     
> > > > > > > > >       in DRM. We removed XBGR, NV21 and I422, as they are
> > > > > > > > >       not
> > > > > > > > >       used
> > > > > > > > >       in the
> > > > > > > > >       current draft implementation.
> > > > > > > > >       https://lwn.net/Articles/806416/
> > > > > > > > >   
> > > > > > > > >   - Removed virtio_video_control, whose usage was not
> > > > > > > > >   documented
> > > > > > > > >   yet
> > > > > > > > >   and
> > > > > > > > > 
> > > > > > > > > which is not necessary for the simplest decoding scenario.
> > > > > > > > > 
> > > > > > > > >   - Removed virtio_video_desc, as it is no longer needed.
> > > > > > > > > 
> > > > > > > > > * Updated struct virtio_video_config for changes around
> > > > > > > > > capabilities.
> > > > > > > > > * Added a way to represent supported combinations of
> > > > > > > > > formats.
> > > > > > > > > 
> > > > > > > > >   - A field "mask" in virtio_video_format_desc plays this
> > > > > > > > >   role.
> > > > > > > > > 
> > > > > > > > > * Removed VIRTIO_VIDEO_T_STREAM_{START,STOP} because they
> > > > > > > > > don't
> > > > > > > > > play
> > > > > > > > > any
> > > > > > > > > meaningful roles. * Removed VIRTIO_VIDEO_T_STREAM_{ATTACH,
> > > > > > > > > DETACH}_BACKING
> > > > > > > > > and merged them into RESOURCE_{CREATE, DESTROY}. * Added a
> > > > > > > > > way
> > > > > > > > > to
> > > > > > > > > notify/specify resource creation method.
> > > > > > > > > 
> > > > > > > > >   - Added a feature flag.
> > > > > > > > >   - Defined enum virtio_video_mem_type.
> > > > > > > > >   - Added new fields in video_stream_create.
> > > > > > > > > 
> > > > > > > > > * Modified fields in virtio_video_params.
> > > > > > > > > 
> > > > > > > > >   - Added crop information.
> > > > > > > > > 
> > > > > > > > > * Removed enum virtio_video_channel_type because we can get
> > > > > > > > > this
> > > > > > > > > information by image format.
> > > > > > > > 
> > > > > > > > Could you please explain this? How do you get the information?
> > > > > > > 
> > > > > > > It means that if image formats are well-defined, channel
> > > > > > > information
> > > > > > > (e.g. the order of channels) is uniquely determined.
> > > > > > > 
> > > > > > > > Suppose you have some piece of HW on the host side that wants
> > > > > > > > I420
> > > > > > > > as
> > > > > > > > one
> > > > > > > > contig buffer w/ some offsets. But on the driver side, say,
> > > > > > > > gralloc
> > > > > > > > gives you three separate buffers, one per channel. How do we
> > > > > > > > pass
> > > > > > > > those to the device then?
> > > > > > > 
> > > > > > > You're talking about CrOS use case where buffers are allocated
> > > > > > > by
> > > > > > > virtio-gpu, right?
> > > > > > > In this case, virtio-gpu allocates one contiguous host-side
> > > > > > > buffer
> > > > > > > and
> > > > > > > the client regards a pair of (buffer FD, offset) as one channel.
> > > > > > > And, we can register this pair to the device when the buffer is
> > > > > > > imported.
> > > > > > > In the virtio-vdec spec draft, this pair corresponds to struct
> > > > > > > virtio_vdec_plane in struct virtio_vdec_plane.
> > > > > > > 
> > > > > > > So, I suppose we will need similar structs when we add a control
> > > > > > > to
> > > > > > > import buffers. However, I don't think it's necessary when guest
> > > > > > > pages
> > > > > > > are used.
> > > > > > 
> > > > > > I think we need some way for the guest to know whether it can
> > > > > > allocate
> > > > > > the planes in separate buffers, even when guest pages are used.
> > > > > > This
> > > > > > would be equivalent to V4L2 M and non-M formats, but mixing this
> > > > > > into
> > > > > > FourCC in V4L2 is an acknowledged mistake, so we should add a
> > > > > > query or
> > > > > > something.
> > > > 
> > > > Yes, this is what I mean. In fact, we already do face the situation
> > > > when
> > > > the device side is not happy with the sgt and wants contig. I think
> > > > we'll
> > > > add a module parameter for now.
> > > 
> > > Okay. So, I suppose we'll be able to update structs:
> > > * Add a flag in virtio_video_format_desc that indicates whether planes
> > > can be in separate buffers, and
> > > * Add a flag in virtio_video_format_desc that indicates that the
> > > device requires contiguous buffers for this format.
> > > 
> > > Does it make sense?
> > 
> > Sorry, I don't understand the difference between the two above: isn't the
> > first case is just when the flag is not set?
> 
> Ah, I was confused and wrote something strange. Yeah,  these two are the
> same. Sorry for that.
> 
> So, the suggestion is to add a field "planes_layout" in
> virtio_video_format_desc, which is one of the following enums:
> 
> enum virtio_video_planes_layout {
>     VIRTIO_VIDEO_PLANES_LAYOUT_UNSPEC = 0,  /* no special requirement */
>     VIRTIO_VIDEO_PLANES_LAYOUT_CONTIGUOUS,
> };
> 
> If we have a better idea or naming, please let me know.
> 

The naming looks good for me, I might only change to CONTIG as we have UNSPEC.

Best regards,
Dmitry.

> Best regards,
> Keiichi
> 
> > Regards,
> > Dmitry.
> > 
> > > Best regards,
> > > Keiichi
> > > 
> > > > Regards,
> > > > Dmitry.
> > > > 
> > > > > > For future V4L2 development we came up with the idea of a format
> > > > > > flag
> > > > > > which could mean that the hardware allows putting planes in
> > > > > > separate
> > > > > > buffers. We could have a similar per-format flag in the
> > > > > > capabilities,
> > > > > > as we already have a list of all the supported formats there.
> > > > > 
> > > > > Sorry, forgot to paste the link from future V4L2 work notes from
> > > > > this
> > > > > year
> > > > > ELCE: https://www.spinics.net/lists/linux-media/msg159789.html
> > > > > 
> > > > > > Best regards,
> > > > > > Tomasz
> > > > > > 
> > > > > > > Best regards,
> > > > > > > Keiichi
> > > > > > > 
> > > > > > > > Best regards,
> > > > > > > > Dmitry.
> > > > > > > > 
> > > > > > > > > * Renamed virtio_video_pin to virtio_video_buf_type.
> > > > > > > > > 
> > > > > > > > >   - It's similar to V4L2_BUF_TYPE_VIDEO_{OUTPUT, CAPTURE}.
> > > > > > > > > 
> > > > > > > > > * Added an error event.
> > > > > > > > > * Reordered some subsections.
> > > > > > > > > * Changed styles to make it consistent with other devices.
> > > > > > > > > 
> > > > > > > > > Dmitry Sepp (1):
> > > > > > > > >   virtio-video: Add virtio video device specification
> > > > > > > > >  
> > > > > > > > >  content.tex      |   1 +
> > > > > > > > >  virtio-video.tex | 579
> > > > > > > > >  +++++++++++++++++++++++++++++++++++++++++++++++
> > > > > > > > >  2 files changed, 580 insertions(+)
> > > > > > > > >  create mode 100644 virtio-video.tex
> > > > > > > > > 
> > > > > > > > > --
> > > > > > > > > 2.24.1.735.g03f4e72817-goog



^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [virtio-dev] Re: [PATCH v2 0/1] VirtIO video device specification
  2020-01-09 14:56                 ` Dmitry Sepp
@ 2020-01-10 10:16                   ` Dmitry Sepp
  2020-01-10 13:53                     ` Keiichi Watanabe
  0 siblings, 1 reply; 73+ messages in thread
From: Dmitry Sepp @ 2020-01-10 10:16 UTC (permalink / raw)
  To: Keiichi Watanabe, virtio-dev
  Cc: Tomasz Figa, virtio-dev, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Enrico Granata, Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril, Dmitry Sepp

Hi,

On Donnerstag, 9. Januar 2020 15:56:08 CET Dmitry Sepp wrote:
> Hi,
> 
> On Dienstag, 7. Januar 2020 11:25:56 CET Keiichi Watanabe wrote:
> > Hi Dmitry,
> > 
> > On Mon, Jan 6, 2020 at 8:28 PM Dmitry Sepp <dmitry.sepp@opensynergy.com>
> 
> wrote:
> > > Hi,
> > > 
> > > On Montag, 6. Januar 2020 11:30:22 CET Keiichi Watanabe wrote:
> > > > Hi Dmitry, Tomasz,
> > > > 
> > > > On Fri, Jan 3, 2020 at 10:05 PM Dmitry Sepp
> > > > <dmitry.sepp@opensynergy.com>
> > > 
> > > wrote:
> > > > > Hi Tomasz, Keiichi,
> > > > > 
> > > > > On Samstag, 21. Dezember 2019 07:19:23 CET Tomasz Figa wrote:
> > > > > > On Sat, Dec 21, 2019 at 3:18 PM Tomasz Figa <tfiga@chromium.org>
> 
> wrote:
> > > > > > > On Sat, Dec 21, 2019 at 1:36 PM Keiichi Watanabe
> > > > > > > <keiichiw@chromium.org>
> > > > > 
> > > > > wrote:
> > > > > > > > Hi Dmitry,
> > > > > > > > 
> > > > > > > > On Sat, Dec 21, 2019 at 12:59 AM Dmitry Sepp
> > > > > > > > 
> > > > > > > > <dmitry.sepp@opensynergy.com> wrote:
> > > > > > > > > Hi Keiichi,
> > > > > > > > > 
> > > > > > > > > On Mittwoch, 18. Dezember 2019 14:02:13 CET Keiichi Watanabe
> > > 
> > > wrote:
> > > > > > > > > > Hi,
> > > > > > > > > > This is the 2nd version of virtio-video patch. The PDF is
> > > > > > > > > > available
> > > > > > > > > > in [1].
> > > > > > > > > > The first version was sent at [2].
> > > > > > > > > > 
> > > > > > > > > > Any feedback would be appreciated. Thank you.
> > > > > > > > > > 
> > > > > > > > > > Best,
> > > > > > > > > > Keiichi
> > > > > > > > > > 
> > > > > > > > > > [1]:
> > > > > > > > > > https://drive.google.com/drive/folders/1eT5fEckBoor2iHZR4f
> > > > > > > > > > 4G
> > > > > > > > > > LxYz
> > > > > > > > > > FMVa
> > > > > > > > > > pOFx?us
> > > > > > > > > > p=sharing [2]:
> > > > > > > > > > https://markmail.org/message/gc6h25acct22niut
> > > > > > > > > > 
> > > > > > > > > > Change log:
> > > > > > > > > > 
> > > > > > > > > > v2:
> > > > > > > > > > * Removed functionalities except encoding and decoding.
> > > > > > > > > > * Splited encoder and decoder into different devices that
> > > > > > > > > > use
> > > > > > > > > > the
> > > > > > > > > > same
> > > > > > > > > > protocol. * Replaced GET_FUNCS with GET_CAPABILITY.
> > > > > > > > > > * Updated structs for capabilities.
> > > > > > > > > > 
> > > > > > > > > >   - Defined new structs and enums such as image formats,
> > > > > > > > > >   profiles,
> > > > > > > > > >   range
> > > > > > > > > > 
> > > > > > > > > > (min, max, step), etc
> > > > > > > > > > 
> > > > > > > > > >     * For virtio_video_pixel_format, chose a naming
> > > > > > > > > >     convention
> > > > > > > > > >     that
> > > > > > > > > >     is used
> > > > > > > > > >     
> > > > > > > > > >       in DRM. We removed XBGR, NV21 and I422, as they are
> > > > > > > > > >       not
> > > > > > > > > >       used
> > > > > > > > > >       in the
> > > > > > > > > >       current draft implementation.
> > > > > > > > > >       https://lwn.net/Articles/806416/
> > > > > > > > > >   
> > > > > > > > > >   - Removed virtio_video_control, whose usage was not
> > > > > > > > > >   documented
> > > > > > > > > >   yet
> > > > > > > > > >   and
> > > > > > > > > > 
> > > > > > > > > > which is not necessary for the simplest decoding scenario.
> > > > > > > > > > 
> > > > > > > > > >   - Removed virtio_video_desc, as it is no longer needed.
> > > > > > > > > > 
> > > > > > > > > > * Updated struct virtio_video_config for changes around
> > > > > > > > > > capabilities.
> > > > > > > > > > * Added a way to represent supported combinations of
> > > > > > > > > > formats.
> > > > > > > > > > 
> > > > > > > > > >   - A field "mask" in virtio_video_format_desc plays this
> > > > > > > > > >   role.
> > > > > > > > > > 
> > > > > > > > > > * Removed VIRTIO_VIDEO_T_STREAM_{START,STOP} because they
> > > > > > > > > > don't
> > > > > > > > > > play
> > > > > > > > > > any
> > > > > > > > > > meaningful roles. * Removed VIRTIO_VIDEO_T_STREAM_{ATTACH,
> > > > > > > > > > DETACH}_BACKING
> > > > > > > > > > and merged them into RESOURCE_{CREATE, DESTROY}. * Added a
> > > > > > > > > > way
> > > > > > > > > > to
> > > > > > > > > > notify/specify resource creation method.
> > > > > > > > > > 
> > > > > > > > > >   - Added a feature flag.
> > > > > > > > > >   - Defined enum virtio_video_mem_type.
> > > > > > > > > >   - Added new fields in video_stream_create.
> > > > > > > > > > 
> > > > > > > > > > * Modified fields in virtio_video_params.
> > > > > > > > > > 
> > > > > > > > > >   - Added crop information.
> > > > > > > > > > 
> > > > > > > > > > * Removed enum virtio_video_channel_type because we can
> > > > > > > > > > get
> > > > > > > > > > this
> > > > > > > > > > information by image format.
> > > > > > > > > 
> > > > > > > > > Could you please explain this? How do you get the
> > > > > > > > > information?
> > > > > > > > 
> > > > > > > > It means that if image formats are well-defined, channel
> > > > > > > > information
> > > > > > > > (e.g. the order of channels) is uniquely determined.
> > > > > > > > 
> > > > > > > > > Suppose you have some piece of HW on the host side that
> > > > > > > > > wants
> > > > > > > > > I420
> > > > > > > > > as
> > > > > > > > > one
> > > > > > > > > contig buffer w/ some offsets. But on the driver side, say,
> > > > > > > > > gralloc
> > > > > > > > > gives you three separate buffers, one per channel. How do we
> > > > > > > > > pass
> > > > > > > > > those to the device then?
> > > > > > > > 
> > > > > > > > You're talking about CrOS use case where buffers are allocated
> > > > > > > > by
> > > > > > > > virtio-gpu, right?
> > > > > > > > In this case, virtio-gpu allocates one contiguous host-side
> > > > > > > > buffer
> > > > > > > > and
> > > > > > > > the client regards a pair of (buffer FD, offset) as one
> > > > > > > > channel.
> > > > > > > > And, we can register this pair to the device when the buffer
> > > > > > > > is
> > > > > > > > imported.
> > > > > > > > In the virtio-vdec spec draft, this pair corresponds to struct
> > > > > > > > virtio_vdec_plane in struct virtio_vdec_plane.
> > > > > > > > 
> > > > > > > > So, I suppose we will need similar structs when we add a
> > > > > > > > control
> > > > > > > > to
> > > > > > > > import buffers. However, I don't think it's necessary when
> > > > > > > > guest
> > > > > > > > pages
> > > > > > > > are used.
> > > > > > > 
> > > > > > > I think we need some way for the guest to know whether it can
> > > > > > > allocate
> > > > > > > the planes in separate buffers, even when guest pages are used.
> > > > > > > This
> > > > > > > would be equivalent to V4L2 M and non-M formats, but mixing this
> > > > > > > into
> > > > > > > FourCC in V4L2 is an acknowledged mistake, so we should add a
> > > > > > > query or
> > > > > > > something.
> > > > > 
> > > > > Yes, this is what I mean. In fact, we already do face the situation
> > > > > when
> > > > > the device side is not happy with the sgt and wants contig. I think
> > > > > we'll
> > > > > add a module parameter for now.
> > > > 
> > > > Okay. So, I suppose we'll be able to update structs:
> > > > * Add a flag in virtio_video_format_desc that indicates whether planes
> > > > can be in separate buffers, and
> > > > * Add a flag in virtio_video_format_desc that indicates that the
> > > > device requires contiguous buffers for this format.
> > > > 
> > > > Does it make sense?
> > > 
> > > Sorry, I don't understand the difference between the two above: isn't
> > > the
> > > first case is just when the flag is not set?
> > 
> > Ah, I was confused and wrote something strange. Yeah,  these two are the
> > same. Sorry for that.
> > 
> > So, the suggestion is to add a field "planes_layout" in
> > virtio_video_format_desc, which is one of the following enums:
> > 
> > enum virtio_video_planes_layout {
> > 
> >     VIRTIO_VIDEO_PLANES_LAYOUT_UNSPEC = 0,  /* no special requirement */
> >     VIRTIO_VIDEO_PLANES_LAYOUT_CONTIGUOUS,
> > 
> > };
> > 
> > If we have a better idea or naming, please let me know.
> 
> The naming looks good for me, I might only change to CONTIG as we have
> UNSPEC.
> 

So here we are talking about plane layout in memory, am I correct? But I think 
we also need a way to communicate memory requirements of the device: the 
device might require CMA buffers or it can be ok with SG lists. What about 
adding something like this to virtio_video_format_desc:

​​​​​enum virtio_video_mem_layout {
	VIRTIO_VIDEO_MEM_LAYOUT_UNDEFINED = 0,

	VIRTIO_VIDEO_MEM_LAYOUT_CONTIG = 0x100,
	VIRTIO_VIDEO_MEM_LAYOUT_NON_CONTIG,
};

struct virtio_video_format_desc {
    __le64 mask;
    __le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
    __le32 planes_layout; /* One of VIRTIO_VIDEO_PLANES_LAYOUT_* types */
    __le32 mem_layout; /* One of VIRTIO_VIDEO_MEM_LAYOUT_* types */
    ...
};

Best regards,
Dmitry.

> Best regards,
> Dmitry.
> 
> > Best regards,
> > Keiichi
> > 
> > > Regards,
> > > Dmitry.
> > > 
> > > > Best regards,
> > > > Keiichi
> > > > 
> > > > > Regards,
> > > > > Dmitry.
> > > > > 
> > > > > > > For future V4L2 development we came up with the idea of a format
> > > > > > > flag
> > > > > > > which could mean that the hardware allows putting planes in
> > > > > > > separate
> > > > > > > buffers. We could have a similar per-format flag in the
> > > > > > > capabilities,
> > > > > > > as we already have a list of all the supported formats there.
> > > > > > 
> > > > > > Sorry, forgot to paste the link from future V4L2 work notes from
> > > > > > this
> > > > > > year
> > > > > > ELCE: https://www.spinics.net/lists/linux-media/msg159789.html
> > > > > > 
> > > > > > > Best regards,
> > > > > > > Tomasz
> > > > > > > 
> > > > > > > > Best regards,
> > > > > > > > Keiichi
> > > > > > > > 
> > > > > > > > > Best regards,
> > > > > > > > > Dmitry.
> > > > > > > > > 
> > > > > > > > > > * Renamed virtio_video_pin to virtio_video_buf_type.
> > > > > > > > > > 
> > > > > > > > > >   - It's similar to V4L2_BUF_TYPE_VIDEO_{OUTPUT, CAPTURE}.
> > > > > > > > > > 
> > > > > > > > > > * Added an error event.
> > > > > > > > > > * Reordered some subsections.
> > > > > > > > > > * Changed styles to make it consistent with other devices.
> > > > > > > > > > 
> > > > > > > > > > Dmitry Sepp (1):
> > > > > > > > > >   virtio-video: Add virtio video device specification
> > > > > > > > > >  
> > > > > > > > > >  content.tex      |   1 +
> > > > > > > > > >  virtio-video.tex | 579
> > > > > > > > > >  +++++++++++++++++++++++++++++++++++++++++++++++
> > > > > > > > > >  2 files changed, 580 insertions(+)
> > > > > > > > > >  create mode 100644 virtio-video.tex
> > > > > > > > > > 
> > > > > > > > > > --
> > > > > > > > > > 2.24.1.735.g03f4e72817-goog
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org



^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [virtio-dev] Re: [PATCH v2 0/1] VirtIO video device specification
  2020-01-10 10:16                   ` [virtio-dev] " Dmitry Sepp
@ 2020-01-10 13:53                     ` Keiichi Watanabe
  2020-01-10 15:11                       ` Dmitry Sepp
  0 siblings, 1 reply; 73+ messages in thread
From: Keiichi Watanabe @ 2020-01-10 13:53 UTC (permalink / raw)
  To: Dmitry Sepp
  Cc: virtio-dev, Tomasz Figa, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Enrico Granata, Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

Hi,


On Fri, Jan 10, 2020 at 7:16 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
>
> Hi,
>
> On Donnerstag, 9. Januar 2020 15:56:08 CET Dmitry Sepp wrote:
> > Hi,
> >
> > On Dienstag, 7. Januar 2020 11:25:56 CET Keiichi Watanabe wrote:
> > > Hi Dmitry,
> > >
> > > On Mon, Jan 6, 2020 at 8:28 PM Dmitry Sepp <dmitry.sepp@opensynergy.com>
> >
> > wrote:
> > > > Hi,
> > > >
> > > > On Montag, 6. Januar 2020 11:30:22 CET Keiichi Watanabe wrote:
> > > > > Hi Dmitry, Tomasz,
> > > > >
> > > > > On Fri, Jan 3, 2020 at 10:05 PM Dmitry Sepp
> > > > > <dmitry.sepp@opensynergy.com>
> > > >
> > > > wrote:
> > > > > > Hi Tomasz, Keiichi,
> > > > > >
> > > > > > On Samstag, 21. Dezember 2019 07:19:23 CET Tomasz Figa wrote:
> > > > > > > On Sat, Dec 21, 2019 at 3:18 PM Tomasz Figa <tfiga@chromium.org>
> >
> > wrote:
> > > > > > > > On Sat, Dec 21, 2019 at 1:36 PM Keiichi Watanabe
> > > > > > > > <keiichiw@chromium.org>
> > > > > >
> > > > > > wrote:
> > > > > > > > > Hi Dmitry,
> > > > > > > > >
> > > > > > > > > On Sat, Dec 21, 2019 at 12:59 AM Dmitry Sepp
> > > > > > > > >
> > > > > > > > > <dmitry.sepp@opensynergy.com> wrote:
> > > > > > > > > > Hi Keiichi,
> > > > > > > > > >
> > > > > > > > > > On Mittwoch, 18. Dezember 2019 14:02:13 CET Keiichi Watanabe
> > > >
> > > > wrote:
> > > > > > > > > > > Hi,
> > > > > > > > > > > This is the 2nd version of virtio-video patch. The PDF is
> > > > > > > > > > > available
> > > > > > > > > > > in [1].
> > > > > > > > > > > The first version was sent at [2].
> > > > > > > > > > >
> > > > > > > > > > > Any feedback would be appreciated. Thank you.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Keiichi
> > > > > > > > > > >
> > > > > > > > > > > [1]:
> > > > > > > > > > > https://drive.google.com/drive/folders/1eT5fEckBoor2iHZR4f
> > > > > > > > > > > 4G
> > > > > > > > > > > LxYz
> > > > > > > > > > > FMVa
> > > > > > > > > > > pOFx?us
> > > > > > > > > > > p=sharing [2]:
> > > > > > > > > > > https://markmail.org/message/gc6h25acct22niut
> > > > > > > > > > >
> > > > > > > > > > > Change log:
> > > > > > > > > > >
> > > > > > > > > > > v2:
> > > > > > > > > > > * Removed functionalities except encoding and decoding.
> > > > > > > > > > > * Splited encoder and decoder into different devices that
> > > > > > > > > > > use
> > > > > > > > > > > the
> > > > > > > > > > > same
> > > > > > > > > > > protocol. * Replaced GET_FUNCS with GET_CAPABILITY.
> > > > > > > > > > > * Updated structs for capabilities.
> > > > > > > > > > >
> > > > > > > > > > >   - Defined new structs and enums such as image formats,
> > > > > > > > > > >   profiles,
> > > > > > > > > > >   range
> > > > > > > > > > >
> > > > > > > > > > > (min, max, step), etc
> > > > > > > > > > >
> > > > > > > > > > >     * For virtio_video_pixel_format, chose a naming
> > > > > > > > > > >     convention
> > > > > > > > > > >     that
> > > > > > > > > > >     is used
> > > > > > > > > > >
> > > > > > > > > > >       in DRM. We removed XBGR, NV21 and I422, as they are
> > > > > > > > > > >       not
> > > > > > > > > > >       used
> > > > > > > > > > >       in the
> > > > > > > > > > >       current draft implementation.
> > > > > > > > > > >       https://lwn.net/Articles/806416/
> > > > > > > > > > >
> > > > > > > > > > >   - Removed virtio_video_control, whose usage was not
> > > > > > > > > > >   documented
> > > > > > > > > > >   yet
> > > > > > > > > > >   and
> > > > > > > > > > >
> > > > > > > > > > > which is not necessary for the simplest decoding scenario.
> > > > > > > > > > >
> > > > > > > > > > >   - Removed virtio_video_desc, as it is no longer needed.
> > > > > > > > > > >
> > > > > > > > > > > * Updated struct virtio_video_config for changes around
> > > > > > > > > > > capabilities.
> > > > > > > > > > > * Added a way to represent supported combinations of
> > > > > > > > > > > formats.
> > > > > > > > > > >
> > > > > > > > > > >   - A field "mask" in virtio_video_format_desc plays this
> > > > > > > > > > >   role.
> > > > > > > > > > >
> > > > > > > > > > > * Removed VIRTIO_VIDEO_T_STREAM_{START,STOP} because they
> > > > > > > > > > > don't
> > > > > > > > > > > play
> > > > > > > > > > > any
> > > > > > > > > > > meaningful roles. * Removed VIRTIO_VIDEO_T_STREAM_{ATTACH,
> > > > > > > > > > > DETACH}_BACKING
> > > > > > > > > > > and merged them into RESOURCE_{CREATE, DESTROY}. * Added a
> > > > > > > > > > > way
> > > > > > > > > > > to
> > > > > > > > > > > notify/specify resource creation method.
> > > > > > > > > > >
> > > > > > > > > > >   - Added a feature flag.
> > > > > > > > > > >   - Defined enum virtio_video_mem_type.
> > > > > > > > > > >   - Added new fields in video_stream_create.
> > > > > > > > > > >
> > > > > > > > > > > * Modified fields in virtio_video_params.
> > > > > > > > > > >
> > > > > > > > > > >   - Added crop information.
> > > > > > > > > > >
> > > > > > > > > > > * Removed enum virtio_video_channel_type because we can
> > > > > > > > > > > get
> > > > > > > > > > > this
> > > > > > > > > > > information by image format.
> > > > > > > > > >
> > > > > > > > > > Could you please explain this? How do you get the
> > > > > > > > > > information?
> > > > > > > > >
> > > > > > > > > It means that if image formats are well-defined, channel
> > > > > > > > > information
> > > > > > > > > (e.g. the order of channels) is uniquely determined.
> > > > > > > > >
> > > > > > > > > > Suppose you have some piece of HW on the host side that
> > > > > > > > > > wants
> > > > > > > > > > I420
> > > > > > > > > > as
> > > > > > > > > > one
> > > > > > > > > > contig buffer w/ some offsets. But on the driver side, say,
> > > > > > > > > > gralloc
> > > > > > > > > > gives you three separate buffers, one per channel. How do we
> > > > > > > > > > pass
> > > > > > > > > > those to the device then?
> > > > > > > > >
> > > > > > > > > You're talking about CrOS use case where buffers are allocated
> > > > > > > > > by
> > > > > > > > > virtio-gpu, right?
> > > > > > > > > In this case, virtio-gpu allocates one contiguous host-side
> > > > > > > > > buffer
> > > > > > > > > and
> > > > > > > > > the client regards a pair of (buffer FD, offset) as one
> > > > > > > > > channel.
> > > > > > > > > And, we can register this pair to the device when the buffer
> > > > > > > > > is
> > > > > > > > > imported.
> > > > > > > > > In the virtio-vdec spec draft, this pair corresponds to struct
> > > > > > > > > virtio_vdec_plane in struct virtio_vdec_plane.
> > > > > > > > >
> > > > > > > > > So, I suppose we will need similar structs when we add a
> > > > > > > > > control
> > > > > > > > > to
> > > > > > > > > import buffers. However, I don't think it's necessary when
> > > > > > > > > guest
> > > > > > > > > pages
> > > > > > > > > are used.
> > > > > > > >
> > > > > > > > I think we need some way for the guest to know whether it can
> > > > > > > > allocate
> > > > > > > > the planes in separate buffers, even when guest pages are used.
> > > > > > > > This
> > > > > > > > would be equivalent to V4L2 M and non-M formats, but mixing this
> > > > > > > > into
> > > > > > > > FourCC in V4L2 is an acknowledged mistake, so we should add a
> > > > > > > > query or
> > > > > > > > something.
> > > > > >
> > > > > > Yes, this is what I mean. In fact, we already do face the situation
> > > > > > when
> > > > > > the device side is not happy with the sgt and wants contig. I think
> > > > > > we'll
> > > > > > add a module parameter for now.
> > > > >
> > > > > Okay. So, I suppose we'll be able to update structs:
> > > > > * Add a flag in virtio_video_format_desc that indicates whether planes
> > > > > can be in separate buffers, and
> > > > > * Add a flag in virtio_video_format_desc that indicates that the
> > > > > device requires contiguous buffers for this format.
> > > > >
> > > > > Does it make sense?
> > > >
> > > > Sorry, I don't understand the difference between the two above: isn't
> > > > the
> > > > first case is just when the flag is not set?
> > >
> > > Ah, I was confused and wrote something strange. Yeah,  these two are the
> > > same. Sorry for that.
> > >
> > > So, the suggestion is to add a field "planes_layout" in
> > > virtio_video_format_desc, which is one of the following enums:
> > >
> > > enum virtio_video_planes_layout {
> > >
> > >     VIRTIO_VIDEO_PLANES_LAYOUT_UNSPEC = 0,  /* no special requirement */
> > >     VIRTIO_VIDEO_PLANES_LAYOUT_CONTIGUOUS,
> > >
> > > };
> > >
> > > If we have a better idea or naming, please let me know.
> >
> > The naming looks good for me, I might only change to CONTIG as we have
> > UNSPEC.
> >
>
> So here we are talking about plane layout in memory, am I correct? But I think
> we also need a way to communicate memory requirements of the device: the
> device might require CMA buffers or it can be ok with SG lists. What about
> adding something like this to virtio_video_format_desc:
>
> enum virtio_video_mem_layout {
>         VIRTIO_VIDEO_MEM_LAYOUT_UNDEFINED = 0,
>
>         VIRTIO_VIDEO_MEM_LAYOUT_CONTIG = 0x100,
>         VIRTIO_VIDEO_MEM_LAYOUT_NON_CONTIG,
> };
>
> struct virtio_video_format_desc {
>     __le64 mask;
>     __le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
>     __le32 planes_layout; /* One of VIRTIO_VIDEO_PLANES_LAYOUT_* types */
>     __le32 mem_layout; /* One of VIRTIO_VIDEO_MEM_LAYOUT_* types */
>     ...
> };

Good.
I'd not like to call it NON_CONTIG, as it sounds like CMA buffers
aren't allowed.
Instead, how about this definition?

enum virtio_video_mem_layout {
        VIRTIO_VIDEO_MEM_LAYOUT_UNSPEC = 0,  /* no special requirement */
        VIRTIO_VIDEO_MEM_LAYOUT_CONTIG = 1,
};

With this enum,
* the device can simply ignore this field if it doesn't have any
requirement and the struct is zero-initialized, and
* if we need to add other types of memory layout requirements, we can
add them as 2, 4, 8, etc to represent combinations of requirements.

Just to confirm, are the following combination of planes_layout and
mem_layout valid?
(1) (planes_layout, mem_layout) = (contig, not specified)
(2) (planes_layout, mem_layout) = (not specified, contig)

In my understanding, (1) means that each plane must be a contiguous
buffer while different planes don't have to be in a contig memory, but
(2) is invalid.
Is it correct?

Best regards,
Keiichi

>
> Best regards,
> Dmitry.
>
> > Best regards,
> > Dmitry.
> >
> > > Best regards,
> > > Keiichi
> > >
> > > > Regards,
> > > > Dmitry.
> > > >
> > > > > Best regards,
> > > > > Keiichi
> > > > >
> > > > > > Regards,
> > > > > > Dmitry.
> > > > > >
> > > > > > > > For future V4L2 development we came up with the idea of a format
> > > > > > > > flag
> > > > > > > > which could mean that the hardware allows putting planes in
> > > > > > > > separate
> > > > > > > > buffers. We could have a similar per-format flag in the
> > > > > > > > capabilities,
> > > > > > > > as we already have a list of all the supported formats there.
> > > > > > >
> > > > > > > Sorry, forgot to paste the link from future V4L2 work notes from
> > > > > > > this
> > > > > > > year
> > > > > > > ELCE: https://www.spinics.net/lists/linux-media/msg159789.html
> > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Tomasz
> > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Keiichi
> > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Dmitry.
> > > > > > > > > >
> > > > > > > > > > > * Renamed virtio_video_pin to virtio_video_buf_type.
> > > > > > > > > > >
> > > > > > > > > > >   - It's similar to V4L2_BUF_TYPE_VIDEO_{OUTPUT, CAPTURE}.
> > > > > > > > > > >
> > > > > > > > > > > * Added an error event.
> > > > > > > > > > > * Reordered some subsections.
> > > > > > > > > > > * Changed styles to make it consistent with other devices.
> > > > > > > > > > >
> > > > > > > > > > > Dmitry Sepp (1):
> > > > > > > > > > >   virtio-video: Add virtio video device specification
> > > > > > > > > > >
> > > > > > > > > > >  content.tex      |   1 +
> > > > > > > > > > >  virtio-video.tex | 579
> > > > > > > > > > >  +++++++++++++++++++++++++++++++++++++++++++++++
> > > > > > > > > > >  2 files changed, 580 insertions(+)
> > > > > > > > > > >  create mode 100644 virtio-video.tex
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > 2.24.1.735.g03f4e72817-goog
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>
>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [virtio-dev] Re: [PATCH v2 0/1] VirtIO video device specification
  2020-01-10 13:53                     ` Keiichi Watanabe
@ 2020-01-10 15:11                       ` Dmitry Sepp
  2020-01-11 16:06                         ` Tomasz Figa
  2020-01-13  9:56                         ` Gerd Hoffmann
  0 siblings, 2 replies; 73+ messages in thread
From: Dmitry Sepp @ 2020-01-10 15:11 UTC (permalink / raw)
  To: Keiichi Watanabe
  Cc: virtio-dev, Tomasz Figa, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Enrico Granata, Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

Hi Keiichi,

On Freitag, 10. Januar 2020 14:53:01 CET Keiichi Watanabe wrote:
> Hi,
> 
> On Fri, Jan 10, 2020 at 7:16 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> 
wrote:
> > Hi,
> > 
> > On Donnerstag, 9. Januar 2020 15:56:08 CET Dmitry Sepp wrote:
> > > Hi,
> > > 
> > > On Dienstag, 7. Januar 2020 11:25:56 CET Keiichi Watanabe wrote:
> > > > Hi Dmitry,
> > > > 
> > > > On Mon, Jan 6, 2020 at 8:28 PM Dmitry Sepp
> > > > <dmitry.sepp@opensynergy.com>
> > > 
> > > wrote:
> > > > > Hi,
> > > > > 
> > > > > On Montag, 6. Januar 2020 11:30:22 CET Keiichi Watanabe wrote:
> > > > > > Hi Dmitry, Tomasz,
> > > > > > 
> > > > > > On Fri, Jan 3, 2020 at 10:05 PM Dmitry Sepp
> > > > > > <dmitry.sepp@opensynergy.com>
> > > > > 
> > > > > wrote:
> > > > > > > Hi Tomasz, Keiichi,
> > > > > > > 
> > > > > > > On Samstag, 21. Dezember 2019 07:19:23 CET Tomasz Figa wrote:
> > > > > > > > On Sat, Dec 21, 2019 at 3:18 PM Tomasz Figa
> > > > > > > > <tfiga@chromium.org>
> > > 
> > > wrote:
> > > > > > > > > On Sat, Dec 21, 2019 at 1:36 PM Keiichi Watanabe
> > > > > > > > > <keiichiw@chromium.org>
> > > > > > > 
> > > > > > > wrote:
> > > > > > > > > > Hi Dmitry,
> > > > > > > > > > 
> > > > > > > > > > On Sat, Dec 21, 2019 at 12:59 AM Dmitry Sepp
> > > > > > > > > > 
> > > > > > > > > > <dmitry.sepp@opensynergy.com> wrote:
> > > > > > > > > > > Hi Keiichi,
> > > > > > > > > > > 
> > > > > > > > > > > On Mittwoch, 18. Dezember 2019 14:02:13 CET Keiichi
> > > > > > > > > > > Watanabe
> > > > > 
> > > > > wrote:
> > > > > > > > > > > > Hi,
> > > > > > > > > > > > This is the 2nd version of virtio-video patch. The PDF
> > > > > > > > > > > > is
> > > > > > > > > > > > available
> > > > > > > > > > > > in [1].
> > > > > > > > > > > > The first version was sent at [2].
> > > > > > > > > > > > 
> > > > > > > > > > > > Any feedback would be appreciated. Thank you.
> > > > > > > > > > > > 
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Keiichi
> > > > > > > > > > > > 
> > > > > > > > > > > > [1]:
> > > > > > > > > > > > https://drive.google.com/drive/folders/1eT5fEckBoor2iH
> > > > > > > > > > > > ZR4f
> > > > > > > > > > > > 4G
> > > > > > > > > > > > LxYz
> > > > > > > > > > > > FMVa
> > > > > > > > > > > > pOFx?us
> > > > > > > > > > > > p=sharing [2]:
> > > > > > > > > > > > https://markmail.org/message/gc6h25acct22niut
> > > > > > > > > > > > 
> > > > > > > > > > > > Change log:
> > > > > > > > > > > > 
> > > > > > > > > > > > v2:
> > > > > > > > > > > > * Removed functionalities except encoding and
> > > > > > > > > > > > decoding.
> > > > > > > > > > > > * Splited encoder and decoder into different devices
> > > > > > > > > > > > that
> > > > > > > > > > > > use
> > > > > > > > > > > > the
> > > > > > > > > > > > same
> > > > > > > > > > > > protocol. * Replaced GET_FUNCS with GET_CAPABILITY.
> > > > > > > > > > > > * Updated structs for capabilities.
> > > > > > > > > > > > 
> > > > > > > > > > > >   - Defined new structs and enums such as image
> > > > > > > > > > > >   formats,
> > > > > > > > > > > >   profiles,
> > > > > > > > > > > >   range
> > > > > > > > > > > > 
> > > > > > > > > > > > (min, max, step), etc
> > > > > > > > > > > > 
> > > > > > > > > > > >     * For virtio_video_pixel_format, chose a naming
> > > > > > > > > > > >     convention
> > > > > > > > > > > >     that
> > > > > > > > > > > >     is used
> > > > > > > > > > > >     
> > > > > > > > > > > >       in DRM. We removed XBGR, NV21 and I422, as they
> > > > > > > > > > > >       are
> > > > > > > > > > > >       not
> > > > > > > > > > > >       used
> > > > > > > > > > > >       in the
> > > > > > > > > > > >       current draft implementation.
> > > > > > > > > > > >       https://lwn.net/Articles/806416/
> > > > > > > > > > > >   
> > > > > > > > > > > >   - Removed virtio_video_control, whose usage was not
> > > > > > > > > > > >   documented
> > > > > > > > > > > >   yet
> > > > > > > > > > > >   and
> > > > > > > > > > > > 
> > > > > > > > > > > > which is not necessary for the simplest decoding
> > > > > > > > > > > > scenario.
> > > > > > > > > > > > 
> > > > > > > > > > > >   - Removed virtio_video_desc, as it is no longer
> > > > > > > > > > > >   needed.
> > > > > > > > > > > > 
> > > > > > > > > > > > * Updated struct virtio_video_config for changes
> > > > > > > > > > > > around
> > > > > > > > > > > > capabilities.
> > > > > > > > > > > > * Added a way to represent supported combinations of
> > > > > > > > > > > > formats.
> > > > > > > > > > > > 
> > > > > > > > > > > >   - A field "mask" in virtio_video_format_desc plays
> > > > > > > > > > > >   this
> > > > > > > > > > > >   role.
> > > > > > > > > > > > 
> > > > > > > > > > > > * Removed VIRTIO_VIDEO_T_STREAM_{START,STOP} because
> > > > > > > > > > > > they
> > > > > > > > > > > > don't
> > > > > > > > > > > > play
> > > > > > > > > > > > any
> > > > > > > > > > > > meaningful roles. * Removed
> > > > > > > > > > > > VIRTIO_VIDEO_T_STREAM_{ATTACH,
> > > > > > > > > > > > DETACH}_BACKING
> > > > > > > > > > > > and merged them into RESOURCE_{CREATE, DESTROY}. *
> > > > > > > > > > > > Added a
> > > > > > > > > > > > way
> > > > > > > > > > > > to
> > > > > > > > > > > > notify/specify resource creation method.
> > > > > > > > > > > > 
> > > > > > > > > > > >   - Added a feature flag.
> > > > > > > > > > > >   - Defined enum virtio_video_mem_type.
> > > > > > > > > > > >   - Added new fields in video_stream_create.
> > > > > > > > > > > > 
> > > > > > > > > > > > * Modified fields in virtio_video_params.
> > > > > > > > > > > > 
> > > > > > > > > > > >   - Added crop information.
> > > > > > > > > > > > 
> > > > > > > > > > > > * Removed enum virtio_video_channel_type because we
> > > > > > > > > > > > can
> > > > > > > > > > > > get
> > > > > > > > > > > > this
> > > > > > > > > > > > information by image format.
> > > > > > > > > > > 
> > > > > > > > > > > Could you please explain this? How do you get the
> > > > > > > > > > > information?
> > > > > > > > > > 
> > > > > > > > > > It means that if image formats are well-defined, channel
> > > > > > > > > > information
> > > > > > > > > > (e.g. the order of channels) is uniquely determined.
> > > > > > > > > > 
> > > > > > > > > > > Suppose you have some piece of HW on the host side that
> > > > > > > > > > > wants
> > > > > > > > > > > I420
> > > > > > > > > > > as
> > > > > > > > > > > one
> > > > > > > > > > > contig buffer w/ some offsets. But on the driver side,
> > > > > > > > > > > say,
> > > > > > > > > > > gralloc
> > > > > > > > > > > gives you three separate buffers, one per channel. How
> > > > > > > > > > > do we
> > > > > > > > > > > pass
> > > > > > > > > > > those to the device then?
> > > > > > > > > > 
> > > > > > > > > > You're talking about CrOS use case where buffers are
> > > > > > > > > > allocated
> > > > > > > > > > by
> > > > > > > > > > virtio-gpu, right?
> > > > > > > > > > In this case, virtio-gpu allocates one contiguous
> > > > > > > > > > host-side
> > > > > > > > > > buffer
> > > > > > > > > > and
> > > > > > > > > > the client regards a pair of (buffer FD, offset) as one
> > > > > > > > > > channel.
> > > > > > > > > > And, we can register this pair to the device when the
> > > > > > > > > > buffer
> > > > > > > > > > is
> > > > > > > > > > imported.
> > > > > > > > > > In the virtio-vdec spec draft, this pair corresponds to
> > > > > > > > > > struct
> > > > > > > > > > virtio_vdec_plane in struct virtio_vdec_plane.
> > > > > > > > > > 
> > > > > > > > > > So, I suppose we will need similar structs when we add a
> > > > > > > > > > control
> > > > > > > > > > to
> > > > > > > > > > import buffers. However, I don't think it's necessary when
> > > > > > > > > > guest
> > > > > > > > > > pages
> > > > > > > > > > are used.
> > > > > > > > > 
> > > > > > > > > I think we need some way for the guest to know whether it
> > > > > > > > > can
> > > > > > > > > allocate
> > > > > > > > > the planes in separate buffers, even when guest pages are
> > > > > > > > > used.
> > > > > > > > > This
> > > > > > > > > would be equivalent to V4L2 M and non-M formats, but mixing
> > > > > > > > > this
> > > > > > > > > into
> > > > > > > > > FourCC in V4L2 is an acknowledged mistake, so we should add
> > > > > > > > > a
> > > > > > > > > query or
> > > > > > > > > something.
> > > > > > > 
> > > > > > > Yes, this is what I mean. In fact, we already do face the
> > > > > > > situation
> > > > > > > when
> > > > > > > the device side is not happy with the sgt and wants contig. I
> > > > > > > think
> > > > > > > we'll
> > > > > > > add a module parameter for now.
> > > > > > 
> > > > > > Okay. So, I suppose we'll be able to update structs:
> > > > > > * Add a flag in virtio_video_format_desc that indicates whether
> > > > > > planes
> > > > > > can be in separate buffers, and
> > > > > > * Add a flag in virtio_video_format_desc that indicates that the
> > > > > > device requires contiguous buffers for this format.
> > > > > > 
> > > > > > Does it make sense?
> > > > > 
> > > > > Sorry, I don't understand the difference between the two above:
> > > > > isn't
> > > > > the
> > > > > first case is just when the flag is not set?
> > > > 
> > > > Ah, I was confused and wrote something strange. Yeah,  these two are
> > > > the
> > > > same. Sorry for that.
> > > > 
> > > > So, the suggestion is to add a field "planes_layout" in
> > > > virtio_video_format_desc, which is one of the following enums:
> > > > 
> > > > enum virtio_video_planes_layout {
> > > > 
> > > >     VIRTIO_VIDEO_PLANES_LAYOUT_UNSPEC = 0,  /* no special requirement
> > > >     */
> > > >     VIRTIO_VIDEO_PLANES_LAYOUT_CONTIGUOUS,
> > > > 
> > > > };
> > > > 
> > > > If we have a better idea or naming, please let me know.
> > > 
> > > The naming looks good for me, I might only change to CONTIG as we have
> > > UNSPEC.
> > 
> > So here we are talking about plane layout in memory, am I correct? But I
> > think we also need a way to communicate memory requirements of the
> > device: the device might require CMA buffers or it can be ok with SG
> > lists. What about adding something like this to virtio_video_format_desc:
> > 
> > enum virtio_video_mem_layout {
> > 
> >         VIRTIO_VIDEO_MEM_LAYOUT_UNDEFINED = 0,
> >         
> >         VIRTIO_VIDEO_MEM_LAYOUT_CONTIG = 0x100,
> >         VIRTIO_VIDEO_MEM_LAYOUT_NON_CONTIG,
> > 
> > };
> > 
> > struct virtio_video_format_desc {
> > 
> >     __le64 mask;
> >     __le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> >     __le32 planes_layout; /* One of VIRTIO_VIDEO_PLANES_LAYOUT_* types */
> >     __le32 mem_layout; /* One of VIRTIO_VIDEO_MEM_LAYOUT_* types */
> >     ...
> > 
> > };
> 
> Good.
> I'd not like to call it NON_CONTIG, as it sounds like CMA buffers
> aren't allowed.
> Instead, how about this definition?
> 
> enum virtio_video_mem_layout {
>         VIRTIO_VIDEO_MEM_LAYOUT_UNSPEC = 0,  /* no special requirement */
>         VIRTIO_VIDEO_MEM_LAYOUT_CONTIG = 1,
> };

Yes, I agree, that would be more correct. We just need an empty line to be 
aligned with other enums.

> 
> With this enum,
> * the device can simply ignore this field if it doesn't have any
> requirement and the struct is zero-initialized, and
> * if we need to add other types of memory layout requirements, we can
> add them as 2, 4, 8, etc to represent combinations of requirements.
> 
> Just to confirm, are the following combination of planes_layout and
> mem_layout valid?
> (1) (planes_layout, mem_layout) = (contig, not specified)
> (2) (planes_layout, mem_layout) = (not specified, contig)
> 
> In my understanding, (1) means that each plane must be a contiguous
> buffer while different planes don't have to be in a contig memory, but
> (2) is invalid.
> Is it correct?

Let me tell a bit more about my vision:

mem_layout: device can handle SG lists (e.g. using iommu) or it cannot and 
needs CMA. If it can handle SG, CMA is also ok. So it is ether 'I don't care' 
or 'give me CMA'.

planes_layout: some devices might want to see multiplanar data in one buffer. 
So if we allocate two sets of mem entries (one per each plane) and send them 
to the host, the device will fail to handle those.

So from my perspective you example can be interpreted as follows:
(1) means all your planes are hosted by one buffer instance (we can see it as 
one fd for all planes plus per-plane offsets inside the buffer), the buffer 
itself consists of arbitrary pages (or can consist, as it can of course also 
be CMA, because of UNSPEC).
(2) valid, means each plane has its own buffer, each buffer is contiguous in 
memory (must be allocated from CMA), but planes are not necessarily adjacent 
to each other in memory.

This also means that we cannot have unspec for planes layout. Device either 
expects planes in separate buffers or in one buffer with some offsets, there 
cannot be mixed cases. So it should look like this:

enum virtio_video_planes_layout {
    VIRTIO_VIDEO_PLANES_LAYOUT_UNSPEC = 0,  /* default, invalid */

    VIRTIO_VIDEO_PLANES_LAYOUT_CONTIG = 0x100,
    VIRTIO_VIDEO_PLANES_LAYOUT_NON_CONTIG,
};

Best regards,
Dmitry

> 
> Best regards,
> Keiichi
> 
> > Best regards,
> > Dmitry.
> > 
> > > Best regards,
> > > Dmitry.
> > > 
> > > > Best regards,
> > > > Keiichi
> > > > 
> > > > > Regards,
> > > > > Dmitry.
> > > > > 
> > > > > > Best regards,
> > > > > > Keiichi
> > > > > > 
> > > > > > > Regards,
> > > > > > > Dmitry.
> > > > > > > 
> > > > > > > > > For future V4L2 development we came up with the idea of a
> > > > > > > > > format
> > > > > > > > > flag
> > > > > > > > > which could mean that the hardware allows putting planes in
> > > > > > > > > separate
> > > > > > > > > buffers. We could have a similar per-format flag in the
> > > > > > > > > capabilities,
> > > > > > > > > as we already have a list of all the supported formats
> > > > > > > > > there.
> > > > > > > > 
> > > > > > > > Sorry, forgot to paste the link from future V4L2 work notes
> > > > > > > > from
> > > > > > > > this
> > > > > > > > year
> > > > > > > > ELCE: https://www.spinics.net/lists/linux-media/msg159789.html
> > > > > > > > 
> > > > > > > > > Best regards,
> > > > > > > > > Tomasz
> > > > > > > > > 
> > > > > > > > > > Best regards,
> > > > > > > > > > Keiichi
> > > > > > > > > > 
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Dmitry.
> > > > > > > > > > > 
> > > > > > > > > > > > * Renamed virtio_video_pin to virtio_video_buf_type.
> > > > > > > > > > > > 
> > > > > > > > > > > >   - It's similar to V4L2_BUF_TYPE_VIDEO_{OUTPUT,
> > > > > > > > > > > >   CAPTURE}.
> > > > > > > > > > > > 
> > > > > > > > > > > > * Added an error event.
> > > > > > > > > > > > * Reordered some subsections.
> > > > > > > > > > > > * Changed styles to make it consistent with other
> > > > > > > > > > > > devices.
> > > > > > > > > > > > 
> > > > > > > > > > > > Dmitry Sepp (1):
> > > > > > > > > > > >   virtio-video: Add virtio video device specification
> > > > > > > > > > > >  
> > > > > > > > > > > >  content.tex      |   1 +
> > > > > > > > > > > >  virtio-video.tex | 579
> > > > > > > > > > > >  +++++++++++++++++++++++++++++++++++++++++++++++
> > > > > > > > > > > >  2 files changed, 580 insertions(+)
> > > > > > > > > > > >  create mode 100644 virtio-video.tex
> > > > > > > > > > > > 
> > > > > > > > > > > > --
> > > > > > > > > > > > 2.24.1.735.g03f4e72817-goog
> > > 
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org



^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [virtio-dev] Re: [PATCH v2 0/1] VirtIO video device specification
  2020-01-10 15:11                       ` Dmitry Sepp
@ 2020-01-11 16:06                         ` Tomasz Figa
  2020-01-13  9:50                           ` Dmitry Sepp
  2020-01-13  9:56                         ` Gerd Hoffmann
  1 sibling, 1 reply; 73+ messages in thread
From: Tomasz Figa @ 2020-01-11 16:06 UTC (permalink / raw)
  To: Dmitry Sepp
  Cc: Keiichi Watanabe, virtio-dev, Linux Media Mailing List,
	Alexandre Courbot, Alex Lau, Daniel Vetter, Dylan Reid,
	Enrico Granata, Frediano Ziglio, Hans Verkuil, Gerd Hoffmann,
	Stéphane Marchesin, Pawel Osciak, spice-devel,
	David Stevens, uril

On Sat, Jan 11, 2020 at 12:12 AM Dmitry Sepp
<dmitry.sepp@opensynergy.com> wrote:
>
> Hi Keiichi,
>
> On Freitag, 10. Januar 2020 14:53:01 CET Keiichi Watanabe wrote:
> > Hi,
> >
> > On Fri, Jan 10, 2020 at 7:16 PM Dmitry Sepp <dmitry.sepp@opensynergy.com>
> wrote:
> > > Hi,
> > >
> > > On Donnerstag, 9. Januar 2020 15:56:08 CET Dmitry Sepp wrote:
> > > > Hi,
> > > >
> > > > On Dienstag, 7. Januar 2020 11:25:56 CET Keiichi Watanabe wrote:
> > > > > Hi Dmitry,
> > > > >
> > > > > On Mon, Jan 6, 2020 at 8:28 PM Dmitry Sepp
> > > > > <dmitry.sepp@opensynergy.com>
> > > >
> > > > wrote:
> > > > > > Hi,
> > > > > >
> > > > > > On Montag, 6. Januar 2020 11:30:22 CET Keiichi Watanabe wrote:
> > > > > > > Hi Dmitry, Tomasz,
> > > > > > >
> > > > > > > On Fri, Jan 3, 2020 at 10:05 PM Dmitry Sepp
> > > > > > > <dmitry.sepp@opensynergy.com>
> > > > > >
> > > > > > wrote:
> > > > > > > > Hi Tomasz, Keiichi,
> > > > > > > >
> > > > > > > > On Samstag, 21. Dezember 2019 07:19:23 CET Tomasz Figa wrote:
> > > > > > > > > On Sat, Dec 21, 2019 at 3:18 PM Tomasz Figa
> > > > > > > > > <tfiga@chromium.org>
> > > >
> > > > wrote:
> > &