[PATCH v3 1/2] virtio-video: Add virtio video device specification

[Keiichi Watanabe <keiichiw@chromium.org>]

From: Dmitry Sepp <dmitry.sepp@opensynergy.com>

The virtio video encoder device and decoder device provide functionalities to
encode and decode video stream respectively.
Though video encoder and decoder are provided as different devices, they use a
same protocol.

Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
---
 .gitignore                        |   1 +
 content.tex                       |   1 +
 images/video-buffer-lifecycle.dot |  18 +
 make-setup-generated.sh           |   8 +
 virtio-video.tex                  | 988 ++++++++++++++++++++++++++++++
 5 files changed, 1016 insertions(+)
 create mode 100644 .gitignore
 create mode 100644 images/video-buffer-lifecycle.dot
 create mode 100644 virtio-video.tex

diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..31272c2
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1 @@
+/images/generated/
diff --git a/content.tex b/content.tex
index b91a132..b75a40f 100644
--- a/content.tex
+++ b/content.tex
@@ -6062,6 +6062,7 @@ \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
 \input{virtio-fs.tex}
 \input{virtio-rpmb.tex}
 \input{virtio-iommu.tex}
+\input{virtio-video.tex}
 
 \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 
diff --git a/images/video-buffer-lifecycle.dot b/images/video-buffer-lifecycle.dot
new file mode 100644
index 0000000..98f379b
--- /dev/null
+++ b/images/video-buffer-lifecycle.dot
@@ -0,0 +1,18 @@
+digraph {
+  graph [ rankdir = LR, layout = dot ];
+
+  init [style = invis]
+  destroyed  [style = invis]
+  created [label="Created", shape=circle]
+  dequeued [label="Dequeued", shape=circle]
+  queued [label="Queued", shape=circle]
+
+  init -> created [label="RESOURCE_CREATE"]
+
+  created -> queued [label="RESOURCE_QUEUE is sent"]
+  dequeued -> queued [label="RESOURCE_QUEUE\n is sent"]
+  queued -> dequeued [label="RESOURCE_QUEUE\n has returned"]
+
+  created -> destroyed [label="RESOURCE_DESTROY_ALL"]
+  dequeued -> destroyed [label="RESOURCE_DESTROY_ALL"]
+}
diff --git a/make-setup-generated.sh b/make-setup-generated.sh
index f15d148..4caff72 100755
--- a/make-setup-generated.sh
+++ b/make-setup-generated.sh
@@ -61,3 +61,11 @@ cat > setup-generated.tex <<EOF
 \newcommand{\virtiodraftstagename}{$STAGENAME}
 \newcommand{\virtiodraftoasisstagename}{$OASISSTAGENAME}
 EOF
+
+# Generate PNG from DOT
+mkdir -p images/generated
+for file in images/*.dot
+do
+    BASENAME=`basename "$file" .dot`
+    dot -Tpng -o images/generated/${BASENAME}.png ${file}
+done
diff --git a/virtio-video.tex b/virtio-video.tex
new file mode 100644
index 0000000..2eeee53
--- /dev/null
+++ b/virtio-video.tex
@@ -0,0 +1,988 @@
+\section{Video Device}\label{sec:Device Types / Video Device}
+
+The virtio video encoder device and decoder device are virtual devices
+that supports encoding and decoding respectively. While the encoder
+and the decoder are different devices, they use the same protocol.
+
+\subsection{Device ID}
+\label{sec:Device Types / Video Device / Device ID}
+
+\begin{description}
+\item[30] encoder device
+\item[31] decoder device
+\end{description}
+
+\subsection{Virtqueues}
+\label{sec:Device Types / Video Device / Virtqueues}
+
+\begin{description}
+\item[0] commandq - queue for sending commands.
+\item[1] eventq - queue for sending events happened in the device.
+\end{description}
+
+\subsection{Feature bits}
+\label{sec:Device Types / Video Device / Feature bits}
+
+\begin{description}
+\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used
+  for video buffers.
+\item[VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG (1)] The device can use
+  non-contiguous memories for video buffers. Without this flag, the
+  driver and device MUST use video buffers that are contiguous in the
+  device-side.
+\end{description}
+
+\devicenormative{\subsubsection}{Feature bits}{Device Types / Video
+  Device / Feature bits}
+
+The device MUST offer VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES.
+
+\subsection{Device configuration layout}
+\label{sec:Device Types / Video Device / Device configuration layout}
+
+Video device configuration uses the following layout structure:
+
+\begin{lstlisting}
+struct virtio_video_config {
+        le32 version;
+        le32 max_caps_length;
+        le32 max_resp_length;
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{version}] is the protocol version that the device talks.
+  The device MUST set this to 0.
+\item[\field{max_caps_length}] defines the maximum length of a
+  descriptor required to call VIRTIO_VIDEO_CMD_QUERY_CAPABILITY in
+  bytes. The device MUST set this value.
+\item[\field{max_resp_length}] defines the maximum length of a
+  descriptor required to call a command other than
+  VIRTIO_VIDEO_CMD_QUERY_CAPABILITY in bytes. The device MUST set this
+  value.
+\end{description}
+
+\subsection{Device Initialization}
+\label{sec:Device Types / Video Device / Device Initialization}
+
+\devicenormative{\subsubsection}{Device Initialization}{Device Types /
+  Video Device / Device Initialization}
+
+The driver SHOULD query device capability by using the
+VIRTIO_VIDEO_CMD_QUERY_CAPABILITY and use that information for the
+initial setup.
+
+\subsection{Device Operation}
+\label{sec:Device Types / Video Device / Device Operation}
+
+The driver allocates input and output buffers and queues the buffers
+to the device. The device performs operations on the buffers according
+to the function in question.
+
+\subsubsection{Command Virtqueue}
+
+\paragraph{Device Operation: Command header}
+
+All commands and responses on the command virtqueue have a fixed
+header using the following layout structure and definitions:
+
+\begin{lstlisting}
+  enum virtio_video_cmd_type {
+        /* Command */
+        VIRTIO_VIDEO_CMD_QUERY_CAPABILITY = 0x0100,
+        VIRTIO_VIDEO_CMD_STREAM_CREATE,
+        VIRTIO_VIDEO_CMD_STREAM_DESTROY,
+        VIRTIO_VIDEO_CMD_STREAM_DRAIN,
+        VIRTIO_VIDEO_CMD_RESOURCE_CREATE,
+        VIRTIO_VIDEO_CMD_RESOURCE_QUEUE,
+        VIRTIO_VIDEO_CMD_RESOURCE_DESTROY_ALL,
+        VIRTIO_VIDEO_CMD_QUEUE_CLEAR,
+        VIRTIO_VIDEO_CMD_GET_PARAMS,
+        VIRTIO_VIDEO_CMD_SET_PARAMS,
+        VIRTIO_VIDEO_CMD_QUERY_CONTROL,
+        VIRTIO_VIDEO_CMD_GET_CONTROL,
+        VIRTIO_VIDEO_CMD_SET_CONTROL,
+
+        /* Response */
+        VIRTIO_VIDEO_RESP_OK_NODATA = 0x0200,
+        VIRTIO_VIDEO_RESP_OK_QUERY_CAPABILITY,
+        VIRTIO_VIDEO_RESP_OK_RESOURCE_QUEUE,
+        VIRTIO_VIDEO_RESP_OK_GET_PARAMS,
+        VIRTIO_VIDEO_RESP_OK_QUERY_CONTROL,
+        VIRTIO_VIDEO_RESP_OK_GET_CONTROL,
+
+        VIRTIO_VIDEO_RESP_ERR_INVALID_OPERATION = 0x0300,
+        VIRTIO_VIDEO_RESP_ERR_OUT_OF_MEMORY,
+        VIRTIO_VIDEO_RESP_ERR_INVALID_STREAM_ID,
+        VIRTIO_VIDEO_RESP_ERR_INVALID_RESOURCE_ID,
+        VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER,
+        VIRTIO_VIDEO_RESP_ERR_UNSUPPORTED_CONTROL,
+};
+
+struct virtio_video_cmd_hdr {
+        le32 type;
+        le32 stream_id;
+};
+\end{lstlisting}
+
+The fixed header \field{virtio_video_cmd_hdr} in each message includes
+the following field:
+\begin{description}
+\item[\field{type}] specifies the type of the driver command
+  (VIRTIO_VIDEO_CMD_*) or the device response (VIRTIO_VIDEO_RESP_*).
+\item[\field{stream_id}] specifies a target stream if needed.
+\end{description}
+
+On success the device will return VIRTIO_VIDEO_RESP_OK_NODATA in case
+there is no payload. Otherwise the \field{type} field will indicate
+the kind of payload.
+
+On error the device will return one of the VIRTIO_VIDEO_RESP_ERR_*
+error codes.
+
+\paragraph{Device Operation: Query device capability}
+
+\begin{description}
+\item[VIRTIO_VIDEO_CMD_QUERY_CAPABILITY] Retrieve information about
+supported formats.
+
+The driver uses \field{virtio_video_query_capability} to send a
+query request.
+
+\begin{lstlisting}
+enum virtio_video_queue_type {
+        VIRTIO_VIDEO_QUEUE_TYPE_INPUT = 0x100,
+        VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT,
+};
+
+struct virtio_video_query_capability {
+        struct virtio_video_ctrl_hdr hdr;
+        le32 queue_type;
+        u8 padding[4];
+};
+\end{lstlisting}
+\begin{description}
+\item[\field{queue_type}] is the queue type that the driver asks
+information about. The driver MUST set either
+\field{VIRTIO_VIDEO_QUEUE_TYPE_INPUT} or
+\field{VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT}.
+\end{description}
+
+The device responds to VIRTIO_VIDEO_CMD_QUERY_CAPABILITY with
+\field{virtio_video_query_capability_resp}.
+\begin{lstlisting}
+struct virtio_video_query_capability_resp {
+        struct virtio_video_cmd_hdr hdr;
+        le32 num_descs;
+        u8 padding[4];
+        /* Followed by struct virtio_video_format_desc descs[] */
+};
+\end{lstlisting}
+
+The device MUST return its capability with \field{
+  virtio_video_capability_resp} that includes the following fields:
+\begin{description}
+\item[\field{num_descs}] is a number of \field{virtio_video_format_desc}
+  that follow. The value MUST not exceed 64.
+\end{description}
+
+The format description \field{virtio_video_format_desc} is defined as
+follows:
+\begin{lstlisting}
+enum virtio_video_format {
+        /* Raw formats */
+        VIRTIO_VIDEO_FORMAT_RAW_MIN = 1,
+        VIRTIO_VIDEO_FORMAT_ARGB8888 = VIRTIO_VIDEO_FORMAT_RAW_MIN,
+        VIRTIO_VIDEO_FORMAT_BGRA8888,
+        VIRTIO_VIDEO_FORMAT_NV12, /* 12  Y/CbCr 4:2:0  */
+        VIRTIO_VIDEO_FORMAT_YUV420, /* 12  YUV 4:2:0     */
+        VIRTIO_VIDEO_FORMAT_YVU420, /* 12  YVU 4:2:0     */
+        VIRTIO_VIDEO_FORMAT_RAW_MAX = VIRTIO_VIDEO_FORMAT_YVU420,
+
+        /* Coded formats */
+        VIRTIO_VIDEO_FORMAT_CODED_MIN = 0x1000,
+        VIRTIO_VIDEO_FORMAT_MPEG2 = VIRTIO_VIDEO_FORMAT_CODED_MIN, /* MPEG-2 Part 2 */
+        VIRTIO_VIDEO_FORMAT_MPEG4, /* MPEG-4 Part 2 */
+        VIRTIO_VIDEO_FORMAT_H264, /* H.264 */
+        VIRTIO_VIDEO_FORMAT_HEVC, /* HEVC aka H.265*/
+        VIRTIO_VIDEO_FORMAT_VP8, /* VP8 */
+        VIRTIO_VIDEO_FORMAT_VP9, /* VP9 */
+        VIRTIO_VIDEO_FORMAT_CODED_MAX = VIRTIO_VIDEO_FORMAT_VP9,
+};
+
+enum virtio_video_planes_layout_flag {
+        VIRTIO_VIDEO_PLANES_LAYOUT_SINGLE_BUFFER = 1 << 0,
+        VIRTIO_VIDEO_PLANES_LAYOUT_PER_PLANE = 1 << 1,
+};
+
+struct virtio_video_format_desc {
+        le64 mask;
+        le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
+        le32 planes_layout; /* Bitmask with VIRTIO_VIDEO_PLANES_LAYOUT_* */
+        le32 plane_align;
+        le32 num_frames;
+        /* Followed by struct virtio_video_format_frame frames[] */
+};
+\end{lstlisting}
+\begin{description}
+\item[\field{mask}] is a bitset that represents the supported
+  combination of input and output format. If \textit{i}-th bit is set
+  in \field{mask} of \textit{j}-th \field{virtio_video_format_desc}
+  for input, the device supports encoding or decoding from the
+  \textit{j}-th input format to \textit{i}-th output format.
+\item[\field{format}] specifies an image format. The device MUST set
+  one of \field{enum virtio_video_format}.
+\item[\field{planes_layout}] is a bitmask representing a set of plane
+  layout types the device supports. This driver MUST ignore this field
+  for encoded formats.
+  \begin{description}
+  \item[\field{VIRTIO_VIDEO_PLANES_LAYOUT_SINGLE_BUFFER}] The device
+    expects planes in one buffer laid out one after another.
+  \item[\field{VIRTIO_VIDEO_PLANES_LAYOUT_PER_PLANE}] The device
+    expects planes to be located in separate buffers.
+  \end{description}
+\item[\field{plane_align}] is a plane alignment the device require
+  when multiple planes are located in one buffer. The driver MUST
+  ignore this field if \field{format} is a bitstream format or
+  \field{planes_layout} doesn't have
+  \field{VIRTIO_VIDEO_PLANES_LAYOUT_SINGLE_BUFFER} bit.
+\item[\field{num_frames}] is the number of
+  \field{virtio_video_format_frame} that follows.
+\end{description}
+
+The frame information \field{virtio_video_format_frame} is defined as
+follows:
+\begin{lstlisting}
+struct virtio_video_format_range {
+        le32 min;
+        le32 max;
+        le32 step;
+        u8 padding[4];
+};
+
+struct virtio_video_format_frame {
+        struct virtio_video_format_range width;
+        struct virtio_video_format_range height;
+        le32 num_rates;
+        u8 padding[4];
+        /* Followed by struct virtio_video_format_range frame_rates[] */
+};
+\end{lstlisting}
+\begin{description}
+\item[\field{width, height}] represents a range of resolutions
+  supported by the device. If its \field{step} is not applicable, its
+  \field{min} is equal to its \field{max}.
+\item[\field{num_rates}] is the number of
+  \field{virtio_video_format_ranges} that follows to represent
+  supported frame rates.
+\end{description}
+\end{description}
+
+\devicenormative{\subparagraph}{Device Operation: Query device
+  capability}{Device Types / Video Device / Device Operation / Device
+  Operation: Query device capability}
+
+The total size of the device response MUST not exceed
+\field{max_cap_length} in bytes reported in the device configuration.
+
+\paragraph{Device Operation: Create streams}
+
+To process buffers, the device needs to associate them with a certain
+video stream (essentially, a context). Streams are created by
+VIRTIO_VIDEO_CMD_STREAM_CREATE with a default set of parameters
+determined by the device.
+
+\begin{description}
+\item[VIRTIO_VIDEO_CMD_STREAM_CREATE] Create a video stream (context)
+  within the device.
+
+\begin{lstlisting}
+enum virtio_video_mem_type {
+        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
+};
+
+struct virtio_video_stream_create {
+        struct virtio_video_cmd_hdr hdr;
+        le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
+        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
+        le32 coded_format; /* One of VIRTIO_VIDEO_FORMAT_* types */
+        u8 padding[4];
+        u8 tag[64];
+};
+\end{lstlisting}
+\begin{description}
+\item[\field{in_mem_type, out_mem_type}] is a type of buffer
+  management for input /output buffers. The driver MUST set a value in
+  \field{enum virtio_video_mem_type} that the device reported a
+  corresponding feature bit.
+\begin{description}
+\item[\field{VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES}] Use guest pages.
+\end{description}
+\item[\field{coded_format}] is the encoded format that will be
+  processed.
+\item[\field{tag}] is the name associated with this stream. The tag
+  MUST be encoded in UTF-8 and NUL-terminated.
+\end{description}
+
+The driver MUST set \field{stream_id} in \field{virtio_video_cmd_hdr}
+to an integer that is not used before. If a used value is passed as
+\field{stream_id}, the device MUST reports an error with
+VIRTIO_VIDEO_RESP_ERR_INVALID_STREAM_ID.
+
+\item[VIRTIO_VIDEO_CMD_STREAM_DESTROY] Destroy a video stream
+  (context) within the device.
+
+\begin{lstlisting}
+struct virtio_video_stream_destroy {
+        struct virtio_video_cmd_hdr hdr;
+};
+\end{lstlisting}
+\end{description}
+
+\paragraph{Device Operation: Import resources}
+
+\begin{itemize*}
+\item Use VIRTIO_VIDEO_CMD_RESOURCE_CREATE to import a resource to use
+  it as a video buffer.
+\item Use VIRTIO_VIDEO_CMD_RESOURCE_DESTROY_ALL to invalidate all the
+  resources imported by VIRTIO_VIDEO_CMD_RESOURCE_CREATE so far.
+\end{itemize*}
+
+\begin{description}
+\item[VIRTIO_VIDEO_CMD_RESOURCE_CREATE] Create a resource descriptor
+  within the device.
+
+The driver sends \field{virtio_video_resource_create} defined as
+follows:
+\begin{lstlisting}
+#define VIRTIO_VIDEO_MAX_PLANES 8
+
+struct virtio_video_resource_create {
+        struct virtio_video_cmd_hdr hdr;
+        le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
+        le32 resource_id;
+        le32 planes_layout;
+        le32 num_planes;
+        le32 plane_offsets[VIRTIO_VIDEO_MAX_PLANES];
+        le32 num_entries[VIRTIO_VIDEO_MAX_PLANES];
+        /* Followed by struct virtio_video_mem_entry entries[] */
+};
+\end{lstlisting}
+\begin{description}
+\item[\field{queue_type}] specifies a direction of queue which the
+  resource will be used for.
+\item[\field{resource_id}] is an identifier of the resource. The
+  driver MUST set it to an integer that is not used before for the
+  given \field{queue_type}. If the specified \field{resource_id} is
+  already in use, the device MUST report an error with
+  VIRTIO_VIDEO_RESP_ERR_INVALID_RESOURCE_ID.
+\item[\field{planes_layout}] specifies a plane layout. The driver MUST
+  set only one bit of \field{virtio_video_planes_layout_flag} that is
+  supported for a current raw format.
+\item[\field{num_planes}] specifies a number of planes.
+\item[\field{plane_offsets}] specifies offsets for each plane.
+\item[\field{num_entries}] is an array of numbers of \field{entries}
+  memory entries for each plane. If \field{planes_layout} is
+  VIRTIO_VIDEO_PLANES_LAYOUT_SINGLE_BUFFER, only the first element is
+  used.
+\end{description}
+
+The \field{virtio_video_resource_create} is followed by an array of
+\field{virtio_video_mem_entry} defined as follows:
+\begin{lstlisting}
+struct virtio_video_mem_entry {
+        le64 addr;
+        le32 length;
+        u8 padding[4];
+};
+\end{lstlisting}
+\begin{description}
+\item[\field{addr}] is the physical guest address.
+\item[\field{length}] is the length of the resource.
+\end{description}
+The number of \field{virtio_video_mem_entry} MUST be equal to the sum
+of integers in the array \field{num_entries}.
+
+\item[VIRTIO_VIDEO_CMD_RESOURCE_DESTROY_ALL] Invalidate all the
+  resource descriptor created so far.
+\begin{lstlisting}
+struct virtio_video_resource_destroy_all {
+        struct virtio_video_cmd_hdr hdr;
+        le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
+        u8 padding[4];
+};
+\end{lstlisting}
+\begin{description}
+\item[\field{queue_type}] is a type of queue.
+\end{description}
+\end{description}
+
+\paragraph{Device Operation: Process buffers}
+
+\begin{itemize*}
+\item Use VIRTIO_VIDEO_CMD_RESOURCE_QUEUE to queue the resource for
+  processing in the device. The request completes asynchronously and
+  out-of-order when the device has finished with the buffer.
+\item Use VIRTIO_VIDEO_CMD_STREAM_DRAIN to ask the device to process
+  and return all of the already queued buffers.
+\item Use VIRTIO_VIDEO_CMD_QUEUE_CLEAR to ask the device to return
+  back already queued buffers from the input or the output queue. This
+  also includes input or output buffers that can be currently owned by
+  the device's processing pipeline.
+\end{itemize*}
+
+\begin{description}
+\item[VIRTIO_VIDEO_CMD_RESOURCE_QUEUE] Add a buffer to the device's
+queue.
+
+\begin{lstlisting}
+struct virtio_video_resource_queue {
+        struct virtio_video_cmd_hdr hdr;
+        le32 queue_type;
+        le32 resource_id;
+        le64 timestamp;
+        le32 num_data_sizes;
+        le32 data_sizes[VIRTIO_VIDEO_MAX_PLANES];
+        u8 padding[4];
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{resource_id}] is the ID of the resource to be queued.
+\item[\field{timestamp}] is an abstract sequence counter that can be
+  used for synchronization. For an input buffer, the driver MUST set a
+  \field{timestamp}.
+\item[\field{num_data_sizes}] is the number of \field{data_sizes}
+  entries in use.
+\item[\field{data_sizes}] number of data bytes within a plane.
+\end{description}
+
+The device returns \field{virtio_video_resource_queue_resp} defined as
+follows:
+\begin{lstlisting}
+enum virtio_video_buffer_flag {
+        VIRTIO_VIDEO_BUFFER_FLAG_ERR = 0x0001,
+        VIRTIO_VIDEO_BUFFER_FLAG_EOS = 0x0002,
+
+        /* Encoder only */
+        VIRTIO_VIDEO_BUFFER_FLAG_IFRAME = 0x0004,
+        VIRTIO_VIDEO_BUFFER_FLAG_PFRAME = 0x0008,
+        VIRTIO_VIDEO_BUFFER_FLAG_BFRAME = 0x0010,
+};
+
+struct virtio_video_resource_queue_resp {
+        struct virtio_video_cmd_hdr hdr;
+        le64 timestamp;
+        le32 flags;
+        le32 size; /* Encoded size */
+};
+\end{lstlisting}
+\begin{description}
+\item[\field{timestamp}] is an abstract sequence counter that can be
+  used for synchronization. For an output buffer, the device MUST copy
+  the \field{timestamp} of the input buffer this output buffer was
+  produced from.
+\item[\field{flags}] marks specific buffers in the sequence with
+  VIRTIO_VIDEO_BUFFER_FLAG_* flags.
+\item[\field{size}] is the data size in the buffer (encoder only).
+\end{description}
+
+\begin{itemize*}
+\item For each VIRTIO_VIDEO_CMD_RESOURCE_QUEUE request, the device
+  MUST send a response to the queue request with
+  VIRTIO_VIDEO_OK_NODATA when it has finished processing the buffer
+  successfully.
+\item The device MUST mark a buffer that triggered a processing error
+  with the VIRTIO_VIDEO_BUFFER_F_ERR flag.
+\item The device MUST mark the last buffer with the
+  VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
+  sequence.
+\item In case of encoder, to denote a particular frame type the device
+  MUST mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
+  VIRTIO_VIDEO_BUFFER_PFRAME, or VIRTIO_VIDEO_BUFFER_BFRAME.
+\item If the processing was stopped due to
+  VIRTIO_VIDEO_CMD_QUEUE_CLEAR, the device MUST respond with
+  VIRTIO_VIDEO_RESP_OK_NODATA as a response type and
+  VIRTIO_VIDEO_BUFFER_FLAG_ERR in \field{flags}.
+\item The driver and device MUST follow requirements about buffer
+  ownership explained in \ref{sec:Device Types / Video Device / Device
+    Operation / Buffer lifecycle}.
+\end{itemize*}
+
+\item[VIRTIO_VIDEO_CMD_STREAM_DRAIN] Ask the device to push all the
+  queued buffers through the pipeline.
+
+\begin{lstlisting}
+struct virtio_video_stream_drain {
+        struct virtio_video_cmd_hdr hdr;
+};
+\end{lstlisting}
+
+\begin{itemize*}
+\item While the device is processing a VIRTIO_VIDEO_CMD_STREAM_DRAIN
+  command, the device MUST return
+  VIRTIO_VIDEO_RESP_ERR_INVALID_OPERATION for incoming commands of
+  VIRTIO_VIDEO_CMD_RESOURCE_QUEUE for input buffers and
+  VIRTIO_VIDEO_CMD_STREAM_DRAIN.
+\item If the processing was stopped due to
+  VIRTIO_VIDEO_CMD_QUEUE_CLEAR, the device MUST respond with
+  VIRTIO_VIDEO_RESP_OK_NODATA as a response type and
+  VIRTIO_VIDEO_BUFFER_FLAG_ERR in \field{flags}.
+\end{itemize*}
+
+\item[VIRTIO_VIDEO_CMD_QUEUE_CLEAR] Return already queued
+  buffers back from the input or the output queue of the device. The
+  device SHOULD return all of the buffers from the respective queue as
+  soon as possible without pushing the buffers through the processing
+  pipeline.
+
+\begin{lstlisting}
+struct virtio_video_resource_queue_clear {
+        struct virtio_video_cmd_hdr hdr;
+        le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
+        u8 padding[4];
+};
+\end{lstlisting}
+\begin{description}
+\item[\field{queue_type}] queue type.
+\end{description}
+
+While the device is processing a VIRTIO_VIDEO_CMD_QUEUE_CLEAR command,
+the device MUST return VIRTIO_VIDEO_RESP_ERR_INVALID_OPERATION for
+incoming commands of VIRTIO_VIDEO_CMD_RESOURCE_QUEUE,
+VIRTIO_VIDEO_CMD_STREAM_DRAIN, and VIRTIO_VIDEO_CMD_QUEUE_CLEAR.
+
+\end{description}
+
+\paragraph{Device Operation: Handle stream parameters}
+
+\begin{itemize*}
+\item Use VIRTIO_VIDEO_CMD_GET_PARAMS to get the current stream
+  parameters for input and output streams from the device.
+\item Use VIRTIO_VIDEO_CMD_SET_PARAMS to provide new stream parameters
+  to the device.
+\item After setting stream parameters, the driver may issue
+  VIRTIO_VIDEO_CMD_GET_PARAMS as some parameters of both input and
+  output can be changed implicitly by the device during the set
+  operation.
+\end{itemize*}
+
+\begin{description}
+\item[VIRTIO_VIDEO_CMD_GET_PARAMS] Get parameters of the input or the
+  output of a stream.
+
+\begin{lstlisting}
+struct virtio_video_plane_format {
+        le32 plane_size;
+        le32 stride;
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{plane_size}] size of the plane in bytes.
+\item[\field{stride}] stride used for the plane in bytes.
+\end{description}
+
+\begin{lstlisting}
+struct virtio_video_crop {
+        le32 left;
+        le32 top;
+        le32 width;
+        le32 height;
+};
+
+struct virtio_video_params {
+        le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
+        le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
+        le32 frame_width;
+        le32 frame_height;
+        le32 min_buffers;
+        le32 max_buffers;
+        struct virtio_video_crop crop;
+        le32 frame_rate;
+        le32 num_planes;
+        struct virtio_video_plane_format plane_formats[VIRTIO_VIDEO_MAX_PLANES];
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{frame_width}] the value to get/set.
+\item[\field{frame_height}] the value to get/set.
+\item[\field{pixel_format}] the value to get/set.
+\item[\field{min_buffers}] minimum buffers required to handle the
+  format (r/o).
+\item[\field{max_buffers}] maximum buffers required to handle the
+  format (r/o).
+\item[\field{crop}] cropping (composing) rectangle.
+\item[\field{frame_rate}] the value to get/set.
+\item[\field{num_planes}] number of planes used to store pixel data
+(r/o).
+\item[\field{plane_formats}] description of each plane.
+\end{description}
+
+\begin{lstlisting}
+struct virtio_video_get_params {
+        struct virtio_video_cmd_hdr hdr;
+        le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
+        u8 padding[4];
+};
+
+struct virtio_video_get_params_resp {
+        struct virtio_video_cmd_hdr hdr;
+        struct virtio_video_params params;
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{queue_type}] queue type.
+\item[\field{params}] parameter values.
+\end{description}
+
+\item[VIRTIO_VIDEO_CMD_SET_PARAMS] Change parameters of a stream.
+
+\begin{lstlisting}
+struct virtio_video_set_params {
+        struct virtio_video_cmd_hdr hdr;
+        struct virtio_video_params params;
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{params}] parameters to set.
+\end{description}
+
+Setting stream parameters might have side effects within the device.
+For example, the device MAY perform alignment of width and height,
+change the number of planes it uses for the format, or do whatever
+changes that are required to continue normal operation using the
+updated parameters. It is up to the driver to check the parameter set
+after the VIRTIO_VIDEO_CMD_SET_PARAMS request has been issued.
+\end{description}
+
+\paragraph{Device Operation: Handle control values}
+
+The driver can query, get and set control values. Though control
+values are associated with a video stream like stream parameters,
+supported values differ depending on the device capabilities and
+formats of videos being processed.
+
+\subparagraph{Commands for controls}
+
+\begin{description}
+\item[VIRTIO_VIDEO_CMD_QUERY_CONTROL] Query supported control values.
+
+\begin{lstlisting}
+struct virtio_video_query_control {
+        struct virtio_video_cmd_hdr hdr;
+        le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
+        u8 padding[4];
+};
+
+struct virtio_video_query_control_resp {
+        struct virtio_video_cmd_hdr hdr;
+        /* Followed by one of struct virtio_video_query_control_resp_* */
+};
+\end{lstlisting}
+
+If the device supports a given control, the device MUST return a
+struct that indicates supported control values after sending
+\field{virtio_video_query_control_resp}. The struct used as the
+response differs depending on the value of requested \field{control}.
+Otherwise, the device MUST report an error with
+VIRTIO_VIDEO_RESP_UNSUPPORTED_CONTROL.
+
+\item[VIRTIO_VIDEO_CMD_GET_CONTROL] Get a control value.
+
+\begin{lstlisting}
+struct virtio_video_get_control {
+        struct virtio_video_cmd_hdr hdr;
+        le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
+        u8 padding[4];
+};
+
+struct virtio_video_get_control_resp {
+        struct virtio_video_cmd_hdr hdr;
+        /* Followed by one of struct virtio_video_control_val_* */
+};
+\end{lstlisting}
+
+If the device supports a given control and a control value is
+available, the device MUST return a control value after
+\field{virtio_video_get_control_resp}.
+The struct used as the response differs depending on the value of
+requested as \field{control}. If the given control is unsupported, the
+device MUST report an error with
+VIRTIO_VIDEO_RESP_UNSUPPORTED_CONTROL.
+
+\item[VIRTIO_VIDEO_CMD_SET_CONTROL] Set a control value.
+
+\begin{lstlisting}
+struct virtio_video_set_control {
+        struct virtio_video_cmd_hdr hdr;
+        le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
+        u8 padding[4];
+        /* Followed by one of struct virtio_video_control_val_* */
+};
+
+struct virtio_video_set_control_resp {
+        struct virtio_video_cmd_hdr hdr;
+};
+\end{lstlisting}
+
+The driver MUST set \field{control} in
+\field{virtio_video_set_control} and send a corresponding struct after
+it. If the given control is unsupported, the device MUST report an
+error with VIRTIO_VIDEO_RESP_UNSUPPORTED_CONTROL.
+\end{description}
+
+\subparagraph{Types of controls}
+
+The types of controls are defined as follows:
+\begin{lstlisting}
+enum virtio_video_control_type {
+        VIRTIO_VIDEO_CONTROL_BITRATE = 1,
+        VIRTIO_VIDEO_CONTROL_PROFILE,
+        VIRTIO_VIDEO_CONTROL_LEVEL,
+};
+\end{lstlisting}
+
+\begin{description}
+\item[VIRTIO_VIDEO_CONTROL_BITRATE] Bitrate of video. (Only for the
+  encoder device.)
+
+The driver can VIRTIO_VIDEO_CMD_GET_CONTROL and
+VIRTIO_VIDEO_CMD_SET_CONTROL to get and set a bitrate for encoding
+respectively.
+The following \field{virtio_video_control_val_bitrate} is used when
+the device returns a value as a response of
+VIRTIO_VIDEO_CMD_GET_CONTROL and when the driver specifies a value to
+be set by VIRTIO_VIDEO_CMD_SET_CONTROL.
+\begin{lstlisting}
+struct virtio_video_control_val_bitrate {
+        le32 bitrate;
+        u8 padding[4];
+};
+\end{lstlisting}
+
+\item[VIRTIO_VIDEO_CONTROL_PROFILE] Profile of a compressed video
+  stream.
+
+VIRTIO_VIDEO_CONTROL_PROFILE is used to handle profiles, which
+represent sets of cabalities for several compressed formats such as
+H.264 and VP9. The enum representing profiles are defined as follows:
+\begin{lstlisting}
+enum virtio_video_profile {
+        /* H.264 */
+        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
+        VIRTIO_VIDEO_PROFILE_H264_BASELINE = VIRTIO_VIDEO_PROFILE_H264_MIN,
+        VIRTIO_VIDEO_PROFILE_H264_MAIN,
+        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
+        VIRTIO_VIDEO_PROFILE_H264_HIGH,
+        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
+        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
+        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
+        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
+        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
+        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
+        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
+        VIRTIO_VIDEO_PROFILE_H264_MAX = VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
+
+        /* HEVC */
+        VIRTIO_VIDEO_PROFILE_HEVC_MIN = 0x200,
+        VIRTIO_VIDEO_PROFILE_HEVC_MAIN = VIRTIO_VIDEO_PROFILE_HEVC_MIN,
+        VIRTIO_VIDEO_PROFILE_HEVC_MAIN10,
+        VIRTIO_VIDEO_PROFILE_HEVC_MAIN_STILL_PICTURE,
+        VIRTIO_VIDEO_PROFILE_HEVC_MAX =
+                VIRTIO_VIDEO_PROFILE_HEVC_MAIN_STILL_PICTURE,
+
+        /* VP8 */
+        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x300,
+        VIRTIO_VIDEO_PROFILE_VP8_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP8_MIN,
+        VIRTIO_VIDEO_PROFILE_VP8_PROFILE1,
+        VIRTIO_VIDEO_PROFILE_VP8_PROFILE2,
+        VIRTIO_VIDEO_PROFILE_VP8_PROFILE3,
+        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_PROFILE3,
+
+        /* VP9 */
+        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x400,
+        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
+        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
+        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
+        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
+        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
+};
+\end{lstlisting}
+
+The driver can query supported profiles for a compressed format by
+VIRTIO_VIDEO_CMD_QUERY_CONTROL command with
+\field{virtio_video_query_control_profile}. If the device supports the
+given format and the format's profiles are listed in \field{enum
+  virtio_video_profile}, the device MUST returns a list of supported
+profiles as a response
+\field{virtio_video_query_control_resp_profile}.
+\begin{lstlisting}
+struct virtio_video_query_control_profile {
+        le32 format; /* One of VIRTIO_VIDEO_FORMAT_* */
+        u8 padding[4];
+};
+
+struct virtio_video_query_control_resp_profile {
+        le32 num;
+        u8 padding[4];
+        /* Followed by an array le32 profiles[] */
+};
+\end{lstlisting}
+
+The device and the driver use \field{virtio_video_control_val_profile}
+for VIRTIO_VIDEO_CMD_GET_CONTROL and VIRTIO_VIDEO_CMD_SET_CONTROL.
+\begin{lstlisting}
+struct virtio_video_control_val_profile {
+        le32 profile;
+        u8 padding[4];
+};
+\end{lstlisting}
+
+\item[VIRTIO_VIDEO_CONTROL_LEVEL] Level of a compressed video stream.
+
+VIRTIO_VIDEO_CONTROL_LEVEL is used to handle levels of video stream.
+Levels are defined for some compressed video formats to specify sets
+of constraints that indicate a degree of required decoder performance.
+The enum representing levels are defined as follows:
+\begin{lstlisting}
+enum virtio_video_level {
+        /* H.264 */
+        VIRTIO_VIDEO_LEVEL_H264_MIN = 0x100,
+        VIRTIO_VIDEO_LEVEL_H264_1_0 = VIRTIO_VIDEO_LEVEL_H264_MIN,
+        VIRTIO_VIDEO_LEVEL_H264_1_1,
+        VIRTIO_VIDEO_LEVEL_H264_1_2,
+        VIRTIO_VIDEO_LEVEL_H264_1_3,
+        VIRTIO_VIDEO_LEVEL_H264_2_0,
+        VIRTIO_VIDEO_LEVEL_H264_2_1,
+        VIRTIO_VIDEO_LEVEL_H264_2_2,
+        VIRTIO_VIDEO_LEVEL_H264_3_0,
+        VIRTIO_VIDEO_LEVEL_H264_3_1,
+        VIRTIO_VIDEO_LEVEL_H264_3_2,
+        VIRTIO_VIDEO_LEVEL_H264_4_0,
+        VIRTIO_VIDEO_LEVEL_H264_4_1,
+        VIRTIO_VIDEO_LEVEL_H264_4_2,
+        VIRTIO_VIDEO_LEVEL_H264_5_0,
+        VIRTIO_VIDEO_LEVEL_H264_5_1,
+        VIRTIO_VIDEO_LEVEL_H264_MAX = VIRTIO_VIDEO_LEVEL_H264_5_1,
+};
+\end{lstlisting}
+
+The driver can query supported levels for a compressed format by
+VIRTIO_VIDEO_CMD_QUERY_CONTROL command with
+\field{virtio_video_query_control_level}. If the device supports the
+given format and the format's levels are listed in \field{enum
+  virtio_video_level}, the device MUST return a list of supported
+levels as a response \field{virtio_video_query_control_resp_level}.
+\begin{lstlisting}
+struct virtio_video_query_control_level {
+        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* */
+        u8 padding[4];
+};
+
+struct virtio_video_query_control_resp_level {
+        le32 num;
+        u8 padding[4];
+        /* Followed by an array le32 level[] */
+};
+\end{lstlisting}
+
+The device and the driver use \field{virtio_video_control_val_level}
+for VIRTIO_VIDEO_CMD_GET_CONTROL and VIRTIO_VIDEO_CMD_SET_CONTROL.
+\begin{lstlisting}
+struct virtio_video_control_val_level {
+        le32 level;
+        u8 padding[4];
+};
+\end{lstlisting}
+\end{description}
+
+\subsubsection{Buffer lifecycle}
+\label{sec:Device Types / Video Device / Device Operation / Buffer
+  lifecycle}
+
+The state machine in Figure~\ref{fig:buffer-lifecycle} shows the life
+cycle of a video buffer.
+
+\begin{figure}[h]
+  \centering
+  \includegraphics[width=\textwidth]{images/generated/video-buffer-lifecycle.png}
+  \caption{Lifecycle of a buffer}
+  \label{fig:buffer-lifecycle}
+\end{figure}
+
+\drivernormative{\subparagraph}{Buffer lifecycle}{Device Types / Video
+  Device / Device Operation / Device Operation: Buffer lifecycle}
+
+The following table shows whether the driver can read or write each
+buffer in each state in Figure~\ref{fig:buffer-lifecycle}. The driver
+MUST not read or write buffers in the state that doesn't permit.
+\begin{center}
+  \begin{tabular}{|c|c|c|}
+    \hline
+    State & Input buffers & Output buffers \\
+    \hline
+    Created  & Read / Write & Read \\
+    Queued   & -            & -    \\
+    Dequeued & Read / Write & Read \\
+    \hline
+  \end{tabular}
+\end{center}
+
+\devicenormative{\subparagraph}{Buffer lifecycle}{Device Types / Video
+  Device / Device Operation / Device Operation: Buffer lifecycle}
+
+The following table shows whether the device can read or write each
+buffer in each state in Figure~\ref{fig:buffer-lifecycle}. The device
+MUST not read or write buffers in the state that doesn't permit.
+\begin{center}
+  \begin{tabular}{ |c|c|c| }
+    \hline
+    State & Input buffers & Output buffers \\
+    \hline
+    Created  & -    & - \\
+    Queued   & Read & Read / Write \\
+    Dequeued & -    & Read \\
+    \hline
+  \end{tabular}
+\end{center}
+
+\subsubsection{Event Virtqueue}
+
+While processing buffers, the device can send asynchronous event
+notifications to the driver. The behaviour depends on the exact
+stream. For example, the decoder device sends a resolution change
+event when it encounters new resolution metadata in the stream.
+
+The device can report events on the event queue. The driver initially
+populates the queue with device-writeable buffers. When the device
+needs to report an event, it fills a buffer and notifies the driver.
+The driver consumes the report and adds a new buffer to the virtqueue.
+
+\begin{lstlisting}
+enum virtio_video_event_type {
+        /* For all devices */
+        VIRTIO_VIDEO_EVENT_ERROR = 0x0100,
+
+        /* For decoder only */
+        VIRTIO_VIDEO_EVENT_DECODER_RESOLUTION_CHANGED = 0x0200,
+};
+
+struct virtio_video_event {
+        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_* types */
+        le32 stream_id;
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{event_type}] type of the triggered event .
+\item[\field{stream_id}] id of the source stream.
+\end{description}
+
+The device MUST send VIRTIO_VIDEO_EVENT_DECODER_RESOLUTION_CHANGED
+whenever it encounters new resolution data in the stream. This
+includes the case of the initial device configuration after metadata
+has been parsed and the case of dynamic resolution change.
-- 
2.25.0.341.g760bfbb309-goog