[virtio-dev] [PATCH 00/11] Additional virtio device resources

[virtio-dev] [PATCH 00/11] Additional virtio device resources

[Nikos Dragazis]

Hi everyone,

This patch introduces the "doorbells" and "device-specific
notifications" as a new set of virtio device resources.

Doorbells are device registers that allow a driver to notify the device
for events other than virtqueue activity.

Device-specific notifications are interrupts that allow a device to
notify the driver for events other than virtqueue activity and
configuration changes.

The purpose of both of these resources is to allow a virtio device to
send/receive **device-specific** notifications to/from the driver. This
is not supported by the current version of the virtio spec, since the
only notifications supported are configuration changes, available and
used buffer notifications.

These resources will be used later by the virtio-vhost-user [1] device
in order to send/receive notifications to/from the driver regarding
activity on the vhost-user virtqueues. By standardizing these resources
as standalone virtio device resources, other future devices will be able
to use them as well.

Also, note that this patch contains some fix commits for some minor
typos/bugs that I bumped into while writing this patch.

Nikos

[1] https://lists.oasis-open.org/archives/virtio-dev/202005/msg00132.html

Nikos Dragazis (11):
  content: Remove duplicate content
  content: Minor fix
  content: Update list of virtio cfg structures
  content: Minor layout fix
  doorbells: Introduce doorbells
  content: Introduce Doorbell cfg type for PCI
  content: Introduce Doorbell register for MMIO
  notifications: Introduce device-specific notifications
  content: Introduce device-specific notification cfg type for PCI
  content: Fix typo
  content: Introduce device-specific notifications for MMIO

 conformance.tex |   2 +
 content.tex     | 177 ++++++++++++++++++++++++++++++++++++++++--------
 doorbells.tex   |  12 ++++
 3 files changed, 163 insertions(+), 28 deletions(-)
 create mode 100644 doorbells.tex

-- 
2.17.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH 01/11] content: Remove duplicate content

[Nikos Dragazis]

Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 content.tex | 8 --------
 1 file changed, 8 deletions(-)

diff --git a/content.tex b/content.tex
index 91735e3..d505e94 100644
--- a/content.tex
+++ b/content.tex
@@ -1101,14 +1101,6 @@ \subsubsection{Shared memory capability}\label{sec:Virtio Transport Options / Vi
 
 The \field{cap.id} MUST be unique for any one device instance.
 
-\devicenormative{\paragraph}{Device-specific configuration}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Shared memory capability}
-
-The region defined by the combination of the \field {cap.offset},
-\field {cap.offset_hi}, and \field {cap.length}, \field
-{cap.length_hi} fields MUST be contained within the declared bar.
-
-The \field{cap.id} MUST be unique for any one device instance.
-
 \subsubsection{Vendor data capability}\label{sec:Virtio
 Transport Options / Virtio Over PCI Bus / PCI Device Layout /
 Vendor data capability}
-- 
2.17.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH 01/11] content: Remove duplicate content

[Alex Bennée]

Nikos Dragazis <ndragazis@arrikto.com> writes:

> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

-- 
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH 02/11] content: Minor fix

[Nikos Dragazis]

Replace "Device-specific configuration" with "Shared memory capability".

Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 content.tex | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/content.tex b/content.tex
index d505e94..3ce768c 100644
--- a/content.tex
+++ b/content.tex
@@ -1093,7 +1093,7 @@ \subsubsection{Shared memory capability}\label{sec:Virtio Transport Options / Vi
 The identifier in \field{cap.id} does not denote a certain order of
 preference; it is only used to uniquely identify a region.
 
-\devicenormative{\paragraph}{Device-specific configuration}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Shared memory capability}
+\devicenormative{\paragraph}{Shared memory capability}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Shared memory capability}
 
 The region defined by the combination of \field{cap.offset},
 \field{offset_hi}, and \field {cap.length}, \field{length_hi}
-- 
2.17.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH 02/11] content: Minor fix

[Alex Bennée]

Nikos Dragazis <ndragazis@arrikto.com> writes:

> Replace "Device-specific configuration" with "Shared memory capability".
>
> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

-- 
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH 03/11] content: Update list of virtio cfg structures

[Nikos Dragazis]

Shared memory and vendor-specific configuration structures have been
introduced in previous commits.

Update the list of virtio configuration structures so that it includes
the shared memory and vendor-specific configuration structures.

Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 content.tex | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/content.tex b/content.tex
index 3ce768c..b5b2f79 100644
--- a/content.tex
+++ b/content.tex
@@ -634,6 +634,8 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
 \item ISR Status
 \item Device-specific configuration (optional)
 \item PCI configuration access
+\item Shared memory regions (optional)
+\item Vendor-specific data (optional)
 \end{itemize}
 
 Each structure can be mapped by a Base Address register (BAR) belonging to
-- 
2.17.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH 03/11] content: Update list of virtio cfg structures

[Alex Bennée]

Nikos Dragazis <ndragazis@arrikto.com> writes:

> Shared memory and vendor-specific configuration structures have been
> introduced in previous commits.
>
> Update the list of virtio configuration structures so that it includes
> the shared memory and vendor-specific configuration structures.
>
> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
> ---
>  content.tex | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/content.tex b/content.tex
> index 3ce768c..b5b2f79 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -634,6 +634,8 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
>  \item ISR Status
>  \item Device-specific configuration (optional)
>  \item PCI configuration access
> +\item Shared memory regions (optional)
> +\item Vendor-specific data (optional)

Are we talking about "Vendor data capability" here? Maybe we should call
it that?

>  \end{itemize}
>  
>  Each structure can be mapped by a Base Address register (BAR) belonging to


-- 
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH 03/11] content: Update list of virtio cfg structures

[Nikos Dragazis]

On 17/9/20 7:34 μ.μ., Alex Bennée wrote:

> Nikos Dragazis <ndragazis@arrikto.com> writes:
>
>> Shared memory and vendor-specific configuration structures have been
>> introduced in previous commits.
>>
>> Update the list of virtio configuration structures so that it includes
>> the shared memory and vendor-specific configuration structures.
>>
>> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
>> ---
>>   content.tex | 2 ++
>>   1 file changed, 2 insertions(+)
>>
>> diff --git a/content.tex b/content.tex
>> index 3ce768c..b5b2f79 100644
>> --- a/content.tex
>> +++ b/content.tex
>> @@ -634,6 +634,8 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
>>   \item ISR Status
>>   \item Device-specific configuration (optional)
>>   \item PCI configuration access
>> +\item Shared memory regions (optional)
>> +\item Vendor-specific data (optional)
> Are we talking about "Vendor data capability" here?

Yep.

> Maybe we should call it that?

I would avoid using the word "capability" here because the items in this
list are referred to as "structures". ACK for s/Vendor-specific/Vendor.
In general, I see that the wording in this section is not consistent in
the sense that the terms "configuration structure" and "capability" are
used interchangeably.

>
>>   \end{itemize}
>>   
>>   Each structure can be mapped by a Base Address register (BAR) belonging to
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH 03/11] content: Update list of virtio cfg structures

[Alex Bennée]

Nikos Dragazis <ndragazis@arrikto.com> writes:

> On 17/9/20 7:34 μ.μ., Alex Bennée wrote:
>
>> Nikos Dragazis <ndragazis@arrikto.com> writes:
>>
>>> Shared memory and vendor-specific configuration structures have been
>>> introduced in previous commits.
>>>
>>> Update the list of virtio configuration structures so that it includes
>>> the shared memory and vendor-specific configuration structures.
>>>
>>> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
>>> ---
>>>   content.tex | 2 ++
>>>   1 file changed, 2 insertions(+)
>>>
>>> diff --git a/content.tex b/content.tex
>>> index 3ce768c..b5b2f79 100644
>>> --- a/content.tex
>>> +++ b/content.tex
>>> @@ -634,6 +634,8 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
>>>   \item ISR Status
>>>   \item Device-specific configuration (optional)
>>>   \item PCI configuration access
>>> +\item Shared memory regions (optional)
>>> +\item Vendor-specific data (optional)
>> Are we talking about "Vendor data capability" here?
>
> Yep.
>
>> Maybe we should call it that?
>
> I would avoid using the word "capability" here because the items in this
> list are referred to as "structures". ACK for s/Vendor-specific/Vendor.
> In general, I see that the wording in this section is not consistent in
> the sense that the terms "configuration structure" and "capability" are
> used interchangeably.

I have a mild preference for configuration structure - but a greater
preference for consistency.

>
>>
>>>   \end{itemize}
>>>   
>>>   Each structure can be mapped by a Base Address register (BAR) belonging to
>>


-- 
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH 04/11] content: Minor layout fix

[Nikos Dragazis]

Fix the device/driver requirements of the vendor data capability so that
they are subsections of the vendor data capability section.

Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 content.tex | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/content.tex b/content.tex
index b5b2f79..94a3693 100644
--- a/content.tex
+++ b/content.tex
@@ -1142,7 +1142,7 @@ \subsubsection{Vendor data capability}\label{sec:Virtio
 To make it safe for a generic driver to access the capability,
 reads from this capability MUST NOT have any side effects.
 
-\devicenormative{\subsection}{Vendor data capability}{Virtio
+\devicenormative{\paragraph}{Vendor data capability}{Virtio
 Transport Options / Virtio Over PCI Bus / PCI Device Layout /
 Vendor data capability}
 
@@ -1159,7 +1159,7 @@ \subsubsection{Vendor data capability}\label{sec:Virtio
 Reads of the Vendor data capability by the driver MUST NOT have any
 side effects.
 
-\drivernormative{\subsection}{Vendor data capability}{Virtio
+\drivernormative{\paragraph}{Vendor data capability}{Virtio
 Transport Options / Virtio Over PCI Bus / PCI Device Layout /
 Vendor data capability}
 
-- 
2.17.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH 04/11] content: Minor layout fix

[Alex Bennée]

Nikos Dragazis <ndragazis@arrikto.com> writes:

> Fix the device/driver requirements of the vendor data capability so that
> they are subsections of the vendor data capability section.
>
> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

-- 
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH 05/11] doorbells: Introduce doorbells

[Nikos Dragazis]

Introduce doorbells as a new virtio device resource.

Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 content.tex   |  2 ++
 doorbells.tex | 12 ++++++++++++
 2 files changed, 14 insertions(+)
 create mode 100644 doorbells.tex

diff --git a/content.tex b/content.tex
index 94a3693..38ed709 100644
--- a/content.tex
+++ b/content.tex
@@ -373,6 +373,8 @@ \section{Driver Notifications} \label{sec:Virtqueues / Driver notifications}
 
 \input{shared-mem.tex}
 
+\input{doorbells.tex}
+
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
 
 We start with an overview of device initialization, then expand on the
diff --git a/doorbells.tex b/doorbells.tex
new file mode 100644
index 0000000..e917536
--- /dev/null
+++ b/doorbells.tex
@@ -0,0 +1,12 @@
+\section{Doorbells}\label{sec:Basic Facilities of a Virtio device / Doorbells}
+
+A doorbell is a device register that triggers a notification to the
+device when accessed by the driver.
+
+Doorbells are an additional facility available to devices that need to
+support receiving driver notifications for events other than virtqueue
+activity (i.e., notifications that are device-specific).
+
+A device may be exporting multiple doorbells to the driver.
+
+The location of doorbells is found in a transport-specific way.
-- 
2.17.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH 05/11] doorbells: Introduce doorbells

[Alex Bennée]

Nikos Dragazis <ndragazis@arrikto.com> writes:

> Introduce doorbells as a new virtio device resource.
>
> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
> ---
>  content.tex   |  2 ++
>  doorbells.tex | 12 ++++++++++++
>  2 files changed, 14 insertions(+)
>  create mode 100644 doorbells.tex
>
> diff --git a/content.tex b/content.tex
> index 94a3693..38ed709 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -373,6 +373,8 @@ \section{Driver Notifications} \label{sec:Virtqueues / Driver notifications}
>  
>  \input{shared-mem.tex}
>  
> +\input{doorbells.tex}
> +
>  \chapter{General Initialization And Device
> Operation}\label{sec:General Initialization And Device Operation}

this has a minor conflict now.

-- 
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH 05/11] doorbells: Introduce doorbells

[Alex Bennée]

Nikos Dragazis <ndragazis@arrikto.com> writes:

> Introduce doorbells as a new virtio device resource.
>
> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

-- 
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH 05/11] doorbells: Introduce doorbells

[Alex Bennée]

Nikos Dragazis <ndragazis@arrikto.com> writes:

> Introduce doorbells as a new virtio device resource.
>
> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
> ---
>  content.tex   |  2 ++
>  doorbells.tex | 12 ++++++++++++
>  2 files changed, 14 insertions(+)
>  create mode 100644 doorbells.tex
>
> diff --git a/content.tex b/content.tex
> index 94a3693..38ed709 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -373,6 +373,8 @@ \section{Driver Notifications} \label{sec:Virtqueues / Driver notifications}
>  
>  \input{shared-mem.tex}
>  
> +\input{doorbells.tex}
> +
>  \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
>  
>  We start with an overview of device initialization, then expand on the
> diff --git a/doorbells.tex b/doorbells.tex
> new file mode 100644
> index 0000000..e917536
> --- /dev/null
> +++ b/doorbells.tex
> @@ -0,0 +1,12 @@
> +\section{Doorbells}\label{sec:Basic Facilities of a Virtio device / Doorbells}
> +
> +A doorbell is a device register that triggers a notification to the
> +device when accessed by the driver.
> +
> +Doorbells are an additional facility available to devices that need to
> +support receiving driver notifications for events other than virtqueue
> +activity (i.e., notifications that are device-specific).
> +
> +A device may be exporting multiple doorbells to the driver.
> +
> +The location of doorbells is found in a transport-specific way.

I just realised we don't specify anything about delivery. Should we say
something about lack of acknowledgement or guarantee that the
notification is delivered here? 

-- 
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH 05/11] doorbells: Introduce doorbells

[Nikos Dragazis]

On 17/9/20 8:01 μ.μ., Alex Bennée wrote:

> Nikos Dragazis <ndragazis@arrikto.com> writes:
>
>> Introduce doorbells as a new virtio device resource.
>>
>> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
>> ---
>>   content.tex   |  2 ++
>>   doorbells.tex | 12 ++++++++++++
>>   2 files changed, 14 insertions(+)
>>   create mode 100644 doorbells.tex
>>
>> diff --git a/content.tex b/content.tex
>> index 94a3693..38ed709 100644
>> --- a/content.tex
>> +++ b/content.tex
>> @@ -373,6 +373,8 @@ \section{Driver Notifications} \label{sec:Virtqueues / Driver notifications}
>>   
>>   \input{shared-mem.tex}
>>   
>> +\input{doorbells.tex}
>> +
>>   \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
>>   
>>   We start with an overview of device initialization, then expand on the
>> diff --git a/doorbells.tex b/doorbells.tex
>> new file mode 100644
>> index 0000000..e917536
>> --- /dev/null
>> +++ b/doorbells.tex
>> @@ -0,0 +1,12 @@
>> +\section{Doorbells}\label{sec:Basic Facilities of a Virtio device / Doorbells}
>> +
>> +A doorbell is a device register that triggers a notification to the
>> +device when accessed by the driver.
>> +
>> +Doorbells are an additional facility available to devices that need to
>> +support receiving driver notifications for events other than virtqueue
>> +activity (i.e., notifications that are device-specific).
>> +
>> +A device may be exporting multiple doorbells to the driver.
>> +
>> +The location of doorbells is found in a transport-specific way.
> I just realised we don't specify anything about delivery. Should we say
> something about lack of acknowledgement or guarantee that the
> notification is delivered here?
>

I don't know. Should we? Based on the spec, the driver knows that each
access delivers a notification to the device. So, I think it is clear that
notifications cannot be lost. An acknowledgment would make sense if
notifications could be lost.

Regarding the rate at which the device can actually process these
notifications, I think this is a whole different story.

If you have something more specific in mind, please share it.

--Nikos

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH 06/11] content: Introduce Doorbell cfg type for PCI

[Nikos Dragazis]

Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 conformance.tex |  1 +
 content.tex     | 50 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 51 insertions(+)

diff --git a/conformance.tex b/conformance.tex
index b6fdec0..f02770f 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -251,6 +251,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / ISR status capability}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Device-specific configuration}
+\item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Doorbell capability}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Shared memory capability}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / PCI configuration access capability}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Non-transitional Device With Legacy Driver}
diff --git a/content.tex b/content.tex
index 38ed709..628265e 100644
--- a/content.tex
+++ b/content.tex
@@ -636,6 +636,7 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
 \item ISR Status
 \item Device-specific configuration (optional)
 \item PCI configuration access
+\item Doorbells (optional)
 \item Shared memory regions (optional)
 \item Vendor-specific data (optional)
 \end{itemize}
@@ -693,6 +694,8 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
 #define VIRTIO_PCI_CAP_DEVICE_CFG        4
 /* PCI configuration access */
 #define VIRTIO_PCI_CAP_PCI_CFG           5
+/* Doorbells */
+#define VIRTIO_PCI_CAP_DOORBELL_CFG      6
 /* Shared memory region */
 #define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG 8
 /* Vendor-specific data */
@@ -1085,6 +1088,53 @@ \subsubsection{Device-specific configuration}\label{sec:Virtio Transport Options
 
 The \field{offset} for the device-specific configuration MUST be 4-byte aligned.
 
+\subsubsection{Doorbell capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Doorbell capability}
+
+The doorbell \ref{sec:Basic Facilities of a Virtio device / Doorbells}
+location is found using the VIRTIO_PCI_CAP_DOORBELL_CFG capability. This
+capability is immediately followed by an additional field, like so:
+
+\begin{lstlisting}
+struct virtio_pci_doorbell_cap {
+        struct virtio_pci_cap cap;
+        le32 doorbell_off_multiplier;
+};
+\end{lstlisting}
+
+The doorbell address within a BAR is calculated as follows:
+
+\begin{lstlisting}
+        cap.offset + doorbell_idx * doorbell_off_multiplier
+\end{lstlisting}
+
+The \field{cap.offset} and \field{doorbell_off_multiplier} are taken
+from the doorbell capability structure above, and the
+\field{doorbell_idx} is the doorbell index. There is no restriction for
+the mapping between doorbells and doorbell indices. The mapping is
+device-specific. One possible mapping would be to use the integers 0 to
+N-1 as the doorbell indices for a total of N doorbells. The total number
+of doorbells exposed by the device is also device-specific.
+
+\devicenormative{\paragraph}{Doorbell capability}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Doorbell capability}
+
+The \field{cap.offset} MUST be 2-byte aligned.
+
+The device MUST either present \field{doorbell_off_multiplier} as an
+even power of 2, or present \field{doorbell_off_multiplier} as 0.
+
+The value \field{cap.length} presented by the device MUST be at least 2
+and MUST be large enough to support doorbell offsets for all supported
+doorbells in all possible configurations.
+
+The value \field{cap.length} presented by the device MUST satisfy:
+
+\begin{lstlisting}
+cap.length >= max_doorbell_idx * doorbell_off_multiplier + 2
+\end{lstlisting}
+
+where \field{max_doorbell_idx} is the maximum doorbell index and is
+dependent on the device.
+
 \subsubsection{Shared memory capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Shared memory capability}
 
 Shared memory regions \ref{sec:Basic Facilities of a Virtio
-- 
2.17.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH 07/11] content: Introduce Doorbell register for MMIO

[Nikos Dragazis]

Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 content.tex | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/content.tex b/content.tex
index 628265e..d6fc17e 100644
--- a/content.tex
+++ b/content.tex
@@ -1884,6 +1884,18 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
     \field{SHMSel} is unused) results in a base address of
     0xffffffffffffffff.
   }
+  \hline
+  \mmioreg{Doorbell}{Doorbell Notifier}{0xc0}{W}{
+    Writing a value to this register
+    \ref{sec:Basic Facilities of a Virtio device / Doorbells} triggers a
+    notification to the device. The value written to this register is
+    interpreted as a doorbell index and the corresponding notification
+    concerns this specific doorbell. The mapping of doorbells to
+    doorbell indices is device-specific. One possible mapping would be
+    to use the integers 0 to N-1 as the doorbell indices for a total of
+    N doorbells. The total number of doorbells exposed by the device is
+    also device-specific.
+  }
   \hline 
   \mmioreg{ConfigGeneration}{Configuration atomicity value}{0x0fc}{R}{
     Reading from this register returns a value describing a version of the device-specific configuration space (see \field{Config}).
@@ -1920,6 +1932,9 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
 
 The device MUST NOT access virtual queue contents when \field{QueueReady} is zero (0x0).
 
+The device MUST ignore doorbell notifications for invalid doorbell
+indices.
+
 \drivernormative{\subsubsection}{MMIO Device Register Layout}{Virtio Transport Options / Virtio Over MMIO / MMIO Device Register Layout}
 The driver MUST NOT access memory locations not described in the
 table \ref{tab:Virtio Trasport Options / Virtio Over MMIO / MMIO Device Register Layout}
@@ -1977,6 +1992,8 @@ \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virti
 Drivers not expecting shared memory MUST NOT use the shared
 memory registers.
 
+Drivers not expecting doorbells MUST NOT use the doorbell register.
+
 Further initialization MUST follow the procedure described in
 \ref{sec:General Initialization And Device Operation / Device Initialization}~\nameref{sec:General Initialization And Device Operation / Device Initialization}.
 
-- 
2.17.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH 08/11] notifications: Introduce device-specific notifications

[Nikos Dragazis]

Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 content.tex | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/content.tex b/content.tex
index d6fc17e..661ebd9 100644
--- a/content.tex
+++ b/content.tex
@@ -165,11 +165,12 @@ \section{Notifications}\label{sec:Basic Facilities of a Virtio Device
 to driver) plays an important role in this specification. The
 modus operandi of the notifications is transport specific.
 
-There are three types of notifications: 
+There are four types of notifications:
 \begin{itemize}
 \item configuration change notification
 \item available buffer notification
-\item used buffer notification. 
+\item used buffer notification
+\item device-specific notification.
 \end{itemize}
 
 Configuration change notifications and used buffer notifications are sent
@@ -182,6 +183,11 @@ \section{Notifications}\label{sec:Basic Facilities of a Virtio Device
 the device. This type of notification indicates that a buffer may have
 been made available on the virtqueue designated by the notification.
 
+Device-specific notifications are sent by the device, the recipient is
+the driver. Their purpose is to allow the device to send notifications
+other than configuration changes and used buffer notifications. These
+notifications are optional and their meaning is device-specific.
+
 The semantics, the transport-specific implementations, and other
 important aspects of the different notifications are specified in detail
 in the following chapters.
-- 
2.17.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH 09/11] content: Introduce device-specific notification cfg type for PCI

[Nikos Dragazis]

Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 conformance.tex |  1 +
 content.tex     | 71 ++++++++++++++++++++++++++++++++++++++++---------
 2 files changed, 60 insertions(+), 12 deletions(-)

diff --git a/conformance.tex b/conformance.tex
index f02770f..e033a0d 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -252,6 +252,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / ISR status capability}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Device-specific configuration}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Doorbell capability}
+\item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Device-specific Notification capability}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Shared memory capability}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / PCI configuration access capability}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Non-transitional Device With Legacy Driver}
diff --git a/content.tex b/content.tex
index 661ebd9..7541a91 100644
--- a/content.tex
+++ b/content.tex
@@ -643,6 +643,7 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
 \item Device-specific configuration (optional)
 \item PCI configuration access
 \item Doorbells (optional)
+\item Device-specific notifications (optional)
 \item Shared memory regions (optional)
 \item Vendor-specific data (optional)
 \end{itemize}
@@ -702,6 +703,8 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
 #define VIRTIO_PCI_CAP_PCI_CFG           5
 /* Doorbells */
 #define VIRTIO_PCI_CAP_DOORBELL_CFG      6
+/* Device-specific notifications */
+#define VIRTIO_PCI_CAP_DEV_NOTIFY_CFG    7
 /* Shared memory region */
 #define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG 8
 /* Vendor-specific data */
@@ -1141,6 +1144,46 @@ \subsubsection{Doorbell capability}\label{sec:Virtio Transport Options / Virtio
 where \field{max_doorbell_idx} is the maximum doorbell index and is
 dependent on the device.
 
+\subsubsection{Device-specific Notification capability}\label{sec:Virtio
+Transport Options / Virtio Over PCI Bus / PCI Device Layout / Device-specific Notification capability}
+
+The device-specific notification
+\ref{sec:Basic Facilities of a Virtio Device / Notifications} location
+is found using the VIRTIO_PCI_CAP_DEV_NOTIFY_CFG capability. The
+device-specific notification structure allows MSI-X vectors to be
+configured for notification interrupts. If MSI-X is not available, bit 2
+of the ISR status \ref{sec:Virtio Transport Options / Virtio Over PCI
+Bus / PCI Device Layout / ISR status capability} indicates that a
+device-specific notification occurred.
+
+The device-specific notification structure is the following:
+
+\begin{lstlisting}
+struct virtio_pci_notification_cfg {
+	le16 notification_select;              /* read-write */
+	le16 notification_msix_vector;         /* read-write */
+};
+\end{lstlisting}
+
+The driver indicates which notification is of interest by writing the
+\field{notification_select} field. The driver then writes the MSI-X
+vector or VIRTIO_MSI_NO_VECTOR to \field{notification_msix_vector} to
+change the MSI-X vector for that notification.
+
+The mapping between notifications and notification indices is
+device-specific. The total number of notifications is also
+device-specific.
+
+\devicenormative{\paragraph}{Device-specific Notification capability}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Device-specific Notification capability}
+
+Device MUST ignore writes to \field{notification_msix_vector} if the
+value written to \field{notification_select} is not a valid notification
+index.
+
+Device MUST return VIRTIO_MSI_NO_VECTOR for reads from
+\field{notification_msix_vector} if the value written to
+\field{notification_select} is not a valid notification index.
+
 \subsubsection{Shared memory capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Shared memory capability}
 
 Shared memory regions \ref{sec:Basic Facilities of a Virtio
@@ -1451,15 +1494,17 @@ \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virti
 \paragraph{MSI-X Vector Configuration}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / MSI-X Vector Configuration}
 
 When MSI-X capability is present and enabled in the device
-(through standard PCI configuration space) \field{config_msix_vector} and \field{queue_msix_vector} are used to map configuration change and queue
-interrupts to MSI-X vectors. In this case, the ISR Status is unused.
+(through standard PCI configuration space) \field{config_msix_vector},
+\field{queue_msix_vector} and \field{notification_msix_vector} are used
+to map configuration change, queue and device-specific interrupts to
+MSI-X vectors respectively. In this case, the ISR Status is unused.
 
 Writing a valid MSI-X Table entry number, 0 to 0x7FF, to
-\field{config_msix_vector}/\field{queue_msix_vector} maps interrupts triggered
-by the configuration change/selected queue events respectively to
-the corresponding MSI-X vector. To disable interrupts for an
-event type, the driver unmaps this event by writing a special NO_VECTOR
-value:
+\field{config_msix_vector}/\field{queue_msix_vector}/\field{notification_msix_vector}
+maps interrupts triggered by the configuration change/selected
+queue/device-specific events respectively to the corresponding MSI-X
+vector. To disable interrupts for an event type, the driver unmaps this
+event by writing a special NO_VECTOR value:
 
 \begin{lstlisting}
 /* Vector value used to disable MSI for queue */
@@ -1486,16 +1531,18 @@ \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virti
 vector 0 to MSI-X \field{Table Size}.
 Device MUST support unmapping any event type.
 
-The device MUST return vector mapped to a given event,
-(NO_VECTOR if unmapped) on read of \field{config_msix_vector}/\field{queue_msix_vector}.
-The device MUST have all queue and configuration change
-events are unmapped upon reset.
+The device MUST return the vector mapped to a given event,
+(NO_VECTOR if unmapped) on read of
+\field{config_msix_vector}/\field{queue_msix_vector}/\field{notification_msix_vector}.
+The device MUST have all queue/configuration change/device-specific
+events unmapped upon reset.
 
 Devices SHOULD NOT cause mapping an event to vector to fail
 unless it is impossible for the device to satisfy the mapping
 request.  Devices MUST report mapping
 failures by returning the NO_VECTOR value when the relevant
-\field{config_msix_vector}/\field{queue_msix_vector} field is read. 
+\field{config_msix_vector}/\field{queue_msix_vector}/\field{notification_msix_vector}
+field is read.
 
 \drivernormative{\subparagraph}{MSI-X Vector Configuration}{Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / MSI-X Vector Configuration}
 
-- 
2.17.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH 10/11] content: Fix typo

[Nikos Dragazis]

Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 content.tex | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/content.tex b/content.tex
index 7541a91..d141e92 100644
--- a/content.tex
+++ b/content.tex
@@ -1550,7 +1550,7 @@ \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virti
 Driver MAY fall back on using INT\#x interrupts for a device
 which only supports one MSI-X vector (MSI-X Table Size = 0).
 
-Driver MAY intepret the Table Size as a hint from the device
+Driver MAY interpret the Table Size as a hint from the device
 for the suggested number of MSI-X vectors to use.
 
 Driver MUST NOT attempt to map an event to a vector
-- 
2.17.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH 11/11] content: Introduce device-specific notifications for MMIO

[Nikos Dragazis]

Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 content.tex | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/content.tex b/content.tex
index d141e92..b0949f9 100644
--- a/content.tex
+++ b/content.tex
@@ -1872,6 +1872,10 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
         in at least one of the active virtual queues.
       \item [Configuration Change Notification] - bit 1 - the interrupt was
         asserted because the configuration of the device has changed.
+      \item [Device-specific Notification] - bit 2 - the interrupt was
+        asserted because a device-specific event
+	\ref{sec:Basic Facilities of a Virtio Device / Notifications}
+	occurred.
     \end{description}
   }
   \hline 
@@ -2100,8 +2104,9 @@ \subsubsection{Notifications From The Device}\label{sec:Virtio Transport Options
 The memory mapped virtio device is using a single, dedicated
 interrupt signal, which is asserted when at least one of the
 bits described in the description of \field{InterruptStatus}
-is set. This is how the device sends a used buffer notification
-or a configuration change notification to the device.
+is set. This is how the device sends a
+used buffer/configuration change/device-specific notification to the
+device.
 
 \drivernormative{\paragraph}{Notifications From The Device}{Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notifications From The Device}
 After receiving an interrupt, the driver MUST read
-- 
2.17.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH 00/11] Additional virtio device resources

[Nikos Dragazis]

On 24/8/20 5:36 μ.μ., Nikos Dragazis wrote:

> Hi everyone,
>
> This patch introduces the "doorbells" and "device-specific
> notifications" as a new set of virtio device resources.
>
> Doorbells are device registers that allow a driver to notify the device
> for events other than virtqueue activity.
>
> Device-specific notifications are interrupts that allow a device to
> notify the driver for events other than virtqueue activity and
> configuration changes.
>
> The purpose of both of these resources is to allow a virtio device to
> send/receive **device-specific** notifications to/from the driver. This
> is not supported by the current version of the virtio spec, since the
> only notifications supported are configuration changes, available and
> used buffer notifications.
>
> These resources will be used later by the virtio-vhost-user [1] device
> in order to send/receive notifications to/from the driver regarding
> activity on the vhost-user virtqueues. By standardizing these resources
> as standalone virtio device resources, other future devices will be able
> to use them as well.
>
> Also, note that this patch contains some fix commits for some minor
> typos/bugs that I bumped into while writing this patch.
>
> Nikos
>
> [1] https://lists.oasis-open.org/archives/virtio-dev/202005/msg00132.html
>
> Nikos Dragazis (11):
>    content: Remove duplicate content
>    content: Minor fix
>    content: Update list of virtio cfg structures
>    content: Minor layout fix
>    doorbells: Introduce doorbells
>    content: Introduce Doorbell cfg type for PCI
>    content: Introduce Doorbell register for MMIO
>    notifications: Introduce device-specific notifications
>    content: Introduce device-specific notification cfg type for PCI
>    content: Fix typo
>    content: Introduce device-specific notifications for MMIO
>
>   conformance.tex |   2 +
>   content.tex     | 177 ++++++++++++++++++++++++++++++++++++++++--------
>   doorbells.tex   |  12 ++++
>   3 files changed, 163 insertions(+), 28 deletions(-)
>   create mode 100644 doorbells.tex
>

Kind reminder on this.

Michael, did you have the chance to look at this?
Looking forward to your feedback so that we can discuss.

Thanks,
Nikos

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org