Re: [PULL] virtio

Re: [PULL] virtio

[Michael S. Tsirkin]

On Tue, Nov 01, 2011 at 10:39:08AM +1030, Rusty Russell wrote:
>  * [new tag]         rusty@rustcorp.com.au-v3.1-7196-gac5be1e -> rusty@rustcorp.com.au-v3.1-7196-gac5be1e
> 
> The following changes since commit 839d8810747bbf39e0a5a7f223b67bffa7945f8d:
> 
>   Merge branch 'i2c-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jdelvare/staging (2011-10-30 15:54:59 -0700)
> 
> are available in the git repository at:
> 
>   git://github.com/rustyrussell/linux.git master
> 
> Alexey Kardashevskiy (1):
>       virtio-pci: Use PCI MMIO instead of PIO when available

I missed this one - wasn't Cc'd neither me, kvm or virtio mailing lists.

It's well known that mmio is much slower than pio on kvm, since
mmio needs to be emulated to get at the address.
So I'm expecting this will cause a performance regression.
IMO we should keep using PIO for VQ and interrupt status access
if PIO is available.

Another consideration is that in an attempt to pack data
densely in the PIO space the layout became messy.
It would be better to have common config space and
per-device config space in separate pages, possibly
with padding between them.

So I'd like a bit more discussion on this patch,
I'm concerned that if this is released in 3.2 as is we'll
have to support this forever. How about a revert for now?


> Amit Shah (8):
>       virtio: console: Fix indentation
>       virtio: console: Ignore port name update request if name already set
>       virtio: console: Use wait_event_freezable instead of _interruptible
>       virtio: console: Fix return type for get_inbuf()
>       virtio: console: make get_inbuf() return port->inbuf if present
>       virtio: console: rename variable
>       virtio: console: make discard_port_data() use get_inbuf()
>       virtio: console: add port stats for bytes received, sent and discarded
> 
> Christian Borntraeger (1):
>       virtio: console: wait for first console port for early console output
> 
> Krishna Kumar (1):
>       virtio: Dont add "config" to list for !per_vq_vector
> 
> Michael S. Tsirkin (1):
>       virtio-blk: use ida to allocate disk index
> 
> Pawel Moll (1):
>       virtio: Add platform bus driver for memory mapped virtio device
> 
> Sasha Levin (3):
>       virtio-console: Use virtio_config_val() for retrieving config
>       virtio_config: Add virtio_config_val_len()
>       virtio-net: Use virtio_config_val() for retrieving config
> 
> Wang Sheng-Hui (2):
>       virtio.h: correct comment for struct virtio_driver
>       virtio: modify vring_init and vring_size to take account of the layout containing *_event_idx
> 
>  Documentation/devicetree/bindings/virtio/mmio.txt |   17 +
>  drivers/block/virtio_blk.c                        |   30 +-
>  drivers/char/virtio_console.c                     |  120 ++++--
>  drivers/net/virtio_net.c                          |   14 +-
>  drivers/virtio/Kconfig                            |   11 +
>  drivers/virtio/Makefile                           |    1 +
>  drivers/virtio/virtio_mmio.c                      |  479 +++++++++++++++++++++
>  drivers/virtio/virtio_pci.c                       |   20 +-
>  include/linux/virtio.h                            |    4 +-
>  include/linux/virtio_config.h                     |    3 +
>  include/linux/virtio_mmio.h                       |  111 +++++
>  include/linux/virtio_ring.h                       |    6 +-
>  12 files changed, 749 insertions(+), 67 deletions(-)
>  create mode 100644 Documentation/devicetree/bindings/virtio/mmio.txt
>  create mode 100644 drivers/virtio/virtio_mmio.c
>  create mode 100644 include/linux/virtio_mmio.h

Re: [PULL] virtio

[Sasha Levin]

On Tue, 2011-11-01 at 13:45 +0200, Michael S. Tsirkin wrote:
> On Tue, Nov 01, 2011 at 10:39:08AM +1030, Rusty Russell wrote:
> >  * [new tag]         rusty@rustcorp.com.au-v3.1-7196-gac5be1e -> rusty@rustcorp.com.au-v3.1-7196-gac5be1e
> > 
> > The following changes since commit 839d8810747bbf39e0a5a7f223b67bffa7945f8d:
> > 
> >   Merge branch 'i2c-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jdelvare/staging (2011-10-30 15:54:59 -0700)
> > 
> > are available in the git repository at:
> > 
> >   git://github.com/rustyrussell/linux.git master
> > 
> > Alexey Kardashevskiy (1):
> >       virtio-pci: Use PCI MMIO instead of PIO when available
> 
> I missed this one - wasn't Cc'd neither me, kvm or virtio mailing lists.
> 
> It's well known that mmio is much slower than pio on kvm, since
> mmio needs to be emulated to get at the address.
> So I'm expecting this will cause a performance regression.
> IMO we should keep using PIO for VQ and interrupt status access
> if PIO is available.
> 
> Another consideration is that in an attempt to pack data
> densely in the PIO space the layout became messy.
> It would be better to have common config space and
> per-device config space in separate pages, possibly
> with padding between them.
> 
> So I'd like a bit more discussion on this patch,
> I'm concerned that if this is released in 3.2 as is we'll
> have to support this forever. How about a revert for now?

Another thing, the patch tries to map BAR 2 and use it as the
configuration space.

It's both not documented properly anywhere, and is not fully backwards
compatible - we currently use BAR 2 as part of our MSIX handling in the
kvm tool and I'm sure we're not the only ones to assume virtio-pci only
uses BAR 0.

A proper solution would be for example a configuration in the PIO config
space which points to the MMIO BAR to use instead.

Unless Michael pointed this patch out, it would have broken (at least)
the kvm tool in a non obvious way which would require a rather long
session of 'git bisect' to figure out whats wrong.

-- 

Sasha.

Re: [PULL] virtio

[Michael S. Tsirkin]

On Tue, Nov 01, 2011 at 02:33:33PM +0200, Sasha Levin wrote:
> On Tue, 2011-11-01 at 13:45 +0200, Michael S. Tsirkin wrote:
> > On Tue, Nov 01, 2011 at 10:39:08AM +1030, Rusty Russell wrote:
> > >  * [new tag]         rusty@rustcorp.com.au-v3.1-7196-gac5be1e -> rusty@rustcorp.com.au-v3.1-7196-gac5be1e
> > > 
> > > The following changes since commit 839d8810747bbf39e0a5a7f223b67bffa7945f8d:
> > > 
> > >   Merge branch 'i2c-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jdelvare/staging (2011-10-30 15:54:59 -0700)
> > > 
> > > are available in the git repository at:
> > > 
> > >   git://github.com/rustyrussell/linux.git master
> > > 
> > > Alexey Kardashevskiy (1):
> > >       virtio-pci: Use PCI MMIO instead of PIO when available
> > 
> > I missed this one - wasn't Cc'd neither me, kvm or virtio mailing lists.
> > 
> > It's well known that mmio is much slower than pio on kvm, since
> > mmio needs to be emulated to get at the address.
> > So I'm expecting this will cause a performance regression.
> > IMO we should keep using PIO for VQ and interrupt status access
> > if PIO is available.
> > 
> > Another consideration is that in an attempt to pack data
> > densely in the PIO space the layout became messy.
> > It would be better to have common config space and
> > per-device config space in separate pages, possibly
> > with padding between them.
> > 
> > So I'd like a bit more discussion on this patch,
> > I'm concerned that if this is released in 3.2 as is we'll
> > have to support this forever. How about a revert for now?
> 
> Another thing, the patch tries to map BAR 2 and use it as the
> configuration space.
> 
> It's both not documented properly anywhere, and is not fully backwards
> compatible - we currently use BAR 2 as part of our MSIX handling in the
> kvm tool and I'm sure we're not the only ones to assume virtio-pci only
> uses BAR 0.
> 
> A proper solution would be for example a configuration in the PIO config
> space which points to the MMIO BAR to use instead.

I think it makes sense to put the configuration in PCI
configuration space, using vendor-specific capability.
This way we can reuse existing functionality for scanning
capability lists.

> Unless Michael pointed this patch out, it would have broken (at least)
> the kvm tool in a non obvious way which would require a rather long
> session of 'git bisect' to figure out whats wrong.
> 
> -- 
> 
> Sasha.

Re: [PULL] virtio

[Sasha Levin]

On Tue, 2011-11-01 at 14:42 +0200, Michael S. Tsirkin wrote:
> On Tue, Nov 01, 2011 at 02:33:33PM +0200, Sasha Levin wrote:
> > On Tue, 2011-11-01 at 13:45 +0200, Michael S. Tsirkin wrote:
> > > On Tue, Nov 01, 2011 at 10:39:08AM +1030, Rusty Russell wrote:
> > > >  * [new tag]         rusty@rustcorp.com.au-v3.1-7196-gac5be1e -> rusty@rustcorp.com.au-v3.1-7196-gac5be1e
> > > > 
> > > > The following changes since commit 839d8810747bbf39e0a5a7f223b67bffa7945f8d:
> > > > 
> > > >   Merge branch 'i2c-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jdelvare/staging (2011-10-30 15:54:59 -0700)
> > > > 
> > > > are available in the git repository at:
> > > > 
> > > >   git://github.com/rustyrussell/linux.git master
> > > > 
> > > > Alexey Kardashevskiy (1):
> > > >       virtio-pci: Use PCI MMIO instead of PIO when available
> > > 
> > > I missed this one - wasn't Cc'd neither me, kvm or virtio mailing lists.
> > > 
> > > It's well known that mmio is much slower than pio on kvm, since
> > > mmio needs to be emulated to get at the address.
> > > So I'm expecting this will cause a performance regression.
> > > IMO we should keep using PIO for VQ and interrupt status access
> > > if PIO is available.
> > > 
> > > Another consideration is that in an attempt to pack data
> > > densely in the PIO space the layout became messy.
> > > It would be better to have common config space and
> > > per-device config space in separate pages, possibly
> > > with padding between them.
> > > 
> > > So I'd like a bit more discussion on this patch,
> > > I'm concerned that if this is released in 3.2 as is we'll
> > > have to support this forever. How about a revert for now?
> > 
> > Another thing, the patch tries to map BAR 2 and use it as the
> > configuration space.
> > 
> > It's both not documented properly anywhere, and is not fully backwards
> > compatible - we currently use BAR 2 as part of our MSIX handling in the
> > kvm tool and I'm sure we're not the only ones to assume virtio-pci only
> > uses BAR 0.
> > 
> > A proper solution would be for example a configuration in the PIO config
> > space which points to the MMIO BAR to use instead.
> 
> I think it makes sense to put the configuration in PCI
> configuration space, using vendor-specific capability.
> This way we can reuse existing functionality for scanning
> capability lists.

Yup, I agree. It would also allow dropping the PIO config BAR altogether
after some period of backwards compatibility.

It was mostly to point out that the patch isn't really backwards
compatible :)

-- 

Sasha.

Re: [PULL] virtio

[Rusty Russell]

On Tue, 1 Nov 2011 14:42:23 +0200, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Tue, Nov 01, 2011 at 02:33:33PM +0200, Sasha Levin wrote:
> > A proper solution would be for example a configuration in the PIO config
> > space which points to the MMIO BAR to use instead.
> 
> I think it makes sense to put the configuration in PCI
> configuration space, using vendor-specific capability.
> This way we can reuse existing functionality for scanning
> capability lists.

Patch welcome; first one to implement it becomes virtio PCI maintainer!
I'll keep maintaining the spec, since it covers much more than the PCI
mechanics, but everything I know about PCI I learn from these
discussions.

Meanwhile, I've dropped this patch; mea culpa.

> > Unless Michael pointed this patch out, it would have broken (at least)
> > the kvm tool in a non obvious way which would require a rather long
> > session of 'git bisect' to figure out whats wrong.

I assumed not, until QEMU provided the second BAR it would simply fall
back.  Or am I completely misunderstanding?

Thanks,
Rusty.

Re: [PULL] virtio

[Sasha Levin]

On Wed, 2011-11-02 at 11:39 +1030, Rusty Russell wrote:
> On Tue, 1 Nov 2011 14:42:23 +0200, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > On Tue, Nov 01, 2011 at 02:33:33PM +0200, Sasha Levin wrote:
> > > Unless Michael pointed this patch out, it would have broken (at least)
> > > the kvm tool in a non obvious way which would require a rather long
> > > session of 'git bisect' to figure out whats wrong.
> 
> I assumed not, until QEMU provided the second BAR it would simply fall
> back.  Or am I completely misunderstanding?

QEMU uses two bars within each virtio-pci device. First one (BAR 0) is
the PIO config BAR, and the second one (BAR 1) is the MSI-X BAR used for
the MSI-X table and the PBA.

Since the PCI spec allows you to have the MSI-X table and the PBA in
completely different regions pointed by two different BARs, we did so in
kvm tools and have BAR 1 pointing to the MSI-X table, and BAR 2 pointing
to the PBA ( - we *already* provided the second BAR and used it for
something else).

This means that with this patch, we'd load BAR 2 and use it as the
config space, instead of the PBA table which it really represents.

-- 

Sasha.

Re: [PULL] virtio

[Rusty Russell]

On Wed, 02 Nov 2011 06:52:44 +0200, Sasha Levin <levinsasha928@gmail.com> wrote:
> Since the PCI spec allows you to have the MSI-X table and the PBA in
> completely different regions pointed by two different BARs, we did so in
> kvm tools and have BAR 1 pointing to the MSI-X table, and BAR 2 pointing
> to the PBA ( - we *already* provided the second BAR and used it for
> something else).

Got it!  Thanks for the explanation.

Cheers,
Rusty.

[PATCH RFC] virtio-pci: flexible configuration layout

[Michael S. Tsirkin]

Add a flexible mechanism to specify virtio configuration layout, using
pci vendor-specific capability.  A separate capability is used for each
of common, device specific and data-path accesses.

Warning: compiled only.
This patch also needs to be split up, pci_iomap changes
also need arch updates for non-x86.

We also will need to update the spec.

See the first chunk for layout documentation.

Posting here for early feedback.

In particular:

Do we need to require offset to be aligned?
Does iowrite16 work with unaligned accesses on all architectures?
Does using ioread/write as we do add overhead as compared to
plain PIO accesses?

Jesse - are you OK with the pci_iomap_range API proposed here
(see last chunks)?
I noticed lots of architectures duplicate the implementation
of pci_iomap - makes sense to clean that up?

Thanks,

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

---

diff --git a/include/linux/virtio_pci.h b/include/linux/virtio_pci.h
index ea66f3f..b8a9de2 100644
--- a/include/linux/virtio_pci.h
+++ b/include/linux/virtio_pci.h
@@ -92,4 +92,38 @@
 /* The alignment to use between consumer and producer parts of vring.
  * x86 pagesize again. */
 #define VIRTIO_PCI_VRING_ALIGN		4096
+
+/*
+ * Layout for Virtio PCI vendor specific capability (little-endian):
+ * 5 bit virtio capability id.
+ * 3 bit BAR index register, specifying which BAR to use.
+ * 4 byte cfg offset within the BAR.
+ * 4 byte cfg size.
+ */
+
+/* A single virtio device has multiple vendor specific capabilities, we use the
+ * 5 bit ID field to distinguish between these. */
+#define VIRTIO_PCI_CAP_ID		3
+#define VIRTIO_PCI_CAP_ID_MASK		0x1f
+#define VIRTIO_PCI_CAP_ID_SHIFT		0
+
+/* IDs for different capabilities. If a specific configuration
+ * is missing, legacy PIO path is used. */
+/* Common configuration */
+#define VIRTIO_PCI_CAP_COMMON_CFG	0
+/* Device specific confiuration */
+#define VIRTIO_PCI_CAP_DEVICE_CFG	1
+/* Notifications and ISR access */
+#define VIRTIO_PCI_CAP_NOTIFY_CFG	2
+
+/* Index of the BAR including this configuration */
+#define VIRTIO_PCI_CAP_CFG_BIR		3
+#define VIRTIO_PCI_CAP_CFG_BIR_MASK	(0x7)
+#define VIRTIO_PCI_CAP_CFG_BIR_SHIFT	5
+
+/* Offset within the BAR */
+#define VIRTIO_PCI_CAP_CFG_OFF		4
+/* Size of the configuration space */
+#define VIRTIO_PCI_CAP_CFG_SIZE		8
+
 #endif
diff --git a/drivers/virtio/virtio_pci.c b/drivers/virtio/virtio_pci.c
index 4bcc8b8..d4ed130 100644
--- a/drivers/virtio/virtio_pci.c
+++ b/drivers/virtio/virtio_pci.c
@@ -37,8 +37,12 @@ struct virtio_pci_device
 	struct virtio_device vdev;
 	struct pci_dev *pci_dev;
 
-	/* the IO mapping for the PCI config space */
+	/* the IO address for the common PCI config space */
 	void __iomem *ioaddr;
+	/* the IO address for device specific config */
+	void __iomem *ioaddr_device;
+	/* the IO address to use for notifications operations */
+	void __iomem *ioaddr_notify;
 
 	/* a list of queues so we can dispatch IRQs */
 	spinlock_t lock;
@@ -57,8 +61,142 @@ struct virtio_pci_device
 	unsigned msix_used_vectors;
 	/* Whether we have vector per vq */
 	bool per_vq_vectors;
+
+	/* Various IO mappings: used for resource tracking only. */
+
+	/* Legacy BAR0: typically PIO. */
+	void __iomem *legacy_map;
+
+	/* Mappings specified by device capabilities: typically in MMIO */
+	void __iomem *notify_map;
+	void __iomem *common_map;
+	void __iomem *device_map;
 };
 
+/*
+ * With PIO, device-specific config moves as MSI-X is enabled/disabled.
+ * Use this accessor to keep pointer to that config in sync.
+ */
+static void virtio_pci_set_msix_enabled(struct virtio_pci_device *vp_dev, int enabled)
+{
+	vp_dev->msix_enabled = enabled;
+	if (vp_dev->device_map)
+		vp_dev->ioaddr_device = vp_dev->device_map;
+	else
+		vp_dev->ioaddr_device = vp_dev->legacy_map +
+			VIRTIO_PCI_CONFIG(vp_dev);
+}
+
+static void __iomem *virtio_pci_map_cfg(struct virtio_pci_device *vp_dev, u8 cap_id)
+{
+        u32 size;
+        u32 offset;
+        u8 bir;
+        u8 cap_len;
+	int pos;
+	struct pci_dev *dev = vp_dev->pci_dev;
+	void __iomem *p;
+
+	for (pos = pci_find_capability(dev, PCI_CAP_ID_VNDR);
+	     pos > 0;
+	     pos = pci_find_next_capability(dev, pos, PCI_CAP_ID_VNDR)) {
+		u8 id;
+		pci_read_config_byte(dev, pos + PCI_VNDR_CAP_LEN, &cap_len);
+		if (cap_len < VIRTIO_PCI_CAP_ID + 1)
+			continue;
+		pci_read_config_byte(dev, pos + VIRTIO_PCI_CAP_ID, &id);
+		id >>= VIRTIO_PCI_CAP_ID_SHIFT;
+		id &= VIRTIO_PCI_CAP_ID_MASK;
+		if (id == cap_id)
+			break;
+	}
+
+	if (pos <= 0)
+		return NULL;
+
+	if (cap_len < VIRTIO_PCI_CAP_CFG_BIR + 1)
+		goto err;
+        pci_read_config_byte(dev, pos + VIRTIO_PCI_CAP_CFG_BIR, &bir);
+	if (cap_len < VIRTIO_PCI_CAP_CFG_OFF + 4)
+		goto err;
+        pci_read_config_dword(dev, pos + VIRTIO_PCI_CAP_CFG_OFF, &offset);
+	if (cap_len < VIRTIO_PCI_CAP_CFG_SIZE + 4)
+		goto err;
+        pci_read_config_dword(dev, pos + VIRTIO_PCI_CAP_CFG_SIZE, &size);
+        bir >>= VIRTIO_PCI_CAP_CFG_BIR_SHIFT;
+        bir &= VIRTIO_PCI_CAP_CFG_BIR_MASK;
+
+	/* It's possible for a device to supply a huge config space,
+	 * but we'll never need to map more than a page ATM. */
+	p = pci_iomap_range(dev, bir, offset, size, PAGE_SIZE);
+	if (!p)
+		dev_err(&vp_dev->vdev.dev, "Unable to map virtio pci memory");
+	return p;
+err:
+	dev_err(&vp_dev->vdev.dev, "Unable to parse virtio pci capability");
+	return NULL;
+}
+
+static void virtio_pci_iounmap(struct virtio_pci_device *vp_dev)
+{
+	if (vp_dev->legacy_map)
+		pci_iounmap(vp_dev->pci_dev, vp_dev->legacy_map);
+	if (vp_dev->notify_map)
+		pci_iounmap(vp_dev->pci_dev, vp_dev->notify_map);
+	if (vp_dev->common_map)
+		pci_iounmap(vp_dev->pci_dev, vp_dev->common_map);
+	if (vp_dev->device_map)
+		pci_iounmap(vp_dev->pci_dev, vp_dev->device_map);
+}
+
+static int virtio_pci_iomap(struct virtio_pci_device *vp_dev)
+{
+	vp_dev->notify_map = virtio_pci_map_cfg(vp_dev,
+						     VIRTIO_PCI_CAP_NOTIFY_CFG);
+	vp_dev->common_map = virtio_pci_map_cfg(vp_dev,
+						     VIRTIO_PCI_CAP_COMMON_CFG);
+	vp_dev->device_map = virtio_pci_map_cfg(vp_dev,
+						     VIRTIO_PCI_CAP_DEVICE_CFG);
+
+	if (!vp_dev->notify_map || !vp_dev->common_map ||
+	    !vp_dev->device_map) {
+		/*
+		 * If not all capabilities present, map legacy PIO.
+		 * Legacy access is at BAR 0. We never need to map
+		 * more than 256 bytes there, since legacy config space
+		 * used PIO which has this size limit.
+		 * */
+		vp_dev->legacy_map = pci_iomap(vp_dev->pci_dev, 0, 256);
+		if (!vp_dev->legacy_map) {
+			dev_err(&vp_dev->vdev.dev, "Unable to map legacy PIO");
+			goto err;
+		}
+	}
+
+	/* Prefer MMIO if available. If not, fallback to legacy PIO. */
+	if (vp_dev->common_map)
+		vp_dev->ioaddr = vp_dev->common_map;
+	else
+		vp_dev->ioaddr = vp_dev->legacy_map;
+
+	if (vp_dev->device_map)
+		vp_dev->ioaddr_device = vp_dev->device_map;
+	else
+		vp_dev->ioaddr_device = vp_dev->legacy_map +
+			VIRTIO_PCI_CONFIG(vp_dev);
+
+	if (vp_dev->notify_map)
+		vp_dev->ioaddr_notify = vp_dev->notify_map;
+	else
+		vp_dev->ioaddr_notify = vp_dev->legacy_map +
+			VIRTIO_PCI_QUEUE_NOTIFY;
+
+	return 0;
+err:
+	virtio_pci_iounmap(vp_dev);
+	return -EINVAL;
+}
+
 /* Constants for MSI-X */
 /* Use first vector for configuration changes, second and the rest for
  * virtqueues Thus, we need at least 2 vectors for MSI. */
@@ -130,8 +268,7 @@ static void vp_get(struct virtio_device *vdev, unsigned offset,
 		   void *buf, unsigned len)
 {
 	struct virtio_pci_device *vp_dev = to_vp_device(vdev);
-	void __iomem *ioaddr = vp_dev->ioaddr +
-				VIRTIO_PCI_CONFIG(vp_dev) + offset;
+	void __iomem *ioaddr = vp_dev->ioaddr_device + offset;
 	u8 *ptr = buf;
 	int i;
 
@@ -145,8 +282,7 @@ static void vp_set(struct virtio_device *vdev, unsigned offset,
 		   const void *buf, unsigned len)
 {
 	struct virtio_pci_device *vp_dev = to_vp_device(vdev);
-	void __iomem *ioaddr = vp_dev->ioaddr +
-				VIRTIO_PCI_CONFIG(vp_dev) + offset;
+	void __iomem *ioaddr = vp_dev->ioaddr_device + offset;
 	const u8 *ptr = buf;
 	int i;
 
@@ -184,7 +320,7 @@ static void vp_notify(struct virtqueue *vq)
 
 	/* we write the queue's selector into the notification register to
 	 * signal the other end */
-	iowrite16(info->queue_index, vp_dev->ioaddr + VIRTIO_PCI_QUEUE_NOTIFY);
+	iowrite16(info->queue_index, vp_dev->ioaddr_notify);
 }
 
 /* Handle a configuration change: Tell driver if it wants to know. */
@@ -231,7 +367,8 @@ static irqreturn_t vp_interrupt(int irq, void *opaque)
 
 	/* reading the ISR has the effect of also clearing it so it's very
 	 * important to save off the value. */
-	isr = ioread8(vp_dev->ioaddr + VIRTIO_PCI_ISR);
+	isr = ioread8(vp_dev->ioaddr_notify +
+		      VIRTIO_PCI_ISR - VIRTIO_PCI_QUEUE_NOTIFY);
 
 	/* It's definitely not us if the ISR was not high */
 	if (!isr)
@@ -265,7 +402,7 @@ static void vp_free_vectors(struct virtio_device *vdev)
 		ioread16(vp_dev->ioaddr + VIRTIO_MSI_CONFIG_VECTOR);
 
 		pci_disable_msix(vp_dev->pci_dev);
-		vp_dev->msix_enabled = 0;
+                virtio_pci_set_msix_enabled(vp_dev, 0);
 		vp_dev->msix_vectors = 0;
 	}
 
@@ -303,7 +440,7 @@ static int vp_request_msix_vectors(struct virtio_device *vdev, int nvectors,
 	if (err)
 		goto error;
 	vp_dev->msix_vectors = nvectors;
-	vp_dev->msix_enabled = 1;
+        virtio_pci_set_msix_enabled(vp_dev, 1);
 
 	/* Set the vector used for configuration */
 	v = vp_dev->msix_used_vectors;
@@ -447,7 +584,10 @@ static void vp_del_vq(struct virtqueue *vq)
 		iowrite16(VIRTIO_MSI_NO_VECTOR,
 			  vp_dev->ioaddr + VIRTIO_MSI_QUEUE_VECTOR);
 		/* Flush the write out to device */
-		ioread8(vp_dev->ioaddr + VIRTIO_PCI_ISR);
+		ioread8(vp_dev->ioaddr + VIRTIO_MSI_QUEUE_VECTOR);
+		/* And clear ISR: TODO: really needed? */
+		ioread8(vp_dev->ioaddr_notify +
+		      VIRTIO_PCI_ISR - VIRTIO_PCI_QUEUE_NOTIFY);
 	}
 
 	vring_del_virtqueue(vq);
@@ -638,8 +778,8 @@ static int __devinit virtio_pci_probe(struct pci_dev *pci_dev,
 	if (err)
 		goto out_enable_device;
 
-	vp_dev->ioaddr = pci_iomap(pci_dev, 0, 0);
-	if (vp_dev->ioaddr == NULL)
+	err = virtio_pci_iomap(vp_dev);
+	if (err)
 		goto out_req_regions;
 
 	pci_set_drvdata(pci_dev, vp_dev);
@@ -661,7 +801,7 @@ static int __devinit virtio_pci_probe(struct pci_dev *pci_dev,
 
 out_set_drvdata:
 	pci_set_drvdata(pci_dev, NULL);
-	pci_iounmap(pci_dev, vp_dev->ioaddr);
+	virtio_pci_iounmap(vp_dev);
 out_req_regions:
 	pci_release_regions(pci_dev);
 out_enable_device:
@@ -679,7 +819,7 @@ static void __devexit virtio_pci_remove(struct pci_dev *pci_dev)
 
 	vp_del_vqs(&vp_dev->vdev);
 	pci_set_drvdata(pci_dev, NULL);
-	pci_iounmap(pci_dev, vp_dev->ioaddr);
+	virtio_pci_iounmap(vp_dev);
 	pci_release_regions(pci_dev);
 	pci_disable_device(pci_dev);
 }
diff --git a/include/asm-generic/io.h b/include/asm-generic/io.h
index 9120887..3cf1787 100644
--- a/include/asm-generic/io.h
+++ b/include/asm-generic/io.h
@@ -286,6 +286,10 @@ static inline void writesb(const void __iomem *addr, const void *buf, int len)
 /* Create a virtual mapping cookie for a PCI BAR (memory or IO) */
 struct pci_dev;
 extern void __iomem *pci_iomap(struct pci_dev *dev, int bar, unsigned long max);
+extern void __iomem *pci_iomap_range(struct pci_dev *dev, int bar,
+                                     unsigned offset,
+                                     unsigned long minlen,
+                                     unsigned long maxlen);
 static inline void pci_iounmap(struct pci_dev *dev, void __iomem *p)
 {
 }
diff --git a/include/asm-generic/iomap.h b/include/asm-generic/iomap.h
index 98dcd76..6f192d4 100644
--- a/include/asm-generic/iomap.h
+++ b/include/asm-generic/iomap.h
@@ -70,8 +70,19 @@ extern void ioport_unmap(void __iomem *);
 /* Create a virtual mapping cookie for a PCI BAR (memory or IO) */
 struct pci_dev;
 extern void __iomem *pci_iomap(struct pci_dev *dev, int bar, unsigned long max);
+extern void __iomem *pci_iomap_range(struct pci_dev *dev, int bar,
+                                     unsigned offset,
+                                     unsigned long minlen,
+                                     unsigned long maxlen);
 extern void pci_iounmap(struct pci_dev *dev, void __iomem *);
 #else
+static inline void __iomem *pci_iomap_range(struct pci_dev *dev, int bar,
+                                            unsigned offset,
+                                            unsigned long minlen,
+                                            unsigned long maxlen)
+{
+	return NULL;
+}
 struct pci_dev;
 static inline void __iomem *pci_iomap(struct pci_dev *dev, int bar, unsigned long max)
 {
diff --git a/include/linux/pci_regs.h b/include/linux/pci_regs.h
index e884096..2ec9f81 100644
--- a/include/linux/pci_regs.h
+++ b/include/linux/pci_regs.h
@@ -375,6 +375,10 @@
 #define  PCI_X_STATUS_266MHZ	0x40000000	/* 266 MHz capable */
 #define  PCI_X_STATUS_533MHZ	0x80000000	/* 533 MHz capable */
 
+/* Vendor specific capability */
+#define PCI_VNDR_CAP_LEN	2	/* Capability length (8 bits), including
+					   bytes: ID, NEXT and LEN itself. */
+
 /* PCI Bridge Subsystem ID registers */
 
 #define PCI_SSVID_VENDOR_ID     4	/* PCI-Bridge subsystem vendor id register */
diff --git a/lib/iomap.c b/lib/iomap.c
index 5dbcb4b..f28720e 100644
--- a/lib/iomap.c
+++ b/lib/iomap.c
@@ -243,26 +243,36 @@ EXPORT_SYMBOL(ioport_unmap);
 
 #ifdef CONFIG_PCI
 /**
- * pci_iomap - create a virtual mapping cookie for a PCI BAR
+ * pci_iomap_range - create a virtual mapping cookie for a PCI BAR
  * @dev: PCI device that owns the BAR
  * @bar: BAR number
- * @maxlen: length of the memory to map
+ * @offset: map memory at the given offset in BAR
+ * @minlen: min length of the memory to map
+ * @maxlen: max length of the memory to map
  *
  * Using this function you will get a __iomem address to your device BAR.
  * You can access it using ioread*() and iowrite*(). These functions hide
  * the details if this is a MMIO or PIO address space and will just do what
  * you expect from them in the correct way.
  *
+ * @minlen specifies the minimum length to map. We check that BAR is
+ * large enough.
  * @maxlen specifies the maximum length to map. If you want to get access to
- * the complete BAR without checking for its length first, pass %0 here.
+ * the complete BAR from offset to the end, pass %0 here.
  * */
-void __iomem *pci_iomap(struct pci_dev *dev, int bar, unsigned long maxlen)
+void __iomem *pci_iomap_range(struct pci_dev *dev, int bar,
+                              unsigned offset,
+                              unsigned long minlen,
+                              unsigned long maxlen)
 {
 	resource_size_t start = pci_resource_start(dev, bar);
 	resource_size_t len = pci_resource_len(dev, bar);
 	unsigned long flags = pci_resource_flags(dev, bar);
 
-	if (!len || !start)
+	if (len <= offset || !start)
+		return NULL;
+        len -= offset;
+        if (len < minlen)
 		return NULL;
 	if (maxlen && len > maxlen)
 		len = maxlen;
@@ -277,10 +287,30 @@ void __iomem *pci_iomap(struct pci_dev *dev, int bar, unsigned long maxlen)
 	return NULL;
 }
 
+/**
+ * pci_iomap - create a virtual mapping cookie for a PCI BAR
+ * @dev: PCI device that owns the BAR
+ * @bar: BAR number
+ * @maxlen: length of the memory to map
+ *
+ * Using this function you will get a __iomem address to your device BAR.
+ * You can access it using ioread*() and iowrite*(). These functions hide
+ * the details if this is a MMIO or PIO address space and will just do what
+ * you expect from them in the correct way.
+ *
+ * @maxlen specifies the maximum length to map. If you want to get access to
+ * the complete BAR without checking for its length first, pass %0 here.
+ * */
+void __iomem *pci_iomap(struct pci_dev *dev, int bar, unsigned long maxlen)
+{
+    return pci_iomap_range(dev, bar, 0, 0, maxlen);
+}
+
 void pci_iounmap(struct pci_dev *dev, void __iomem * addr)
 {
 	IO_COND(addr, /* nothing */, iounmap(addr));
 }
 EXPORT_SYMBOL(pci_iomap);
+EXPORT_SYMBOL(pci_iomap_range);
 EXPORT_SYMBOL(pci_iounmap);
 #endif /* CONFIG_PCI */

-- 
MST

Re: [PATCH RFC] virtio-pci: flexible configuration layout

[Sasha Levin]

Hi Michael,

On Thu, 2011-11-03 at 01:31 +0200, Michael S. Tsirkin wrote:
> Add a flexible mechanism to specify virtio configuration layout, using
> pci vendor-specific capability.  A separate capability is used for each
> of common, device specific and data-path accesses.
> 
> Warning: compiled only.
> This patch also needs to be split up, pci_iomap changes
> also need arch updates for non-x86.
> 
> We also will need to update the spec.
> 
> See the first chunk for layout documentation.
> 
> Posting here for early feedback.
> 
> In particular:
> 
> Do we need to require offset to be aligned?
> Does iowrite16 work with unaligned accesses on all architectures?
> Does using ioread/write as we do add overhead as compared to
> plain PIO accesses?
> 
> Jesse - are you OK with the pci_iomap_range API proposed here
> (see last chunks)?
> I noticed lots of architectures duplicate the implementation
> of pci_iomap - makes sense to clean that up?
> 
> Thanks,
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> 
> ---
> 
> diff --git a/include/linux/virtio_pci.h b/include/linux/virtio_pci.h
> index ea66f3f..b8a9de2 100644
> --- a/include/linux/virtio_pci.h
> +++ b/include/linux/virtio_pci.h
> @@ -92,4 +92,38 @@
>  /* The alignment to use between consumer and producer parts of vring.
>   * x86 pagesize again. */
>  #define VIRTIO_PCI_VRING_ALIGN		4096
> +
> +/*
> + * Layout for Virtio PCI vendor specific capability (little-endian):
> + * 5 bit virtio capability id.
> + * 3 bit BAR index register, specifying which BAR to use.
> + * 4 byte cfg offset within the BAR.

Why not make the 3 bits of the BIR as the lower 3 bits of the offset
like its done in the PCI spec?

This way you also get 'free' 32bit alignment and simpler masking.

> + * 4 byte cfg size.
> + */
> +
> +/* A single virtio device has multiple vendor specific capabilities, we use the
> + * 5 bit ID field to distinguish between these. */
> +#define VIRTIO_PCI_CAP_ID		3
> +#define VIRTIO_PCI_CAP_ID_MASK		0x1f
> +#define VIRTIO_PCI_CAP_ID_SHIFT		0
> +
> +/* IDs for different capabilities. If a specific configuration
> + * is missing, legacy PIO path is used. */
> +/* Common configuration */
> +#define VIRTIO_PCI_CAP_COMMON_CFG	0
> +/* Device specific confiuration */
> +#define VIRTIO_PCI_CAP_DEVICE_CFG	1
> +/* Notifications and ISR access */
> +#define VIRTIO_PCI_CAP_NOTIFY_CFG	2

I think that separating notification here is very kvm specific. Maybe it
could be replaced by allowing virtio to signal eventfds or something
similar?

> +
> +/* Index of the BAR including this configuration */
> +#define VIRTIO_PCI_CAP_CFG_BIR		3
> +#define VIRTIO_PCI_CAP_CFG_BIR_MASK	(0x7)
> +#define VIRTIO_PCI_CAP_CFG_BIR_SHIFT	5
> +
> +/* Offset within the BAR */
> +#define VIRTIO_PCI_CAP_CFG_OFF		4
> +/* Size of the configuration space */
> +#define VIRTIO_PCI_CAP_CFG_SIZE		8
> +
>  #endif
> diff --git a/drivers/virtio/virtio_pci.c b/drivers/virtio/virtio_pci.c
> index 4bcc8b8..d4ed130 100644
> --- a/drivers/virtio/virtio_pci.c
> +++ b/drivers/virtio/virtio_pci.c
> @@ -37,8 +37,12 @@ struct virtio_pci_device
>  	struct virtio_device vdev;
>  	struct pci_dev *pci_dev;
>  
> -	/* the IO mapping for the PCI config space */
> +	/* the IO address for the common PCI config space */
>  	void __iomem *ioaddr;
> +	/* the IO address for device specific config */
> +	void __iomem *ioaddr_device;
> +	/* the IO address to use for notifications operations */
> +	void __iomem *ioaddr_notify;
>  
>  	/* a list of queues so we can dispatch IRQs */
>  	spinlock_t lock;
> @@ -57,8 +61,142 @@ struct virtio_pci_device
>  	unsigned msix_used_vectors;
>  	/* Whether we have vector per vq */
>  	bool per_vq_vectors;
> +
> +	/* Various IO mappings: used for resource tracking only. */
> +
> +	/* Legacy BAR0: typically PIO. */
> +	void __iomem *legacy_map;
> +
> +	/* Mappings specified by device capabilities: typically in MMIO */
> +	void __iomem *notify_map;
> +	void __iomem *common_map;
> +	void __iomem *device_map;
>  };
>  
> +/*
> + * With PIO, device-specific config moves as MSI-X is enabled/disabled.
> + * Use this accessor to keep pointer to that config in sync.
> + */
> +static void virtio_pci_set_msix_enabled(struct virtio_pci_device *vp_dev, int enabled)
> +{
> +	vp_dev->msix_enabled = enabled;
> +	if (vp_dev->device_map)
> +		vp_dev->ioaddr_device = vp_dev->device_map;
> +	else
> +		vp_dev->ioaddr_device = vp_dev->legacy_map +
> +			VIRTIO_PCI_CONFIG(vp_dev);
> +}
> +
> +static void __iomem *virtio_pci_map_cfg(struct virtio_pci_device *vp_dev, u8 cap_id)
> +{
> +        u32 size;
> +        u32 offset;
> +        u8 bir;
> +        u8 cap_len;
> +	int pos;
> +	struct pci_dev *dev = vp_dev->pci_dev;
> +	void __iomem *p;
> +
> +	for (pos = pci_find_capability(dev, PCI_CAP_ID_VNDR);
> +	     pos > 0;
> +	     pos = pci_find_next_capability(dev, pos, PCI_CAP_ID_VNDR)) {
> +		u8 id;
> +		pci_read_config_byte(dev, pos + PCI_VNDR_CAP_LEN, &cap_len);
> +		if (cap_len < VIRTIO_PCI_CAP_ID + 1)
> +			continue;
> +		pci_read_config_byte(dev, pos + VIRTIO_PCI_CAP_ID, &id);
> +		id >>= VIRTIO_PCI_CAP_ID_SHIFT;
> +		id &= VIRTIO_PCI_CAP_ID_MASK;
> +		if (id == cap_id)
> +			break;
> +	}
> +
> +	if (pos <= 0)
> +		return NULL;
> +
> +	if (cap_len < VIRTIO_PCI_CAP_CFG_BIR + 1)
> +		goto err;
> +        pci_read_config_byte(dev, pos + VIRTIO_PCI_CAP_CFG_BIR, &bir);
> +	if (cap_len < VIRTIO_PCI_CAP_CFG_OFF + 4)
> +		goto err;
> +        pci_read_config_dword(dev, pos + VIRTIO_PCI_CAP_CFG_OFF, &offset);
> +	if (cap_len < VIRTIO_PCI_CAP_CFG_SIZE + 4)
> +		goto err;
> +        pci_read_config_dword(dev, pos + VIRTIO_PCI_CAP_CFG_SIZE, &size);
> +        bir >>= VIRTIO_PCI_CAP_CFG_BIR_SHIFT;
> +        bir &= VIRTIO_PCI_CAP_CFG_BIR_MASK;
> +
> +	/* It's possible for a device to supply a huge config space,
> +	 * but we'll never need to map more than a page ATM. */
> +	p = pci_iomap_range(dev, bir, offset, size, PAGE_SIZE);
> +	if (!p)
> +		dev_err(&vp_dev->vdev.dev, "Unable to map virtio pci memory");
> +	return p;
> +err:
> +	dev_err(&vp_dev->vdev.dev, "Unable to parse virtio pci capability");
> +	return NULL;
> +}
> +
> +static void virtio_pci_iounmap(struct virtio_pci_device *vp_dev)
> +{
> +	if (vp_dev->legacy_map)
> +		pci_iounmap(vp_dev->pci_dev, vp_dev->legacy_map);
> +	if (vp_dev->notify_map)
> +		pci_iounmap(vp_dev->pci_dev, vp_dev->notify_map);
> +	if (vp_dev->common_map)
> +		pci_iounmap(vp_dev->pci_dev, vp_dev->common_map);
> +	if (vp_dev->device_map)
> +		pci_iounmap(vp_dev->pci_dev, vp_dev->device_map);
> +}
> +
> +static int virtio_pci_iomap(struct virtio_pci_device *vp_dev)
> +{
> +	vp_dev->notify_map = virtio_pci_map_cfg(vp_dev,
> +						     VIRTIO_PCI_CAP_NOTIFY_CFG);
> +	vp_dev->common_map = virtio_pci_map_cfg(vp_dev,
> +						     VIRTIO_PCI_CAP_COMMON_CFG);
> +	vp_dev->device_map = virtio_pci_map_cfg(vp_dev,
> +						     VIRTIO_PCI_CAP_DEVICE_CFG);
> +
> +	if (!vp_dev->notify_map || !vp_dev->common_map ||
> +	    !vp_dev->device_map) {
> +		/*
> +		 * If not all capabilities present, map legacy PIO.
> +		 * Legacy access is at BAR 0. We never need to map
> +		 * more than 256 bytes there, since legacy config space
> +		 * used PIO which has this size limit.
> +		 * */

Whats the point of putting each of those new configs into a separate cap
structure if you must have all 3 of them for the new config layout to
work anyway?

Also, why not use this change to simplify the config space? The attempts
to squeeze it into the limited PIO space made it quite messy, and this
can be a good chance to fix some things like the MSI-X and offsets
issue.

-- 

Sasha.

Re: [PATCH RFC] virtio-pci: flexible configuration layout

[Michael S. Tsirkin]

On Thu, Nov 03, 2011 at 02:19:03AM +0200, Sasha Levin wrote:
> Hi Michael,
> 
> On Thu, 2011-11-03 at 01:31 +0200, Michael S. Tsirkin wrote:
> > Add a flexible mechanism to specify virtio configuration layout, using
> > pci vendor-specific capability.  A separate capability is used for each
> > of common, device specific and data-path accesses.
> > 
> > Warning: compiled only.
> > This patch also needs to be split up, pci_iomap changes
> > also need arch updates for non-x86.
> > 
> > We also will need to update the spec.
> > 
> > See the first chunk for layout documentation.
> > 
> > Posting here for early feedback.
> > 
> > In particular:
> > 
> > Do we need to require offset to be aligned?
> > Does iowrite16 work with unaligned accesses on all architectures?
> > Does using ioread/write as we do add overhead as compared to
> > plain PIO accesses?
> > 
> > Jesse - are you OK with the pci_iomap_range API proposed here
> > (see last chunks)?
> > I noticed lots of architectures duplicate the implementation
> > of pci_iomap - makes sense to clean that up?
> > 
> > Thanks,
> > 
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > 
> > ---
> > 
> > diff --git a/include/linux/virtio_pci.h b/include/linux/virtio_pci.h
> > index ea66f3f..b8a9de2 100644
> > --- a/include/linux/virtio_pci.h
> > +++ b/include/linux/virtio_pci.h
> > @@ -92,4 +92,38 @@
> >  /* The alignment to use between consumer and producer parts of vring.
> >   * x86 pagesize again. */
> >  #define VIRTIO_PCI_VRING_ALIGN		4096
> > +
> > +/*
> > + * Layout for Virtio PCI vendor specific capability (little-endian):
> > + * 5 bit virtio capability id.
> > + * 3 bit BAR index register, specifying which BAR to use.
> > + * 4 byte cfg offset within the BAR.
> 
> Why not make the 3 bits of the BIR as the lower 3 bits of the offset
> like its done in the PCI spec?
> 
> This way you also get 'free' 32bit alignment and simpler masking.

3 bit gets us 64 bit alignment, no? Seems a bit more than we need:
we need at most 16 bit alignment.

> > + * 4 byte cfg size.
> > + */
> > +
> > +/* A single virtio device has multiple vendor specific capabilities, we use the
> > + * 5 bit ID field to distinguish between these. */
> > +#define VIRTIO_PCI_CAP_ID		3
> > +#define VIRTIO_PCI_CAP_ID_MASK		0x1f
> > +#define VIRTIO_PCI_CAP_ID_SHIFT		0
> > +
> > +/* IDs for different capabilities. If a specific configuration
> > + * is missing, legacy PIO path is used. */
> > +/* Common configuration */
> > +#define VIRTIO_PCI_CAP_COMMON_CFG	0
> > +/* Device specific confiuration */
> > +#define VIRTIO_PCI_CAP_DEVICE_CFG	1
> > +/* Notifications and ISR access */
> > +#define VIRTIO_PCI_CAP_NOTIFY_CFG	2
> 
> I think that separating notification here is very kvm specific.

I don't see how it's kvm specific. This just makes it possible to split data path and
configuration. Good for security as well. I was actually debating a
separate configuration for ISR and NOTIFICATION.

> Maybe it
> could be replaced by allowing virtio to signal eventfds or something
> similar?

No idea what you mean - it's a pci device, how should it signal eventfds?

> > +
> > +/* Index of the BAR including this configuration */
> > +#define VIRTIO_PCI_CAP_CFG_BIR		3
> > +#define VIRTIO_PCI_CAP_CFG_BIR_MASK	(0x7)
> > +#define VIRTIO_PCI_CAP_CFG_BIR_SHIFT	5
> > +
> > +/* Offset within the BAR */
> > +#define VIRTIO_PCI_CAP_CFG_OFF		4
> > +/* Size of the configuration space */
> > +#define VIRTIO_PCI_CAP_CFG_SIZE		8
> > +
> >  #endif
> > diff --git a/drivers/virtio/virtio_pci.c b/drivers/virtio/virtio_pci.c
> > index 4bcc8b8..d4ed130 100644
> > --- a/drivers/virtio/virtio_pci.c
> > +++ b/drivers/virtio/virtio_pci.c
> > @@ -37,8 +37,12 @@ struct virtio_pci_device
> >  	struct virtio_device vdev;
> >  	struct pci_dev *pci_dev;
> >  
> > -	/* the IO mapping for the PCI config space */
> > +	/* the IO address for the common PCI config space */
> >  	void __iomem *ioaddr;
> > +	/* the IO address for device specific config */
> > +	void __iomem *ioaddr_device;
> > +	/* the IO address to use for notifications operations */
> > +	void __iomem *ioaddr_notify;
> >  
> >  	/* a list of queues so we can dispatch IRQs */
> >  	spinlock_t lock;
> > @@ -57,8 +61,142 @@ struct virtio_pci_device
> >  	unsigned msix_used_vectors;
> >  	/* Whether we have vector per vq */
> >  	bool per_vq_vectors;
> > +
> > +	/* Various IO mappings: used for resource tracking only. */
> > +
> > +	/* Legacy BAR0: typically PIO. */
> > +	void __iomem *legacy_map;
> > +
> > +	/* Mappings specified by device capabilities: typically in MMIO */
> > +	void __iomem *notify_map;
> > +	void __iomem *common_map;
> > +	void __iomem *device_map;
> >  };
> >  
> > +/*
> > + * With PIO, device-specific config moves as MSI-X is enabled/disabled.
> > + * Use this accessor to keep pointer to that config in sync.
> > + */
> > +static void virtio_pci_set_msix_enabled(struct virtio_pci_device *vp_dev, int enabled)
> > +{
> > +	vp_dev->msix_enabled = enabled;
> > +	if (vp_dev->device_map)
> > +		vp_dev->ioaddr_device = vp_dev->device_map;
> > +	else
> > +		vp_dev->ioaddr_device = vp_dev->legacy_map +
> > +			VIRTIO_PCI_CONFIG(vp_dev);
> > +}
> > +
> > +static void __iomem *virtio_pci_map_cfg(struct virtio_pci_device *vp_dev, u8 cap_id)
> > +{
> > +        u32 size;
> > +        u32 offset;
> > +        u8 bir;
> > +        u8 cap_len;
> > +	int pos;
> > +	struct pci_dev *dev = vp_dev->pci_dev;
> > +	void __iomem *p;
> > +
> > +	for (pos = pci_find_capability(dev, PCI_CAP_ID_VNDR);
> > +	     pos > 0;
> > +	     pos = pci_find_next_capability(dev, pos, PCI_CAP_ID_VNDR)) {
> > +		u8 id;
> > +		pci_read_config_byte(dev, pos + PCI_VNDR_CAP_LEN, &cap_len);
> > +		if (cap_len < VIRTIO_PCI_CAP_ID + 1)
> > +			continue;
> > +		pci_read_config_byte(dev, pos + VIRTIO_PCI_CAP_ID, &id);
> > +		id >>= VIRTIO_PCI_CAP_ID_SHIFT;
> > +		id &= VIRTIO_PCI_CAP_ID_MASK;
> > +		if (id == cap_id)
> > +			break;
> > +	}
> > +
> > +	if (pos <= 0)
> > +		return NULL;
> > +
> > +	if (cap_len < VIRTIO_PCI_CAP_CFG_BIR + 1)
> > +		goto err;
> > +        pci_read_config_byte(dev, pos + VIRTIO_PCI_CAP_CFG_BIR, &bir);
> > +	if (cap_len < VIRTIO_PCI_CAP_CFG_OFF + 4)
> > +		goto err;
> > +        pci_read_config_dword(dev, pos + VIRTIO_PCI_CAP_CFG_OFF, &offset);
> > +	if (cap_len < VIRTIO_PCI_CAP_CFG_SIZE + 4)
> > +		goto err;
> > +        pci_read_config_dword(dev, pos + VIRTIO_PCI_CAP_CFG_SIZE, &size);
> > +        bir >>= VIRTIO_PCI_CAP_CFG_BIR_SHIFT;
> > +        bir &= VIRTIO_PCI_CAP_CFG_BIR_MASK;
> > +
> > +	/* It's possible for a device to supply a huge config space,
> > +	 * but we'll never need to map more than a page ATM. */
> > +	p = pci_iomap_range(dev, bir, offset, size, PAGE_SIZE);
> > +	if (!p)
> > +		dev_err(&vp_dev->vdev.dev, "Unable to map virtio pci memory");
> > +	return p;
> > +err:
> > +	dev_err(&vp_dev->vdev.dev, "Unable to parse virtio pci capability");
> > +	return NULL;
> > +}
> > +
> > +static void virtio_pci_iounmap(struct virtio_pci_device *vp_dev)
> > +{
> > +	if (vp_dev->legacy_map)
> > +		pci_iounmap(vp_dev->pci_dev, vp_dev->legacy_map);
> > +	if (vp_dev->notify_map)
> > +		pci_iounmap(vp_dev->pci_dev, vp_dev->notify_map);
> > +	if (vp_dev->common_map)
> > +		pci_iounmap(vp_dev->pci_dev, vp_dev->common_map);
> > +	if (vp_dev->device_map)
> > +		pci_iounmap(vp_dev->pci_dev, vp_dev->device_map);
> > +}
> > +
> > +static int virtio_pci_iomap(struct virtio_pci_device *vp_dev)
> > +{
> > +	vp_dev->notify_map = virtio_pci_map_cfg(vp_dev,
> > +						     VIRTIO_PCI_CAP_NOTIFY_CFG);
> > +	vp_dev->common_map = virtio_pci_map_cfg(vp_dev,
> > +						     VIRTIO_PCI_CAP_COMMON_CFG);
> > +	vp_dev->device_map = virtio_pci_map_cfg(vp_dev,
> > +						     VIRTIO_PCI_CAP_DEVICE_CFG);
> > +
> > +	if (!vp_dev->notify_map || !vp_dev->common_map ||
> > +	    !vp_dev->device_map) {
> > +		/*
> > +		 * If not all capabilities present, map legacy PIO.
> > +		 * Legacy access is at BAR 0. We never need to map
> > +		 * more than 256 bytes there, since legacy config space
> > +		 * used PIO which has this size limit.
> > +		 * */
> 
> Whats the point of putting each of those new configs into a separate cap
> structure if you must have all 3 of them for the new config layout to
> work anyway?

Pls look closer, we don't require all 3 of them. For example, we can have BAR0
DEVICE and COMMON, and notifications will use BAR0.

By using an identical structure we make is much easier to reuse
and extend it. For example we might add a separate structure for ISR.

> Also, why not use this change to simplify the config space? The attempts
> to squeeze it into the limited PIO space made it quite messy, and this
> can be a good chance to fix some things like the MSI-X and offsets
> issue.

For MSI-X this is exactly what this change does, by splitting
device-specific parts out. So if DEVICE capability is present,
this tells us where the device specific data starts and
enabling MSI-X does not relocate it.

Are there other fields we want to move?

> 
> -- 
> 
> Sasha.

Re: [PATCH RFC] virtio-pci: flexible configuration layout

[Avi Kivity]

On 11/03/2011 01:31 AM, Michael S. Tsirkin wrote:
> Add a flexible mechanism to specify virtio configuration layout, using
> pci vendor-specific capability.  A separate capability is used for each
> of common, device specific and data-path accesses.
>
>

How about posting the spec change instead of patches?

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH RFC] virtio-pci: flexible configuration layout

[Sasha Levin]

On Thu, 2011-11-03 at 12:33 +0200, Michael S. Tsirkin wrote:
> On Thu, Nov 03, 2011 at 02:19:03AM +0200, Sasha Levin wrote:
> > Hi Michael,
> > 
> > On Thu, 2011-11-03 at 01:31 +0200, Michael S. Tsirkin wrote:
> > > Add a flexible mechanism to specify virtio configuration layout, using
> > > pci vendor-specific capability.  A separate capability is used for each
> > > of common, device specific and data-path accesses.
> > > 
> > > Warning: compiled only.
> > > This patch also needs to be split up, pci_iomap changes
> > > also need arch updates for non-x86.
> > > 
> > > We also will need to update the spec.
> > > 
> > > See the first chunk for layout documentation.
> > > 
> > > Posting here for early feedback.
> > > 
> > > In particular:
> > > 
> > > Do we need to require offset to be aligned?
> > > Does iowrite16 work with unaligned accesses on all architectures?
> > > Does using ioread/write as we do add overhead as compared to
> > > plain PIO accesses?
> > > 
> > > Jesse - are you OK with the pci_iomap_range API proposed here
> > > (see last chunks)?
> > > I noticed lots of architectures duplicate the implementation
> > > of pci_iomap - makes sense to clean that up?
> > > 
> > > Thanks,
> > > 
> > > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > > 
> > > ---
> > > 
> > > diff --git a/include/linux/virtio_pci.h b/include/linux/virtio_pci.h
> > > index ea66f3f..b8a9de2 100644
> > > --- a/include/linux/virtio_pci.h
> > > +++ b/include/linux/virtio_pci.h
> > > @@ -92,4 +92,38 @@
> > >  /* The alignment to use between consumer and producer parts of vring.
> > >   * x86 pagesize again. */
> > >  #define VIRTIO_PCI_VRING_ALIGN		4096
> > > +
> > > +/*
> > > + * Layout for Virtio PCI vendor specific capability (little-endian):
> > > + * 5 bit virtio capability id.
> > > + * 3 bit BAR index register, specifying which BAR to use.
> > > + * 4 byte cfg offset within the BAR.
> > 
> > Why not make the 3 bits of the BIR as the lower 3 bits of the offset
> > like its done in the PCI spec?
> > 
> > This way you also get 'free' 32bit alignment and simpler masking.
> 
> 3 bit gets us 64 bit alignment, no? Seems a bit more than we need:
> we need at most 16 bit alignment.

Yup, it's actually 64 bit, but whats the problem with that? It will also
look more similar to the PCI spec.

> 
> > > + * 4 byte cfg size.
> > > + */
> > > +
> > > +/* A single virtio device has multiple vendor specific capabilities, we use the
> > > + * 5 bit ID field to distinguish between these. */
> > > +#define VIRTIO_PCI_CAP_ID		3
> > > +#define VIRTIO_PCI_CAP_ID_MASK		0x1f
> > > +#define VIRTIO_PCI_CAP_ID_SHIFT		0
> > > +
> > > +/* IDs for different capabilities. If a specific configuration
> > > + * is missing, legacy PIO path is used. */
> > > +/* Common configuration */
> > > +#define VIRTIO_PCI_CAP_COMMON_CFG	0
> > > +/* Device specific confiuration */
> > > +#define VIRTIO_PCI_CAP_DEVICE_CFG	1
> > > +/* Notifications and ISR access */
> > > +#define VIRTIO_PCI_CAP_NOTIFY_CFG	2
> > 
> > I think that separating notification here is very kvm specific.
> 
> I don't see how it's kvm specific. This just makes it possible to split data path and
> configuration. Good for security as well. I was actually debating a
> separate configuration for ISR and NOTIFICATION.

It's not really a clean split between data and config, you still have
device status stuck between notification and ISR, which isn't really in
the data path.

A split as you suggested sounds like a good idea, but we just need to
move some fields around to make it clean.

> > Maybe it
> > could be replaced by allowing virtio to signal eventfds or something
> > similar?
> 
> No idea what you mean - it's a pci device, how should it signal eventfds?

I'm not really sure what I was thinking about, ignore that :)

> 
> > > +
> > > +/* Index of the BAR including this configuration */
> > > +#define VIRTIO_PCI_CAP_CFG_BIR		3
> > > +#define VIRTIO_PCI_CAP_CFG_BIR_MASK	(0x7)
> > > +#define VIRTIO_PCI_CAP_CFG_BIR_SHIFT	5
> > > +
> > > +/* Offset within the BAR */
> > > +#define VIRTIO_PCI_CAP_CFG_OFF		4
> > > +/* Size of the configuration space */
> > > +#define VIRTIO_PCI_CAP_CFG_SIZE		8
> > > +
> > >  #endif
> > > diff --git a/drivers/virtio/virtio_pci.c b/drivers/virtio/virtio_pci.c
> > > index 4bcc8b8..d4ed130 100644
> > > --- a/drivers/virtio/virtio_pci.c
> > > +++ b/drivers/virtio/virtio_pci.c
> > > @@ -37,8 +37,12 @@ struct virtio_pci_device
> > >  	struct virtio_device vdev;
> > >  	struct pci_dev *pci_dev;
> > >  
> > > -	/* the IO mapping for the PCI config space */
> > > +	/* the IO address for the common PCI config space */
> > >  	void __iomem *ioaddr;
> > > +	/* the IO address for device specific config */
> > > +	void __iomem *ioaddr_device;
> > > +	/* the IO address to use for notifications operations */
> > > +	void __iomem *ioaddr_notify;
> > >  
> > >  	/* a list of queues so we can dispatch IRQs */
> > >  	spinlock_t lock;
> > > @@ -57,8 +61,142 @@ struct virtio_pci_device
> > >  	unsigned msix_used_vectors;
> > >  	/* Whether we have vector per vq */
> > >  	bool per_vq_vectors;
> > > +
> > > +	/* Various IO mappings: used for resource tracking only. */
> > > +
> > > +	/* Legacy BAR0: typically PIO. */
> > > +	void __iomem *legacy_map;
> > > +
> > > +	/* Mappings specified by device capabilities: typically in MMIO */
> > > +	void __iomem *notify_map;
> > > +	void __iomem *common_map;
> > > +	void __iomem *device_map;
> > >  };
> > >  
> > > +/*
> > > + * With PIO, device-specific config moves as MSI-X is enabled/disabled.
> > > + * Use this accessor to keep pointer to that config in sync.
> > > + */
> > > +static void virtio_pci_set_msix_enabled(struct virtio_pci_device *vp_dev, int enabled)
> > > +{
> > > +	vp_dev->msix_enabled = enabled;
> > > +	if (vp_dev->device_map)
> > > +		vp_dev->ioaddr_device = vp_dev->device_map;
> > > +	else
> > > +		vp_dev->ioaddr_device = vp_dev->legacy_map +
> > > +			VIRTIO_PCI_CONFIG(vp_dev);
> > > +}
> > > +
> > > +static void __iomem *virtio_pci_map_cfg(struct virtio_pci_device *vp_dev, u8 cap_id)
> > > +{
> > > +        u32 size;
> > > +        u32 offset;
> > > +        u8 bir;
> > > +        u8 cap_len;
> > > +	int pos;
> > > +	struct pci_dev *dev = vp_dev->pci_dev;
> > > +	void __iomem *p;
> > > +
> > > +	for (pos = pci_find_capability(dev, PCI_CAP_ID_VNDR);
> > > +	     pos > 0;
> > > +	     pos = pci_find_next_capability(dev, pos, PCI_CAP_ID_VNDR)) {
> > > +		u8 id;
> > > +		pci_read_config_byte(dev, pos + PCI_VNDR_CAP_LEN, &cap_len);
> > > +		if (cap_len < VIRTIO_PCI_CAP_ID + 1)
> > > +			continue;
> > > +		pci_read_config_byte(dev, pos + VIRTIO_PCI_CAP_ID, &id);
> > > +		id >>= VIRTIO_PCI_CAP_ID_SHIFT;
> > > +		id &= VIRTIO_PCI_CAP_ID_MASK;
> > > +		if (id == cap_id)
> > > +			break;
> > > +	}
> > > +
> > > +	if (pos <= 0)
> > > +		return NULL;
> > > +
> > > +	if (cap_len < VIRTIO_PCI_CAP_CFG_BIR + 1)
> > > +		goto err;
> > > +        pci_read_config_byte(dev, pos + VIRTIO_PCI_CAP_CFG_BIR, &bir);
> > > +	if (cap_len < VIRTIO_PCI_CAP_CFG_OFF + 4)
> > > +		goto err;
> > > +        pci_read_config_dword(dev, pos + VIRTIO_PCI_CAP_CFG_OFF, &offset);
> > > +	if (cap_len < VIRTIO_PCI_CAP_CFG_SIZE + 4)
> > > +		goto err;
> > > +        pci_read_config_dword(dev, pos + VIRTIO_PCI_CAP_CFG_SIZE, &size);
> > > +        bir >>= VIRTIO_PCI_CAP_CFG_BIR_SHIFT;
> > > +        bir &= VIRTIO_PCI_CAP_CFG_BIR_MASK;
> > > +
> > > +	/* It's possible for a device to supply a huge config space,
> > > +	 * but we'll never need to map more than a page ATM. */
> > > +	p = pci_iomap_range(dev, bir, offset, size, PAGE_SIZE);
> > > +	if (!p)
> > > +		dev_err(&vp_dev->vdev.dev, "Unable to map virtio pci memory");
> > > +	return p;
> > > +err:
> > > +	dev_err(&vp_dev->vdev.dev, "Unable to parse virtio pci capability");
> > > +	return NULL;
> > > +}
> > > +
> > > +static void virtio_pci_iounmap(struct virtio_pci_device *vp_dev)
> > > +{
> > > +	if (vp_dev->legacy_map)
> > > +		pci_iounmap(vp_dev->pci_dev, vp_dev->legacy_map);
> > > +	if (vp_dev->notify_map)
> > > +		pci_iounmap(vp_dev->pci_dev, vp_dev->notify_map);
> > > +	if (vp_dev->common_map)
> > > +		pci_iounmap(vp_dev->pci_dev, vp_dev->common_map);
> > > +	if (vp_dev->device_map)
> > > +		pci_iounmap(vp_dev->pci_dev, vp_dev->device_map);
> > > +}
> > > +
> > > +static int virtio_pci_iomap(struct virtio_pci_device *vp_dev)
> > > +{
> > > +	vp_dev->notify_map = virtio_pci_map_cfg(vp_dev,
> > > +						     VIRTIO_PCI_CAP_NOTIFY_CFG);
> > > +	vp_dev->common_map = virtio_pci_map_cfg(vp_dev,
> > > +						     VIRTIO_PCI_CAP_COMMON_CFG);
> > > +	vp_dev->device_map = virtio_pci_map_cfg(vp_dev,
> > > +						     VIRTIO_PCI_CAP_DEVICE_CFG);
> > > +
> > > +	if (!vp_dev->notify_map || !vp_dev->common_map ||
> > > +	    !vp_dev->device_map) {
> > > +		/*
> > > +		 * If not all capabilities present, map legacy PIO.
> > > +		 * Legacy access is at BAR 0. We never need to map
> > > +		 * more than 256 bytes there, since legacy config space
> > > +		 * used PIO which has this size limit.
> > > +		 * */
> > 
> > Whats the point of putting each of those new configs into a separate cap
> > structure if you must have all 3 of them for the new config layout to
> > work anyway?
> 
> Pls look closer, we don't require all 3 of them. For example, we can have BAR0
> DEVICE and COMMON, and notifications will use BAR0.
> 
> By using an identical structure we make is much easier to reuse
> and extend it. For example we might add a separate structure for ISR.

Right.

> > Also, why not use this change to simplify the config space? The attempts
> > to squeeze it into the limited PIO space made it quite messy, and this
> > can be a good chance to fix some things like the MSI-X and offsets
> > issue.
> 
> For MSI-X this is exactly what this change does, by splitting
> device-specific parts out. So if DEVICE capability is present,
> this tells us where the device specific data starts and
> enabling MSI-X does not relocate it.

You still have relocations within the shared config space, once 64bit
features or other future extensions are implemented.

> Are there other fields we want to move?

We can do several things:

1. Since we're no longer limited in space - remove queue selector and
just have a full queue table.

2. Move device specific features into the device specific region.
Currently the features field is a mix between virtio-pci and device
specific features.

I did more similar things in the spec suggestion I've sent, could you
look at the second part for more ideas please?

-- 

Sasha.

Re: [PATCH RFC] virtio-pci: flexible configuration layout

[Michael S. Tsirkin]

On Thu, Nov 03, 2011 at 01:09:51PM +0200, Sasha Levin wrote:
> On Thu, 2011-11-03 at 12:33 +0200, Michael S. Tsirkin wrote:
> > On Thu, Nov 03, 2011 at 02:19:03AM +0200, Sasha Levin wrote:
> > > Hi Michael,
> > > 
> > > On Thu, 2011-11-03 at 01:31 +0200, Michael S. Tsirkin wrote:
> > > > Add a flexible mechanism to specify virtio configuration layout, using
> > > > pci vendor-specific capability.  A separate capability is used for each
> > > > of common, device specific and data-path accesses.
> > > > 
> > > > Warning: compiled only.
> > > > This patch also needs to be split up, pci_iomap changes
> > > > also need arch updates for non-x86.
> > > > 
> > > > We also will need to update the spec.
> > > > 
> > > > See the first chunk for layout documentation.
> > > > 
> > > > Posting here for early feedback.
> > > > 
> > > > In particular:
> > > > 
> > > > Do we need to require offset to be aligned?
> > > > Does iowrite16 work with unaligned accesses on all architectures?
> > > > Does using ioread/write as we do add overhead as compared to
> > > > plain PIO accesses?
> > > > 
> > > > Jesse - are you OK with the pci_iomap_range API proposed here
> > > > (see last chunks)?
> > > > I noticed lots of architectures duplicate the implementation
> > > > of pci_iomap - makes sense to clean that up?
> > > > 
> > > > Thanks,
> > > > 
> > > > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > > > 
> > > > ---
> > > > 
> > > > diff --git a/include/linux/virtio_pci.h b/include/linux/virtio_pci.h
> > > > index ea66f3f..b8a9de2 100644
> > > > --- a/include/linux/virtio_pci.h
> > > > +++ b/include/linux/virtio_pci.h
> > > > @@ -92,4 +92,38 @@
> > > >  /* The alignment to use between consumer and producer parts of vring.
> > > >   * x86 pagesize again. */
> > > >  #define VIRTIO_PCI_VRING_ALIGN		4096
> > > > +
> > > > +/*
> > > > + * Layout for Virtio PCI vendor specific capability (little-endian):
> > > > + * 5 bit virtio capability id.
> > > > + * 3 bit BAR index register, specifying which BAR to use.
> > > > + * 4 byte cfg offset within the BAR.
> > > 
> > > Why not make the 3 bits of the BIR as the lower 3 bits of the offset
> > > like its done in the PCI spec?
> > > 
> > > This way you also get 'free' 32bit alignment and simpler masking.
> > 
> > 3 bit gets us 64 bit alignment, no? Seems a bit more than we need:
> > we need at most 16 bit alignment.
> 
> Yup, it's actually 64 bit, but whats the problem with that?

Wastes space. Remember we can put stuff in PIO space
which MSI-X doesn't allow, but we need to allow as long
as MMIO is slower.

> It will also
> look more similar to the PCI spec.

You mean MSI-X in the PCI spec, right?
MSI-X needs alignment to function

> > 
> > > > + * 4 byte cfg size.
> > > > + */
> > > > +
> > > > +/* A single virtio device has multiple vendor specific capabilities, we use the
> > > > + * 5 bit ID field to distinguish between these. */
> > > > +#define VIRTIO_PCI_CAP_ID		3
> > > > +#define VIRTIO_PCI_CAP_ID_MASK		0x1f
> > > > +#define VIRTIO_PCI_CAP_ID_SHIFT		0
> > > > +
> > > > +/* IDs for different capabilities. If a specific configuration
> > > > + * is missing, legacy PIO path is used. */
> > > > +/* Common configuration */
> > > > +#define VIRTIO_PCI_CAP_COMMON_CFG	0
> > > > +/* Device specific confiuration */
> > > > +#define VIRTIO_PCI_CAP_DEVICE_CFG	1
> > > > +/* Notifications and ISR access */
> > > > +#define VIRTIO_PCI_CAP_NOTIFY_CFG	2
> > > 
> > > I think that separating notification here is very kvm specific.
> > 
> > I don't see how it's kvm specific. This just makes it possible to split data path and
> > configuration. Good for security as well. I was actually debating a
> > separate configuration for ISR and NOTIFICATION.
> 
> It's not really a clean split between data and config, you still have
> device status stuck between notification and ISR, which isn't really in
> the data path.

No, it's in a separate capability. Look at the patch: device status
access used common_map, ISR and notification notify_map.
They can be in separate BAR or whereever we like.

> A split as you suggested sounds like a good idea, but we just need to
> move some fields around to make it clean.

OK, I'll add an ISR capability. This let us move it around where
we want.

> > > Maybe it
> > > could be replaced by allowing virtio to signal eventfds or something
> > > similar?
> > 
> > No idea what you mean - it's a pci device, how should it signal eventfds?
> 
> I'm not really sure what I was thinking about, ignore that :)
> 
> > 
> > > > +
> > > > +/* Index of the BAR including this configuration */
> > > > +#define VIRTIO_PCI_CAP_CFG_BIR		3
> > > > +#define VIRTIO_PCI_CAP_CFG_BIR_MASK	(0x7)
> > > > +#define VIRTIO_PCI_CAP_CFG_BIR_SHIFT	5
> > > > +
> > > > +/* Offset within the BAR */
> > > > +#define VIRTIO_PCI_CAP_CFG_OFF		4
> > > > +/* Size of the configuration space */
> > > > +#define VIRTIO_PCI_CAP_CFG_SIZE		8
> > > > +
> > > >  #endif
> > > > diff --git a/drivers/virtio/virtio_pci.c b/drivers/virtio/virtio_pci.c
> > > > index 4bcc8b8..d4ed130 100644
> > > > --- a/drivers/virtio/virtio_pci.c
> > > > +++ b/drivers/virtio/virtio_pci.c
> > > > @@ -37,8 +37,12 @@ struct virtio_pci_device
> > > >  	struct virtio_device vdev;
> > > >  	struct pci_dev *pci_dev;
> > > >  
> > > > -	/* the IO mapping for the PCI config space */
> > > > +	/* the IO address for the common PCI config space */
> > > >  	void __iomem *ioaddr;
> > > > +	/* the IO address for device specific config */
> > > > +	void __iomem *ioaddr_device;
> > > > +	/* the IO address to use for notifications operations */
> > > > +	void __iomem *ioaddr_notify;
> > > >  
> > > >  	/* a list of queues so we can dispatch IRQs */
> > > >  	spinlock_t lock;
> > > > @@ -57,8 +61,142 @@ struct virtio_pci_device
> > > >  	unsigned msix_used_vectors;
> > > >  	/* Whether we have vector per vq */
> > > >  	bool per_vq_vectors;
> > > > +
> > > > +	/* Various IO mappings: used for resource tracking only. */
> > > > +
> > > > +	/* Legacy BAR0: typically PIO. */
> > > > +	void __iomem *legacy_map;
> > > > +
> > > > +	/* Mappings specified by device capabilities: typically in MMIO */
> > > > +	void __iomem *notify_map;
> > > > +	void __iomem *common_map;
> > > > +	void __iomem *device_map;
> > > >  };
> > > >  
> > > > +/*
> > > > + * With PIO, device-specific config moves as MSI-X is enabled/disabled.
> > > > + * Use this accessor to keep pointer to that config in sync.
> > > > + */
> > > > +static void virtio_pci_set_msix_enabled(struct virtio_pci_device *vp_dev, int enabled)
> > > > +{
> > > > +	vp_dev->msix_enabled = enabled;
> > > > +	if (vp_dev->device_map)
> > > > +		vp_dev->ioaddr_device = vp_dev->device_map;
> > > > +	else
> > > > +		vp_dev->ioaddr_device = vp_dev->legacy_map +
> > > > +			VIRTIO_PCI_CONFIG(vp_dev);
> > > > +}
> > > > +
> > > > +static void __iomem *virtio_pci_map_cfg(struct virtio_pci_device *vp_dev, u8 cap_id)
> > > > +{
> > > > +        u32 size;
> > > > +        u32 offset;
> > > > +        u8 bir;
> > > > +        u8 cap_len;
> > > > +	int pos;
> > > > +	struct pci_dev *dev = vp_dev->pci_dev;
> > > > +	void __iomem *p;
> > > > +
> > > > +	for (pos = pci_find_capability(dev, PCI_CAP_ID_VNDR);
> > > > +	     pos > 0;
> > > > +	     pos = pci_find_next_capability(dev, pos, PCI_CAP_ID_VNDR)) {
> > > > +		u8 id;
> > > > +		pci_read_config_byte(dev, pos + PCI_VNDR_CAP_LEN, &cap_len);
> > > > +		if (cap_len < VIRTIO_PCI_CAP_ID + 1)
> > > > +			continue;
> > > > +		pci_read_config_byte(dev, pos + VIRTIO_PCI_CAP_ID, &id);
> > > > +		id >>= VIRTIO_PCI_CAP_ID_SHIFT;
> > > > +		id &= VIRTIO_PCI_CAP_ID_MASK;
> > > > +		if (id == cap_id)
> > > > +			break;
> > > > +	}
> > > > +
> > > > +	if (pos <= 0)
> > > > +		return NULL;
> > > > +
> > > > +	if (cap_len < VIRTIO_PCI_CAP_CFG_BIR + 1)
> > > > +		goto err;
> > > > +        pci_read_config_byte(dev, pos + VIRTIO_PCI_CAP_CFG_BIR, &bir);
> > > > +	if (cap_len < VIRTIO_PCI_CAP_CFG_OFF + 4)
> > > > +		goto err;
> > > > +        pci_read_config_dword(dev, pos + VIRTIO_PCI_CAP_CFG_OFF, &offset);
> > > > +	if (cap_len < VIRTIO_PCI_CAP_CFG_SIZE + 4)
> > > > +		goto err;
> > > > +        pci_read_config_dword(dev, pos + VIRTIO_PCI_CAP_CFG_SIZE, &size);
> > > > +        bir >>= VIRTIO_PCI_CAP_CFG_BIR_SHIFT;
> > > > +        bir &= VIRTIO_PCI_CAP_CFG_BIR_MASK;
> > > > +
> > > > +	/* It's possible for a device to supply a huge config space,
> > > > +	 * but we'll never need to map more than a page ATM. */
> > > > +	p = pci_iomap_range(dev, bir, offset, size, PAGE_SIZE);
> > > > +	if (!p)
> > > > +		dev_err(&vp_dev->vdev.dev, "Unable to map virtio pci memory");
> > > > +	return p;
> > > > +err:
> > > > +	dev_err(&vp_dev->vdev.dev, "Unable to parse virtio pci capability");
> > > > +	return NULL;
> > > > +}
> > > > +
> > > > +static void virtio_pci_iounmap(struct virtio_pci_device *vp_dev)
> > > > +{
> > > > +	if (vp_dev->legacy_map)
> > > > +		pci_iounmap(vp_dev->pci_dev, vp_dev->legacy_map);
> > > > +	if (vp_dev->notify_map)
> > > > +		pci_iounmap(vp_dev->pci_dev, vp_dev->notify_map);
> > > > +	if (vp_dev->common_map)
> > > > +		pci_iounmap(vp_dev->pci_dev, vp_dev->common_map);
> > > > +	if (vp_dev->device_map)
> > > > +		pci_iounmap(vp_dev->pci_dev, vp_dev->device_map);
> > > > +}
> > > > +
> > > > +static int virtio_pci_iomap(struct virtio_pci_device *vp_dev)
> > > > +{
> > > > +	vp_dev->notify_map = virtio_pci_map_cfg(vp_dev,
> > > > +						     VIRTIO_PCI_CAP_NOTIFY_CFG);
> > > > +	vp_dev->common_map = virtio_pci_map_cfg(vp_dev,
> > > > +						     VIRTIO_PCI_CAP_COMMON_CFG);
> > > > +	vp_dev->device_map = virtio_pci_map_cfg(vp_dev,
> > > > +						     VIRTIO_PCI_CAP_DEVICE_CFG);
> > > > +
> > > > +	if (!vp_dev->notify_map || !vp_dev->common_map ||
> > > > +	    !vp_dev->device_map) {
> > > > +		/*
> > > > +		 * If not all capabilities present, map legacy PIO.
> > > > +		 * Legacy access is at BAR 0. We never need to map
> > > > +		 * more than 256 bytes there, since legacy config space
> > > > +		 * used PIO which has this size limit.
> > > > +		 * */
> > > 
> > > Whats the point of putting each of those new configs into a separate cap
> > > structure if you must have all 3 of them for the new config layout to
> > > work anyway?
> > 
> > Pls look closer, we don't require all 3 of them. For example, we can have BAR0
> > DEVICE and COMMON, and notifications will use BAR0.
> > 
> > By using an identical structure we make is much easier to reuse
> > and extend it. For example we might add a separate structure for ISR.
> 
> Right.
> 
> > > Also, why not use this change to simplify the config space? The attempts
> > > to squeeze it into the limited PIO space made it quite messy, and this
> > > can be a good chance to fix some things like the MSI-X and offsets
> > > issue.
> > 
> > For MSI-X this is exactly what this change does, by splitting
> > device-specific parts out. So if DEVICE capability is present,
> > this tells us where the device specific data starts and
> > enabling MSI-X does not relocate it.
> 
> You still have relocations within the shared config space, once 64bit
> features or other future extensions are implemented.
> 
> > Are there other fields we want to move?
> 
> We can do several things:
> 
> 1. Since we're no longer limited in space - remove queue selector and
> just have a full queue table.

I'm not sure it's a good idea.
MMIO space is cheaper than PIO but it's not like it is completely free.
What does this get us?

> 2. Move device specific features into the device specific region.
> Currently the features field is a mix between virtio-pci and device
> specific features.
> 
> I did more similar things in the spec suggestion I've sent, could you
> look at the second part for more ideas please?

Will do.

> -- 
> 
> Sasha.

Re: [PATCH RFC] virtio-pci: flexible configuration layout

[Michael S. Tsirkin]

On Thu, Nov 03, 2011 at 12:37:04PM +0200, Avi Kivity wrote:
> On 11/03/2011 01:31 AM, Michael S. Tsirkin wrote:
> > Add a flexible mechanism to specify virtio configuration layout, using
> > pci vendor-specific capability.  A separate capability is used for each
> > of common, device specific and data-path accesses.
> >
> >
> 
> How about posting the spec change instead of patches?

We need both of course but I typically start with patches.
Easier to see whether it's even possible to implement the
spec without overhauling all of the code.

> -- 
> error compiling committee.c: too many arguments to function

Re: [PATCH RFC] virtio-pci: flexible configuration layout

[Michael S. Tsirkin]

> 2. Move device specific features into the device specific region.
> Currently the features field is a mix between virtio-pci and device
> specific features.

A single feature field with bits partitioned to transport
specific and device specific fields is a generic virtio
thing. So there's relatively little to be gained from
moving the device bits out.

> I did more similar things in the spec suggestion I've sent,

What lacked there is the motivation.

I am aware of several issues that need to be addressed,
as they block extending virtio going forward:
- There's no place to put more transport registers.
  For MSIX I hacked around that with moving device config when a feature
  bit is set, but in hindsight this setup is fragile
  and is panful to extend any further.
  Splitting out device and transport helps.
- PIO space is limited. We are wasting it on non data path
  configuration. There might also be a wish to have largish data
  in configuration space, like a 1024 byte ID field.
- PIO is faster than MMIO on x86 KVM so we need to keep
  using it for data path on that architecture
- Support architectures which have host/guest at different
  endian-ness.

I addressed the first 3 and added capability infrastructure
which we can use in the future to support the last one
(by nature of using little endian format for capabilities).

> could you
> look at the second part for more ideas please?

I saw the per-queue features - but I don't see how it will work really
or what is the issue they solve.
Any more ideas I missed?

> -- 
> 
> Sasha.

Re: [PATCH RFC] virtio-pci: flexible configuration layout

[Avi Kivity]

On 11/03/2011 02:11 PM, Michael S. Tsirkin wrote:
> On Thu, Nov 03, 2011 at 12:37:04PM +0200, Avi Kivity wrote:
> > On 11/03/2011 01:31 AM, Michael S. Tsirkin wrote:
> > > Add a flexible mechanism to specify virtio configuration layout, using
> > > pci vendor-specific capability.  A separate capability is used for each
> > > of common, device specific and data-path accesses.
> > >
> > >
> > 
> > How about posting the spec change instead of patches?
>
> We need both of course but I typically start with patches.
> Easier to see whether it's even possible to implement the
> spec without overhauling all of the code.

Patches are okay for someone familiar with the code.  For others, they
require more effort.

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH RFC] virtio-pci: flexible configuration layout

[Michael S. Tsirkin]

On Thu, Nov 03, 2011 at 03:37:42PM +0200, Avi Kivity wrote:
> On 11/03/2011 02:11 PM, Michael S. Tsirkin wrote:
> > On Thu, Nov 03, 2011 at 12:37:04PM +0200, Avi Kivity wrote:
> > > On 11/03/2011 01:31 AM, Michael S. Tsirkin wrote:
> > > > Add a flexible mechanism to specify virtio configuration layout, using
> > > > pci vendor-specific capability.  A separate capability is used for each
> > > > of common, device specific and data-path accesses.
> > > >
> > > >
> > > 
> > > How about posting the spec change instead of patches?
> >
> > We need both of course but I typically start with patches.
> > Easier to see whether it's even possible to implement the
> > spec without overhauling all of the code.
> 
> Patches are okay for someone familiar with the code.  For others, they
> require more effort.

Yes. It's just that I needed to convince myself that the approach is good,
and that meant writing the supporting code. Since I had the code
I didn't see a reason not to send it :)

> -- 
> error compiling committee.c: too many arguments to function

Re: [PATCH RFC] virtio-pci: flexible configuration layout

[Jesse Barnes]

[-- Attachment #1: Type: text/plain, Size: 1429 bytes --]

On Thu, 3 Nov 2011 01:31:11 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> Add a flexible mechanism to specify virtio configuration layout, using
> pci vendor-specific capability.  A separate capability is used for each
> of common, device specific and data-path accesses.
> 
> Warning: compiled only.
> This patch also needs to be split up, pci_iomap changes
> also need arch updates for non-x86.
> 
> We also will need to update the spec.
> 
> See the first chunk for layout documentation.
> 
> Posting here for early feedback.
> 
> In particular:
> 
> Do we need to require offset to be aligned?
> Does iowrite16 work with unaligned accesses on all architectures?
> Does using ioread/write as we do add overhead as compared to
> plain PIO accesses?
> 
> Jesse - are you OK with the pci_iomap_range API proposed here
> (see last chunks)?
> I noticed lots of architectures duplicate the implementation
> of pci_iomap - makes sense to clean that up?

Probably makes sense to have a weak version at least for arches that
just do the same thing.  But some arches really do need separate
versions because they wrap read/write as well...

Given the arch constraints, it's probably not possible to get rid of
the min/max args in favor of a simple offset/len pair like we have for
ioremap.  But if we could that would be even better.

-- 
Jesse Barnes, Intel Open Source Technology Center

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[PATCH RFC] virtio-spec: flexible configuration layout

[Michael S. Tsirkin]

Here's a spec change documenting what my C patch does
(almost - I tweaked the layout a bit, but the idea is the same).
Some more cleanups are needed but I thought I'd send it
for early flames/comments.

The idea is simple: we split functionally unrelated
register groups to independent structures, and let
the device place is anywhere using a capability
in PCI configuration space.

It can then go into MMIO space which is cheaper than PIO.

A legacy portion of the configuration is mirrored
in the first BAR, to keep legacy drivers working.
Any new fields can be added in existing structures
at the end, so they won't affect legacy.
Alternatively we can add new structures with new
structure IDs, pointed to from PCI configuration space.

As we don't yet have devices or drivers with 64 bit features,
I decided we don't need high feature bits in legacy space.
This also frees up feature bit 31 as we don't need it
to enable high feature bits anymore.

As this solves the dynamic placement of MSIX vectors
and high feature bits,
I thought it's easier to just reserve space for that
programming than give it a separate structure. This
can be changed by a patch on top.

Note that data path is split from configuration.

PDF will follow.
----

diff --git a/virtio-spec.lyx b/virtio-spec.lyx
index 6426f8f..5aec38c 100644
--- a/virtio-spec.lyx
+++ b/virtio-spec.lyx
@@ -56,6 +56,7 @@
 \html_math_output 0
 \html_css_as_file 0
 \html_be_strict false
+\author 1986246365 "Michael S. Tsirkin" 
 \end_header
 
 \begin_body
@@ -570,7 +571,34 @@ Device Configuration
 \end_layout
 
 \begin_layout Standard
-To configure the device, we use the first I/O region of the PCI device.
+To configure the device, we 
+\change_inserted 1986246365 1320783354
+use I/O and/or memory regions of the PCI device.
+ These contain the virtio header registers, the notification register, the
+ ISR status register and device specific registers, as specified by Virtio
+ Structure PCI Capabilities
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320787659
+For backwards compatibility, devices should present part of these configuration
+ registers in a legacy configuration structure in the first I/O region of
+ the PCI device
+\change_unchanged
+
+\end_layout
+
+\end_inset
+
+.
+\end_layout
+
+\begin_layout Standard
+
+\change_deleted 1986246365 1320766512
+use the first I/O region of the PCI device.
  This contains a 
 \emph on
 virtio header
@@ -578,6 +606,8 @@ virtio header
  followed by a 
 \emph on
 device-specific region.
+\change_unchanged
+
 \end_layout
 
 \begin_layout Standard
@@ -595,9 +625,23 @@ natural
 \end_layout
 
 \begin_layout Standard
-Note that this is possible because while the virtio header is PCI (i.e.
- little) endian, the device-specific region is encoded in the native endian
- of the guest (where such distinction is applicable).
+Note that this is possible because while the virtio header
+\change_inserted 1986246365 1320767249
+, notification and ISR status are
+\change_deleted 1986246365 1320767251
+ is
+\change_unchanged
+ PCI (i.e.
+ little) endian, the device-specific
+\change_deleted 1986246365 1320767335
+ region is
+\change_inserted 1986246365 1320767337
+registers are
+\change_unchanged
+ encoded in the native endian of the guest (where such distinction is applicable
+).
+\change_inserted 1986246365 1320767660
+
 \end_layout
 
 \begin_layout Subsection
@@ -635,51 +679,2954 @@ Device-specific setup, including reading the Device Feature Bits, discovery
  writing the virtio configuration space.
 \end_layout
 
-\begin_layout Enumerate
-The subset of Device Feature Bits understood by the driver is written to
- the device.
+\begin_layout Enumerate
+The subset of Device Feature Bits understood by the driver is written to
+ the device.
+\end_layout
+
+\begin_layout Enumerate
+The DRIVER_OK status bit is set.
+\end_layout
+
+\begin_layout Enumerate
+The device can now be used (ie.
+ buffers added to the virtqueues)
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+Historically, drivers have used the device before steps 5 and 6.
+ This is only allowed if the driver does not use any features which would
+ alter this early use of the device.
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+If any of these steps go irrecoverably wrong, the guest should set the FAILED
+ status bit to indicate that it has given up on the device (it can reset
+ the device later to restart if desired).
+\end_layout
+
+\begin_layout Standard
+We now cover the fields required for general setup in detail.
+\change_inserted 1986246365 1320766663
+
+\end_layout
+
+\begin_layout Subsection
+
+\change_inserted 1986246365 1320781584
+Virtio Structure PCI Capability
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320781061
+Virtio configuration layout includes virtio configuration header, notification
+ and ISR status and device configuration structures.
+ Each structure is mapped by a Base Address register (BAR) belonging to
+ the function, located beginning at 10h in Configuration Space.
+ 
+\emph on
+Structure ID 
+\emph default
+indentifies the structure
+\emph on
+, Size
+\emph default
+ indicates the structure size, in bytes, a 
+\emph on
+BAR Indicator register (BIR)
+\emph default
+ indicates which BAR, and 
+\emph on
+Offset
+\emph default
+ indicates where the structure begins relative to the base address associated
+ with the BAR.
+ The BAR is permitted to be either 32-bit or 64-bit, it can map Memory Space
+ or I/O Space.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320781133
+These registers are specified using vendor-specific PCI capability located
+ on capability list in PCI configuration space of the device.
+ This virtio structure capability uses little-endian format; all bits are
+ read-only:
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320772579
+\begin_inset Tabular
+<lyxtabular version="3" rows="4" columns="34">
+<features tabularvalignment="middle">
+<column alignment="center" valignment="top" width="0pt">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0pt">
+<row>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771774
+Bit
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771777
+31
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771779
+30
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771781
+29
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771783
+28
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771786
+27
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771810
+26
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771813
+25
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771815
+24
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771817
+23
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771819
+22
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771821
+21
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771823
+20
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771825
+19
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771829
+18
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771831
+17
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771833
+16
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771835
+15
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771840
+14
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771841
+13
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771843
+12
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771845
+11
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771847
+10
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771850
+9
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771852
+8
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771854
+7
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771856
+6
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771858
+5
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771859
+4
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771861
+3
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771862
+2
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771864
+1
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771888
+0
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771997
+Offset
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="1" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320781029
+Structure ID
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="1" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320772123
+Capability Length = 12
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="1" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320772095
+Next Capability Pointer
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="1" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320772055
+Capability ID = 9
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320772004
+0
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320772549
+Size
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="1" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320779698
+BAR Indicator Register
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320772007
+4
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="1" alignment="center" valignment="top" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320772540
+Offset
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320772011
+8
+\end_layout
+
+\end_inset
+</cell>
+</row>
+</lyxtabular>
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320779667
+Purpose:
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320780912
+
+\emph on
+Capability ID
+\emph default
+, 
+\emph on
+Next Capability Pointer
+\emph default
+, 
+\emph on
+Capability Length
+\emph default
+ - these fields are specified by PCI local bus specification, Rev 3.0
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320785073
+
+\emph on
+Structure ID
+\emph default
+ - identifies the structure mapped.
+ If an ID representing a specific structure is absent, the structure is
+ located in the legacy virtio configuration in the first PCI BAR.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320782689
+
+\emph on
+Size
+\emph default
+ - size of the structure mapped.
+ This size might include padding, or fields unused by the driver.
+ Drivers are also recommended to only map part of configuration structure
+ large enough for device operation.
+ For example, a future device might present a large structure size of several
+ MBytes.
+ As current devices never utilize structures larger than 4KBytes in size,
+ driver can limit the mapped structure size to e.g.
+ 4KBytes to allow forward compatibility with such devices without loss of
+ functionality and without wasting resources.
+\change_unchanged
+
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320781965
+
+\emph on
+BAR Indicator Register
+\emph default
+ - Indicates which one of the BAR registers, located beginning at 10h in
+ Configuration Space, is used to map the structure into Memory or I/O Space.
+ Legal values are 0 - 5.
+ BAR offset is calculated as 
+\emph on
+BAR
+\emph default
+ = 10h + 4 * 
+\emph on
+BIR.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320781002
+
+\emph on
+Offset
+\emph default
+ - Used as an offset from the address contained by one of the function’s
+ Base Address registers to point to the base of the structure.
+ Depending on the structure, the lower bits are masked off (set to zero)
+ by software to form an aligned offset.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320785098
+A single device has multiple virtio capabilities on the PCI capability linked-li
+st.
+ To locate a specific register, driver scans the PCI capability list looking
+ for capabilities using the vendor specific 
+\emph on
+Capability ID
+\emph default
+, that is 
+\emph on
+Capability ID
+\emph default
+ value 
+\emph on
+9
+\emph default
+, according to the PCI spec.
+ For each capability, it verifies that the 
+\emph on
+Capability Length
+\emph default
+ field matches the expected value (12) - if it doesn't, this indicates a
+ layout change, which causes driver to ignore the specific PCI capability
+ and proceed to the next PCI capability in the list.
+ Next,
+\emph on
+ Structure ID
+\emph default
+ single-byte field is read at offset 3.
+ Unrecognized 
+\emph on
+Structure ID
+\emph default
+ values are ignored.
+ Low bits in the 
+\emph on
+Offset
+\emph default
+ register are masked to match alignment requirements, if any.
+ Device drivers are recommended to sanity check 
+\emph on
+BIR
+\emph default
+, 
+\emph on
+Offset
+\emph default
+, and 
+\emph on
+Size
+\emph default
+ fields, ignoring the capability or failing gracefully on error.
+ Drivers are also recommended to only map part of configuration structure
+ large enough for device operation.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320786035
+For backwards compatibility, devices should also present legacy configuration
+ space in the first I/O region of the PCI device, mirroring virtio configuration
+ structures in this space.
+ Drivers should fall back on this legacy structure if a specific Virtio
+ Structure capability is missing in the PCI capability list.
+ However, devices are not required to map the whole configuration structure
+ in legacy space - only the initial segment expected by legacy drivers needs
+ to be mapped.
+ The minimal size of such legacy segment size, for each structure, is listed
+ below.
+ Drivers are discouraged from using legacy space if a specific Structure
+ ID is present; this is to allow deprecating the legacy space in the future.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320786428
+Below is the list of supported Virtio structures, including Structure ID,
+ offset alignment, and where it can be located in the legacy configuration
+ space.
+ Note that legacy space layout changes as the MSI-X capability is enabled
+ and disabled
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786428
+ie.
+ once you enable MSI-X on the device, the other fields move.
+ If you turn it off again, they move back!
+\end_layout
+
+\end_inset
+
+.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320786868
+\begin_inset Tabular
+<lyxtabular version="3" rows="5" columns="6">
+<features tabularvalignment="middle">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<row>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Structure
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+ID
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Alignment
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786921
+Minimal Size
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320785821
+Offset in legacy space
+\change_deleted 1986246365 1320785817
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786926
+Minimal Size in legacy space
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320785855
+Virtio Header
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320785860
+0x1
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786157
+4 bytes
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786225
+28 bytes
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786145
+0
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786236
+24 bytes if MSI-X is enabled, 20 if disabled
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786188
+Queue Notify
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786191
+0x2
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786267
+2 bytes
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786228
+2 bytes
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786505
+16 bytes
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786303
+2 bytes
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786878
+ISR Status
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786260
+0x3
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786270
+1 byte
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786273
+1 byte
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786534
+19 bytes
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786306
+1 byte
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786847
+Device Header
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786474
+0x4
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786477
+1 byte
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786487
+Device specific
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786513
+24 bytes if MSI-X is enabled, 20 if disabled
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786492
+Device specific
+\end_layout
+
+\end_inset
+</cell>
+</row>
+</lyxtabular>
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320786973
+Drivers should assume that each structure size can be extended in the future.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320787003
+Queue Notify and ISR status are single-field registers.
+ Device Header has device specific structure.
+ Virtio Header structure is documented below.
+\change_unchanged
+
+\end_layout
+
+\begin_layout Subsection
+Virtio Header
+\end_layout
+
+\begin_layout Standard
+The virtio header looks as follows:
+\end_layout
+
+\begin_layout Standard
+\begin_inset Tabular
+<lyxtabular version="3" rows="4" columns="9">
+<features tabularvalignment="middle">
+<column alignment="left" valignment="top" width="0">
+<column alignment="left" valignment="top" width="0">
+<column alignment="left" valignment="top" width="0">
+<column alignment="left" valignment="top" width="0">
+<column alignment="left" valignment="top" width="0">
+<column alignment="left" valignment="top" width="0">
+<column alignment="left" valignment="top" width="0">
+<column alignment="left" valignment="top" width="0">
+<column alignment="left" valignment="top" width="0">
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Bits
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+32
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+32
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+32
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+16
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+16
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+16
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+8
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+8
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Read/Write
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+R
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+R+W
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+R+W
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+R
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+R+W
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+R+W
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+R+W
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+R
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Purpose
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\size footnotesize
+Device
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\size footnotesize
+Guest
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\size footnotesize
+Queue
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\size footnotesize
+Queue
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\size footnotesize
+Queue
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786386
+
+\size footnotesize
+Reserved
+\change_deleted 1986246365 1320786378
+Queue
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\size footnotesize
+Device
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786393
+
+\size footnotesize
+Reserved
+\change_deleted 1986246365 1320786390
+ISR
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\size footnotesize
+Features bits 0:31
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\size footnotesize
+Features bits 0:31
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\size footnotesize
+Address
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\size footnotesize
+Size
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\size footnotesize
+Select
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_deleted 1986246365 1320786378
+
+\size footnotesize
+Notify
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\size footnotesize
+Status
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_deleted 1986246365 1320786388
+
+\size footnotesize
+Status
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+</lyxtabular>
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+
+\change_deleted 1986246365 1320786398
+If MSI-X is enabled for the device, two additional fields immediately follow
+ this header:
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+ie.
+ once you enable MSI-X on the device, the other fields move.
+ If you turn it off again, they move back!
+\end_layout
+
+\end_inset
+
+
+\change_unchanged
+
+\end_layout
+
+\begin_layout Standard
+\begin_inset Tabular
+<lyxtabular version="3" rows="4" columns="3">
+<features tabularvalignment="middle">
+<column alignment="left" valignment="top" width="0">
+<column alignment="left" valignment="top" width="0">
+<column alignment="left" valignment="top" width="0">
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Bits
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+16
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+16
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Read/Write
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+R+W
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+R+W
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Purpose
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\size footnotesize
+Configuration
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\size footnotesize
+Queue
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+(MSI-X)
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\size footnotesize
+Vector
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\size footnotesize
+Vector
+\end_layout
+
+\end_inset
+</cell>
+</row>
+</lyxtabular>
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+
+\change_deleted 1986246365 1320786816
+Finally, if feature bits (VIRTIO_F_FEATURES_HI)
+\change_unchanged
+ this is immediately followed by two additional fields:
+\end_layout
+
+\begin_layout Standard
+\begin_inset Tabular
+<lyxtabular version="3" rows="4" columns="3">
+<features tabularvalignment="middle">
+<column alignment="left" valignment="top" width="0">
+<column alignment="left" valignment="top" width="0">
+<column alignment="left" valignment="top" width="0">
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Bits
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+32
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+32
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Read/Write
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+R
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+R+W
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Purpose
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\size footnotesize
+Device
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\size footnotesize
+Guest
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\size footnotesize
+Features bits 32:63
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\size footnotesize
+Features bits 32:63
+\end_layout
+
+\end_inset
+</cell>
+</row>
+</lyxtabular>
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+
+\change_deleted 1986246365 1320786783
+Immediately following these general headers, there may be device-specific
+ headers:
+\end_layout
+
+\begin_layout Standard
+
+\change_deleted 1986246365 1320786783
+\begin_inset Tabular
+<lyxtabular version="3" rows="4" columns="2">
+<features tabularvalignment="middle">
+<column alignment="left" valignment="top" width="0">
+<column alignment="left" valignment="top" width="0">
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Bits
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Device Specific
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Read/Write
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Device Specific
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Purpose
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\size footnotesize
+Device Specific...
 \end_layout
 
-\begin_layout Enumerate
-The DRIVER_OK status bit is set.
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
 \end_layout
 
-\begin_layout Enumerate
-The device can now be used (ie.
- buffers added to the virtqueues)
-\begin_inset Foot
-status open
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
 
 \begin_layout Plain Layout
-Historically, drivers have used the device before steps 5 and 6.
- This is only allowed if the driver does not use any features which would
- alter this early use of the device.
+
 \end_layout
 
 \end_inset
+</cell>
+</row>
+</lyxtabular>
 
+\end_inset
 
-\end_layout
 
-\begin_layout Standard
-If any of these steps go irrecoverably wrong, the guest should set the FAILED
- status bit to indicate that it has given up on the device (it can reset
- the device later to restart if desired).
-\end_layout
+\change_inserted 1986246365 1320784618
 
-\begin_layout Standard
-We now cover the fields required for general setup in detail.
 \end_layout
 
 \begin_layout Subsection
-Virtio Header
+
+\change_inserted 1986246365 1320786351
+Legacy Virtio Header
 \end_layout
 
 \begin_layout Standard
-The virtio header looks as follows:
+
+\change_inserted 1986246365 1320786351
+For reference, the legacy virtio header looks as follows; one observes that
+ most register offsets match the new virtio header, to make compatibility
+ easier:
 \end_layout
 
 \begin_layout Standard
+
+\change_inserted 1986246365 1320786351
 \begin_inset Tabular
 <lyxtabular version="3" rows="4" columns="9">
 <features tabularvalignment="middle">
@@ -697,6 +3644,8 @@ The virtio header looks as follows:
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 Bits
 \end_layout
 
@@ -706,6 +3655,8 @@ Bits
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 32
 \end_layout
 
@@ -715,6 +3666,8 @@ Bits
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 32
 \end_layout
 
@@ -724,6 +3677,8 @@ Bits
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 32
 \end_layout
 
@@ -733,6 +3688,8 @@ Bits
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 16
 \end_layout
 
@@ -742,6 +3699,8 @@ Bits
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 16
 \end_layout
 
@@ -751,6 +3710,8 @@ Bits
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 16
 \end_layout
 
@@ -760,6 +3721,8 @@ Bits
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 8
 \end_layout
 
@@ -769,6 +3732,8 @@ Bits
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 8
 \end_layout
 
@@ -780,6 +3745,8 @@ Bits
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 Read/Write
 \end_layout
 
@@ -789,6 +3756,8 @@ Read/Write
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 R
 \end_layout
 
@@ -798,6 +3767,8 @@ R
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 R+W
 \end_layout
 
@@ -807,6 +3778,8 @@ R+W
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 R+W
 \end_layout
 
@@ -816,6 +3789,8 @@ R+W
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 R
 \end_layout
 
@@ -825,6 +3800,8 @@ R
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 R+W
 \end_layout
 
@@ -834,6 +3811,8 @@ R+W
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 R+W
 \end_layout
 
@@ -843,6 +3822,8 @@ R+W
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 R+W
 \end_layout
 
@@ -852,6 +3833,8 @@ R+W
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 R
 \end_layout
 
@@ -863,6 +3846,8 @@ R
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 Purpose
 \end_layout
 
@@ -873,6 +3858,8 @@ Purpose
 
 \begin_layout Plain Layout
 
+\change_inserted 1986246365 1320786351
+
 \size footnotesize
 Device
 \end_layout
@@ -884,6 +3871,8 @@ Device
 
 \begin_layout Plain Layout
 
+\change_inserted 1986246365 1320786351
+
 \size footnotesize
 Guest
 \end_layout
@@ -895,6 +3884,8 @@ Guest
 
 \begin_layout Plain Layout
 
+\change_inserted 1986246365 1320786351
+
 \size footnotesize
 Queue
 \end_layout
@@ -906,6 +3897,8 @@ Queue
 
 \begin_layout Plain Layout
 
+\change_inserted 1986246365 1320786351
+
 \size footnotesize
 Queue
 \end_layout
@@ -917,6 +3910,8 @@ Queue
 
 \begin_layout Plain Layout
 
+\change_inserted 1986246365 1320786351
+
 \size footnotesize
 Queue
 \end_layout
@@ -928,6 +3923,8 @@ Queue
 
 \begin_layout Plain Layout
 
+\change_inserted 1986246365 1320786351
+
 \size footnotesize
 Queue
 \end_layout
@@ -939,6 +3936,8 @@ Queue
 
 \begin_layout Plain Layout
 
+\change_inserted 1986246365 1320786351
+
 \size footnotesize
 Device
 \end_layout
@@ -950,6 +3949,8 @@ Device
 
 \begin_layout Plain Layout
 
+\change_inserted 1986246365 1320786351
+
 \size footnotesize
 ISR
 \end_layout
@@ -963,6 +3964,8 @@ ISR
 
 \begin_layout Plain Layout
 
+\change_inserted 1986246365 1320786351
+
 \end_layout
 
 \end_inset
@@ -972,6 +3975,8 @@ ISR
 
 \begin_layout Plain Layout
 
+\change_inserted 1986246365 1320786351
+
 \size footnotesize
 Features bits 0:31
 \end_layout
@@ -983,6 +3988,8 @@ Features bits 0:31
 
 \begin_layout Plain Layout
 
+\change_inserted 1986246365 1320786351
+
 \size footnotesize
 Features bits 0:31
 \end_layout
@@ -994,6 +4001,8 @@ Features bits 0:31
 
 \begin_layout Plain Layout
 
+\change_inserted 1986246365 1320786351
+
 \size footnotesize
 Address
 \end_layout
@@ -1005,6 +4014,8 @@ Address
 
 \begin_layout Plain Layout
 
+\change_inserted 1986246365 1320786351
+
 \size footnotesize
 Size
 \end_layout
@@ -1016,6 +4027,8 @@ Size
 
 \begin_layout Plain Layout
 
+\change_inserted 1986246365 1320786351
+
 \size footnotesize
 Select
 \end_layout
@@ -1027,6 +4040,8 @@ Select
 
 \begin_layout Plain Layout
 
+\change_inserted 1986246365 1320786351
+
 \size footnotesize
 Notify
 \end_layout
@@ -1038,6 +4053,8 @@ Notify
 
 \begin_layout Plain Layout
 
+\change_inserted 1986246365 1320786351
+
 \size footnotesize
 Status
 \end_layout
@@ -1049,6 +4066,8 @@ Status
 
 \begin_layout Plain Layout
 
+\change_inserted 1986246365 1320786351
+
 \size footnotesize
 Status
 \end_layout
@@ -1064,12 +4083,16 @@ Status
 \end_layout
 
 \begin_layout Standard
+
+\change_inserted 1986246365 1320786351
 If MSI-X is enabled for the device, two additional fields immediately follow
  this header:
 \begin_inset Foot
 status collapsed
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 ie.
  once you enable MSI-X on the device, the other fields move.
  If you turn it off again, they move back!
@@ -1081,6 +4104,8 @@ ie.
 \end_layout
 
 \begin_layout Standard
+
+\change_inserted 1986246365 1320786351
 \begin_inset Tabular
 <lyxtabular version="3" rows="4" columns="3">
 <features tabularvalignment="middle">
@@ -1092,6 +4117,8 @@ ie.
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 Bits
 \end_layout
 
@@ -1101,6 +4128,8 @@ Bits
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 16
 \end_layout
 
@@ -1110,6 +4139,8 @@ Bits
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 16
 \end_layout
 
@@ -1121,6 +4152,8 @@ Bits
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 Read/Write
 \end_layout
 
@@ -1130,6 +4163,8 @@ Read/Write
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 R+W
 \end_layout
 
@@ -1139,6 +4174,8 @@ R+W
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 R+W
 \end_layout
 
@@ -1150,6 +4187,8 @@ R+W
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 Purpose
 \end_layout
 
@@ -1160,6 +4199,8 @@ Purpose
 
 \begin_layout Plain Layout
 
+\change_inserted 1986246365 1320786351
+
 \size footnotesize
 Configuration
 \end_layout
@@ -1171,6 +4212,8 @@ Configuration
 
 \begin_layout Plain Layout
 
+\change_inserted 1986246365 1320786351
+
 \size footnotesize
 Queue
 \end_layout
@@ -1183,6 +4226,8 @@ Queue
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 (MSI-X)
 \end_layout
 
@@ -1193,16 +4238,7 @@ Queue
 
 \begin_layout Plain Layout
 
-\size footnotesize
-Vector
-\end_layout
-
-\end_inset
-</cell>
-<cell alignment="center" valignment="top" bottomline="true" leftline="true" rightline="true" usebox="none">
-\begin_inset Text
-
-\begin_layout Plain Layout
+\change_inserted 1986246365 1320786351
 
 \size footnotesize
 Vector
@@ -1210,145 +4246,15 @@ Vector
 
 \end_inset
 </cell>
-</row>
-</lyxtabular>
-
-\end_inset
-
-
-\end_layout
-
-\begin_layout Standard
-Finally, if feature bits (VIRTIO_F_FEATURES_HI) this is immediately followed
- by two additional fields:
-\end_layout
-
-\begin_layout Standard
-\begin_inset Tabular
-<lyxtabular version="3" rows="4" columns="3">
-<features tabularvalignment="middle">
-<column alignment="left" valignment="top" width="0">
-<column alignment="left" valignment="top" width="0">
-<column alignment="left" valignment="top" width="0">
-<row>
-<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
-\begin_inset Text
-
-\begin_layout Plain Layout
-Bits
-\end_layout
-
-\end_inset
-</cell>
-<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
-\begin_inset Text
-
-\begin_layout Plain Layout
-32
-\end_layout
-
-\end_inset
-</cell>
-<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
-\begin_inset Text
-
-\begin_layout Plain Layout
-32
-\end_layout
-
-\end_inset
-</cell>
-</row>
-<row>
-<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
-\begin_inset Text
-
-\begin_layout Plain Layout
-Read/Write
-\end_layout
-
-\end_inset
-</cell>
-<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
-\begin_inset Text
-
-\begin_layout Plain Layout
-R
-\end_layout
-
-\end_inset
-</cell>
-<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
-\begin_inset Text
-
-\begin_layout Plain Layout
-R+W
-\end_layout
-
-\end_inset
-</cell>
-</row>
-<row>
-<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
-\begin_inset Text
-
-\begin_layout Plain Layout
-Purpose
-\end_layout
-
-\end_inset
-</cell>
-<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
-\begin_inset Text
-
-\begin_layout Plain Layout
-
-\size footnotesize
-Device
-\end_layout
-
-\end_inset
-</cell>
-<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
-\begin_inset Text
-
-\begin_layout Plain Layout
-
-\size footnotesize
-Guest
-\end_layout
-
-\end_inset
-</cell>
-</row>
-<row>
 <cell alignment="center" valignment="top" bottomline="true" leftline="true" rightline="true" usebox="none">
 \begin_inset Text
 
 \begin_layout Plain Layout
 
-\end_layout
-
-\end_inset
-</cell>
-<cell alignment="center" valignment="top" bottomline="true" leftline="true" usebox="none">
-\begin_inset Text
-
-\begin_layout Plain Layout
-
-\size footnotesize
-Features bits 32:63
-\end_layout
-
-\end_inset
-</cell>
-<cell alignment="center" valignment="top" bottomline="true" leftline="true" usebox="none">
-\begin_inset Text
-
-\begin_layout Plain Layout
+\change_inserted 1986246365 1320786351
 
 \size footnotesize
-Features bits 32:63
+Vector
 \end_layout
 
 \end_inset
@@ -1362,11 +4268,15 @@ Features bits 32:63
 \end_layout
 
 \begin_layout Standard
-Immediately following these general headers, there may be device-specific
+
+\change_inserted 1986246365 1320786351
+Immediately following these general headers, there may be legacy device-specific
  headers:
 \end_layout
 
 \begin_layout Standard
+
+\change_inserted 1986246365 1320786351
 \begin_inset Tabular
 <lyxtabular version="3" rows="4" columns="2">
 <features tabularvalignment="middle">
@@ -1377,6 +4287,8 @@ Immediately following these general headers, there may be device-specific
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 Bits
 \end_layout
 
@@ -1386,6 +4298,8 @@ Bits
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 Device Specific
 \end_layout
 
@@ -1397,6 +4311,8 @@ Device Specific
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 Read/Write
 \end_layout
 
@@ -1406,6 +4322,8 @@ Read/Write
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 Device Specific
 \end_layout
 
@@ -1417,6 +4335,8 @@ Device Specific
 \begin_inset Text
 
 \begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786351
 Purpose
 \end_layout
 
@@ -1427,6 +4347,8 @@ Purpose
 
 \begin_layout Plain Layout
 
+\change_inserted 1986246365 1320786351
+
 \size footnotesize
 Device Specific...
 \end_layout
@@ -1440,6 +4362,8 @@ Device Specific...
 
 \begin_layout Plain Layout
 
+\change_inserted 1986246365 1320786351
+
 \end_layout
 
 \end_inset
@@ -1449,6 +4373,8 @@ Device Specific...
 
 \begin_layout Plain Layout
 
+\change_inserted 1986246365 1320786351
+
 \end_layout
 
 \end_inset
@@ -1461,6 +4387,15 @@ Device Specific...
 
 \end_layout
 
+\begin_layout Standard
+
+\change_inserted 1986246365 1320786619
+Legacy device-specific headers are supported for backwards compatibility.
+ As such, they are not required to map the whole device configuration structure
+ - only the initial segment expected by legacy drivers needs to be mapped.
+ The size of such legacy segment size is device specific.
+\end_layout
+
 \begin_layout Subsubsection
 Device Status
 \end_layout
@@ -1531,8 +4466,10 @@ name "sub:Feature-Bits"
 The least significant 31 bits of the first configuration field indicates
  the features that the device supports (the high bit is reserved, and will
  be used to indicate the presence of future feature bits elsewhere).
- If more than 31 feature bits are supported, the device indicates so by
- setting feature bit 31 (see 
+ 
+\change_deleted 1986246365 1320784929
+If more than 31 feature bits are supported, the device indicates so by setting
+ feature bit 31 (see 
 \begin_inset CommandInset ref
 LatexCommand ref
 reference "cha:Reserved-Feature-Bits"
@@ -1540,7 +4477,9 @@ reference "cha:Reserved-Feature-Bits"
 \end_inset
 
 ).
- The bits are allocated as follows:
+ 
+\change_unchanged
+The bits are allocated as follows:
 \end_layout
 
 \begin_layout Description
@@ -1615,10 +4554,14 @@ This allows for forwards and backwards compatibility: if the device is enhanced
 \end_layout
 
 \begin_layout Standard
+
+\change_deleted 1986246365 1320784940
 Access to feature bits 32 to 63 is enabled by Guest by setting feature bit
  31.
  If this bit is unset, Device must assume that all feature bits > 31 are
  unset.
+\change_unchanged
+
 \end_layout
 
 \begin_layout Subsubsection
@@ -1629,10 +4572,14 @@ Configuration/Queue Vectors
 When MSI-X capability is present and enabled in the device (through standard
  PCI configuration space) 4 bytes at byte offset 20 are used to map configuratio
 n change and queue interrupts to MSI-X vectors.
- In this case, the ISR Status field is unused, and device specific configuration
- starts at byte offset 24 in virtio header structure.
+ In this case, the ISR Status field is unused
+\change_deleted 1986246365 1320784981
+, and device specific configuration starts at byte offset 24 in virtio header
+ structure.
  When MSI-X capability is not enabled, device specific configuration starts
- at byte offset 20 in virtio header.
+ at byte offset 20 in virtio header
+\change_unchanged
+.
 \end_layout
 
 \begin_layout Standard

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Sasha Levin]

On Tue, 2011-11-08 at 23:40 +0200, Michael S. Tsirkin wrote:
> Here's a spec change documenting what my C patch does
> (almost - I tweaked the layout a bit, but the idea is the same).
> Some more cleanups are needed but I thought I'd send it
> for early flames/comments.
> 
> The idea is simple: we split functionally unrelated
> register groups to independent structures, and let
> the device place is anywhere using a capability
> in PCI configuration space.
> 
> It can then go into MMIO space which is cheaper than PIO.
> 
> A legacy portion of the configuration is mirrored
> in the first BAR, to keep legacy drivers working.
> Any new fields can be added in existing structures
> at the end, so they won't affect legacy.

If newer specs add more structures at the end of the config space, and
use the same config space for legacy, that space now becomes device
specific config space and not new-shiny-feature space, so we must
remember to handle those cases.

> Alternatively we can add new structures with new
> structure IDs, pointed to from PCI configuration space.
> 
> As we don't yet have devices or drivers with 64 bit features,
> I decided we don't need high feature bits in legacy space.
> This also frees up feature bit 31 as we don't need it
> to enable high feature bits anymore.

KVM tool actually has support for 64bit features, we can probably remove
that when Pekka isn't looking :)

> 
> As this solves the dynamic placement of MSIX vectors
> and high feature bits,
> I thought it's easier to just reserve space for that
> programming than give it a separate structure. This
> can be changed by a patch on top.
> 
> Note that data path is split from configuration.
> 
> PDF will follow.
> ----
> 

The device initialization sequence might use an update as well. Maybe
also a description of how device handles missing structure IDs.

[snip]

> +
> +\begin_layout Standard
> +
> +\change_inserted 1986246365 1320781133
> +These registers are specified using vendor-specific PCI capability located
> + on capability list in PCI configuration space of the device.
> + This virtio structure capability uses little-endian format; all bits are
> + read-only:
> +\end_layout
> +
> +\begin_layout Standard
> +
> +\change_inserted 1986246365 1320772579
> +\begin_inset Tabular

Just a note, these tables are way too wide to work properly in PDFs :)

> +<lyxtabular version="3" rows="4" columns="34">
> +<features tabularvalignment="middle">
> +<column alignment="center" valignment="top" width="0pt">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0">
> +<column alignment="center" valignment="top" width="0pt">
> +<row>

[snip]

> +
> +\begin_layout Standard
> +
> +\change_inserted 1986246365 1320779667
> +Purpose:
> +\end_layout
> +
> +\begin_layout Standard
> +
> +\change_inserted 1986246365 1320780912
> +
> +\emph on
> +Capability ID
> +\emph default
> +, 
> +\emph on
> +Next Capability Pointer
> +\emph default
> +, 
> +\emph on
> +Capability Length
> +\emph default
> + - these fields are specified by PCI local bus specification, Rev 3.0

I'm not sure what capability length is, can't find it in the spec
either.

[snip]

> +\begin_layout Plain Layout
> +ie.
> + once you enable MSI-X on the device, the other fields move.
> + If you turn it off again, they move back!

Is it still true? We're talking about the new layout here (there are
several of this footnote, this one is located right *before* the section
which talks about legacy config space.

[snip]

> + 
> +\change_deleted 1986246365 1320784929
> +If more than 31 feature bits are supported, the device indicates so by setting
> + feature bit 31 (see 

The bit numbers below this text should be corrected as well.

-- 

Sasha.

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Sasha Levin]

On Tue, 2011-11-08 at 23:40 +0200, Michael S. Tsirkin wrote:
> Here's a spec change documenting what my C patch does
> (almost - I tweaked the layout a bit, but the idea is the same).
> Some more cleanups are needed but I thought I'd send it
> for early flames/comments.
> 
> The idea is simple: we split functionally unrelated
> register groups to independent structures, and let
> the device place is anywhere using a capability
> in PCI configuration space.

I'm also wondering it it's ok to move virtio configuration out of virtio
space and into PCI space for archs that don't have PCI (such as ARM).

Would it mean they get stuck with legacy configuration (and no new
features)? Or is there an alternative for them?

-- 

Sasha.

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Michael S. Tsirkin]

On Wed, Nov 09, 2011 at 10:46:06AM +0200, Sasha Levin wrote:
> On Tue, 2011-11-08 at 23:40 +0200, Michael S. Tsirkin wrote:
> > Here's a spec change documenting what my C patch does
> > (almost - I tweaked the layout a bit, but the idea is the same).
> > Some more cleanups are needed but I thought I'd send it
> > for early flames/comments.
> > 
> > The idea is simple: we split functionally unrelated
> > register groups to independent structures, and let
> > the device place is anywhere using a capability
> > in PCI configuration space.
> > 
> > It can then go into MMIO space which is cheaper than PIO.
> > 
> > A legacy portion of the configuration is mirrored
> > in the first BAR, to keep legacy drivers working.
> > Any new fields can be added in existing structures
> > at the end, so they won't affect legacy.
> 
> If newer specs add more structures at the end of the config space, and
> use the same config space for legacy, that space now becomes device
> specific config space and not new-shiny-feature space, so we must
> remember to handle those cases.

Yes. Devices should mirror virtio header and device structures
in MMIO and point Structures there. It seems unavoidable for
device structures (because legacy layout is dynamic
and we don't want to keep doing that), so I thought it's not worth the
trouble for virtio header either.

Notification and ISR Structures can point into PIO.

I'll add some more text to clarify that.

> > Alternatively we can add new structures with new
> > structure IDs, pointed to from PCI configuration space.
> > 
> > As we don't yet have devices or drivers with 64 bit features,
> > I decided we don't need high feature bits in legacy space.
> > This also frees up feature bit 31 as we don't need it
> > to enable high feature bits anymore.
> 
> KVM tool actually has support for 64bit features, we can probably remove
> that when Pekka isn't looking :)

It's not yet released so maybe it's not an issue yet.
If it's too late I can re-add them to legacy too.

Pekka, 64 features aren't yet used and we are discussing
changing the layout for that field. Mind taking it out
of kvm tool for now?

> > 
> > As this solves the dynamic placement of MSIX vectors
> > and high feature bits,
> > I thought it's easier to just reserve space for that
> > programming than give it a separate structure. This
> > can be changed by a patch on top.
> > 
> > Note that data path is split from configuration.
> > 
> > PDF will follow.
> > ----
> > 
> 
> The device initialization sequence might use an update as well.

What is needed? Add an item where the driver scans the PCI capability
list to detect the layout?

> Maybe
> also a description of how device handles missing structure IDs.
> 
> [snip]

Hmm. I just have
'Drivers should fall back on this legacy structure if a
 Virtio Structure capability is missing in the PCI capability
 list'. 

What else would be helpful? An example? 

> > +
> > +\begin_layout Standard
> > +
> > +\change_inserted 1986246365 1320781133
> > +These registers are specified using vendor-specific PCI capability located
> > + on capability list in PCI configuration space of the device.
> > + This virtio structure capability uses little-endian format; all bits are
> > + read-only:
> > +\end_layout
> > +
> > +\begin_layout Standard
> > +
> > +\change_inserted 1986246365 1320772579
> > +\begin_inset Tabular
> 
> Just a note, these tables are way too wide to work properly in PDFs :)

True, looks like I need to abbreviate

> > +<lyxtabular version="3" rows="4" columns="34">
> > +<features tabularvalignment="middle">
> > +<column alignment="center" valignment="top" width="0pt">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0">
> > +<column alignment="center" valignment="top" width="0pt">
> > +<row>
> 
> [snip]
> 
> > +
> > +\begin_layout Standard
> > +
> > +\change_inserted 1986246365 1320779667
> > +Purpose:
> > +\end_layout
> > +
> > +\begin_layout Standard
> > +
> > +\change_inserted 1986246365 1320780912
> > +
> > +\emph on
> > +Capability ID
> > +\emph default
> > +, 
> > +\emph on
> > +Next Capability Pointer
> > +\emph default
> > +, 
> > +\emph on
> > +Capability Length
> > +\emph default
> > + - these fields are specified by PCI local bus specification, Rev 3.0
> 
> I'm not sure what capability length is, can't find it in the spec
> either.
> 
> [snip]

It's the legth of the vendor specific capability structure in bytes.
'the byte immediately following the “Next”
 pointer in the capability structure is defined to be a length field'
It's on page 330 in my copy.



> > +\begin_layout Plain Layout
> > +ie.
> > + once you enable MSI-X on the device, the other fields move.
> > + If you turn it off again, they move back!
> 
> Is it still true? We're talking about the new layout here (there are
> several of this footnote, this one is located right *before* the section
> which talks about legacy config space.
> 
> [snip]

Of course not.  I'll move the footnote, this only applies to the
legacy naturally.

> > + 
> > +\change_deleted 1986246365 1320784929
> > +If more than 31 feature bits are supported, the device indicates so by setting
> > + feature bit 31 (see 
> 
> The bit numbers below this text should be corrected as well.

Will fix. Thanks!

> -- 
> 
> Sasha.

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Michael S. Tsirkin]

On Wed, Nov 09, 2011 at 11:55:02AM +0200, Sasha Levin wrote:
> On Tue, 2011-11-08 at 23:40 +0200, Michael S. Tsirkin wrote:
> > Here's a spec change documenting what my C patch does
> > (almost - I tweaked the layout a bit, but the idea is the same).
> > Some more cleanups are needed but I thought I'd send it
> > for early flames/comments.
> > 
> > The idea is simple: we split functionally unrelated
> > register groups to independent structures, and let
> > the device place is anywhere using a capability
> > in PCI configuration space.
> 
> I'm also wondering it it's ok to move virtio configuration out of virtio
> space and into PCI space for archs that don't have PCI (such as ARM).

A bit of confusion here: no register is moved from memory into
configuration space. Configuration space only gains a list of
pointers into memory/pio space.

> Would it mean they get stuck with legacy configuration (and no new
> features)? Or is there an alternative for them?

The change only affects the layout of virtio PCI. Arches that don't
have PCI don't use virtio PCI, presumably?

BTW, the spec only covers x86 ATM, this needs to be fixed.

> 
> -- 
> 
> Sasha.

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Sasha Levin]

On Wed, 2011-11-09 at 12:18 +0200, Michael S. Tsirkin wrote:
> On Wed, Nov 09, 2011 at 11:55:02AM +0200, Sasha Levin wrote:
> > On Tue, 2011-11-08 at 23:40 +0200, Michael S. Tsirkin wrote:
> > > Here's a spec change documenting what my C patch does
> > > (almost - I tweaked the layout a bit, but the idea is the same).
> > > Some more cleanups are needed but I thought I'd send it
> > > for early flames/comments.
> > > 
> > > The idea is simple: we split functionally unrelated
> > > register groups to independent structures, and let
> > > the device place is anywhere using a capability
> > > in PCI configuration space.
> > 
> > I'm also wondering it it's ok to move virtio configuration out of virtio
> > space and into PCI space for archs that don't have PCI (such as ARM).
> 
> A bit of confusion here: no register is moved from memory into
> configuration space. Configuration space only gains a list of
> pointers into memory/pio space.

The definition of the way the virtio configuration space looks is moved
into PCI configuration space, it was constant before.

> > Would it mean they get stuck with legacy configuration (and no new
> > features)? Or is there an alternative for them?
> 
> The change only affects the layout of virtio PCI. Arches that don't
> have PCI don't use virtio PCI, presumably?
> 
> BTW, the spec only covers x86 ATM, this needs to be fixed.

>From what I see there is a WIP by Pawel Moll <pawel.moll@arm.com> to add
virtio platform drivers which get virtio working on ARM for example, and
by Peter Maydell <peter.maydell@linaro.org> to modify the spec to
support MMIO access (besides PCI).

Maybe worth seeing if it works for them...

-- 

Sasha.

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Sasha Levin]

On Wed, 2011-11-09 at 12:13 +0200, Michael S. Tsirkin wrote:
> On Wed, Nov 09, 2011 at 10:46:06AM +0200, Sasha Levin wrote:
> > The device initialization sequence might use an update as well.
> 
> What is needed? Add an item where the driver scans the PCI capability
> list to detect the layout?

Yup, something similar like that - just to make it obvious.

> > Maybe
> > also a description of how device handles missing structure IDs.
> > 
> > [snip]
> 
> Hmm. I just have
> 'Drivers should fall back on this legacy structure if a
>  Virtio Structure capability is missing in the PCI capability
>  list'. 
> 
> What else would be helpful? An example? 

I just remembered from your patch that you could define some structure
IDs, but not necessarily all of them, if it's not longer the case then
ignore me. If it is, then the scenario of a missing structure should be
specified (For example, if the ISR structure wasn't defined, fall back
to...).

> > I'm not sure what capability length is, can't find it in the spec
> > either.
> > 
> > [snip]
> 
> It's the legth of the vendor specific capability structure in bytes.
> 'the byte immediately following the “Next”
>  pointer in the capability structure is defined to be a length field'
> It's on page 330 in my copy.

Right, I was looking only at the regular cap definition.

-- 

Sasha.

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Pawel Moll]

On Wed, 2011-11-09 at 10:20 +0000, Sasha Levin wrote:
> > > I'm also wondering it it's ok to move virtio configuration out of virtio
> > > space and into PCI space for archs that don't have PCI (such as ARM).

Just a note - ARM-based chips can by all means have PCI (grep -r PCI
arch/arm/ ;-). The fact is that most of the SOCs available on the market
don't have it, but this is slowly changing.

The main architectural difference is that ARM doesn't provide separate
I/O space so the PCI I/O space is usually remapped somewhere into normal
address space (grep -r "#define __io_address" arch/arm/)
 
> > > Would it mean they get stuck with legacy configuration (and no new
> > > features)? Or is there an alternative for them?
> > 
> > The change only affects the layout of virtio PCI. Arches that don't
> > have PCI don't use virtio PCI, presumably?
> > 
> > BTW, the spec only covers x86 ATM, this needs to be fixed.
> 
> From what I see there is a WIP by Pawel Moll <pawel.moll@arm.com> to add
> virtio platform drivers which get virtio working on ARM for example, and
> by Peter Maydell <peter.maydell@linaro.org> to modify the spec to
> support MMIO access (besides PCI).

Yep, it's actually already in 3.2-rc1 (drivers/virtio/virtio_mmio.c) and
in the spec (see Appendix X). And actually the control registers layout
I used was originally based on the PCI "legacy" headers (slightly
simplified), but evolved a bit since. My understanding is that the
changes Michael is proposing affect the PCI device interface only so
they shouldn't affect "my" interface.

By the way, I vaguely remember Peter mentioning that he got the PCI
device "experimentally" running some time ago on one of the PCI-enabled
ARM platform models (realview or versatile)...

Cheers!

Pawel

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Michael S. Tsirkin]

On Wed, Nov 09, 2011 at 12:26:08PM +0200, Sasha Levin wrote:
> On Wed, 2011-11-09 at 12:13 +0200, Michael S. Tsirkin wrote:
> > On Wed, Nov 09, 2011 at 10:46:06AM +0200, Sasha Levin wrote:
> > > The device initialization sequence might use an update as well.
> > 
> > What is needed? Add an item where the driver scans the PCI capability
> > list to detect the layout?
> 
> Yup, something similar like that - just to make it obvious.
> 
> > > Maybe
> > > also a description of how device handles missing structure IDs.
> > > 
> > > [snip]
> > 
> > Hmm. I just have
> > 'Drivers should fall back on this legacy structure if a
> >  Virtio Structure capability is missing in the PCI capability
> >  list'. 
> > 
> > What else would be helpful? An example? 
> 
> I just remembered from your patch that you could define some structure
> IDs, but not necessarily all of them, if it's not longer the case then
> ignore me. If it is, then the scenario of a missing structure should be
> specified (For example, if the ISR structure wasn't defined, fall back
> to...).

Yes, this was the intent of the text above. I'll make that explicit.

> > > I'm not sure what capability length is, can't find it in the spec
> > > either.
> > > 
> > > [snip]
> > 
> > It's the legth of the vendor specific capability structure in bytes.
> > 'the byte immediately following the “Next”
> >  pointer in the capability structure is defined to be a length field'
> > It's on page 330 in my copy.
> 
> Right, I was looking only at the regular cap definition.
> 
> -- 
> 
> Sasha.

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Sasha Levin]

On Wed, 2011-11-09 at 10:47 +0000, Pawel Moll wrote:
> On Wed, 2011-11-09 at 10:20 +0000, Sasha Levin wrote:
> > > > I'm also wondering it it's ok to move virtio configuration out of virtio
> > > > space and into PCI space for archs that don't have PCI (such as ARM).
> 
> Just a note - ARM-based chips can by all means have PCI (grep -r PCI
> arch/arm/ ;-). The fact is that most of the SOCs available on the market
> don't have it, but this is slowly changing.
> 
> The main architectural difference is that ARM doesn't provide separate
> I/O space so the PCI I/O space is usually remapped somewhere into normal
> address space (grep -r "#define __io_address" arch/arm/)
>  
> > > > Would it mean they get stuck with legacy configuration (and no new
> > > > features)? Or is there an alternative for them?
> > > 
> > > The change only affects the layout of virtio PCI. Arches that don't
> > > have PCI don't use virtio PCI, presumably?
> > > 
> > > BTW, the spec only covers x86 ATM, this needs to be fixed.
> > 
> > From what I see there is a WIP by Pawel Moll <pawel.moll@arm.com> to add
> > virtio platform drivers which get virtio working on ARM for example, and
> > by Peter Maydell <peter.maydell@linaro.org> to modify the spec to
> > support MMIO access (besides PCI).
> 
> Yep, it's actually already in 3.2-rc1 (drivers/virtio/virtio_mmio.c) and
> in the spec (see Appendix X). And actually the control registers layout
> I used was originally based on the PCI "legacy" headers (slightly
> simplified), but evolved a bit since. My understanding is that the
> changes Michael is proposing affect the PCI device interface only so
> they shouldn't affect "my" interface.

I didn't know it's in already, might be interesting adding support to it
to x86 userspace tools.

I thought you used the 'legacy' layout, which is why I was worried that
these changed might cause problems for you - but from what I see you
have a different layout there, so as you said, it shouldn't cause any
issues there.

> 
> By the way, I vaguely remember Peter mentioning that he got the PCI
> device "experimentally" running some time ago on one of the PCI-enabled
> ARM platform models (realview or versatile)...


-- 

Sasha.

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Pawel Moll]

On Wed, 2011-11-09 at 10:55 +0000, Sasha Levin wrote:
> On Wed, 2011-11-09 at 10:47 +0000, Pawel Moll wrote:
> > Yep, it's actually already in 3.2-rc1 (drivers/virtio/virtio_mmio.c) and
> > in the spec (see Appendix X). And actually the control registers layout
> > I used was originally based on the PCI "legacy" headers (slightly
> > simplified), but evolved a bit since. My understanding is that the
> > changes Michael is proposing affect the PCI device interface only so
> > they shouldn't affect "my" interface.
> 
> I didn't know it's in already, might be interesting adding support to it
> to x86 userspace tools.

Do you mean the qemu or the (non-qemu) KVM tools I know nothing
about? ;-)

If qemu, Peter got it already running with virtio_mmio on ARM models
(not upstream yet as far as I know), but I presume the code would we
mostly non-ARM specific.

Cheers!

Pawel

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Peter Maydell]

On 9 November 2011 11:06, Pawel Moll <pawel.moll@arm.com> wrote:
> On Wed, 2011-11-09 at 10:55 +0000, Sasha Levin wrote:
>> On Wed, 2011-11-09 at 10:47 +0000, Pawel Moll wrote:
>> > Yep, it's actually already in 3.2-rc1 (drivers/virtio/virtio_mmio.c) and
>> > in the spec (see Appendix X). And actually the control registers layout
>> > I used was originally based on the PCI "legacy" headers (slightly
>> > simplified), but evolved a bit since. My understanding is that the
>> > changes Michael is proposing affect the PCI device interface only so
>> > they shouldn't affect "my" interface.
>>
>> I didn't know it's in already, might be interesting adding support to it
>> to x86 userspace tools.
>
> Do you mean the qemu or the (non-qemu) KVM tools I know nothing
> about? ;-)
>
> If qemu, Peter got it already running with virtio_mmio on ARM models
> (not upstream yet as far as I know), but I presume the code would we
> mostly non-ARM specific.

The QEMU code is completely non-ARM-specific (apart from the line
in the relevant ARM board models where we instantiate the devices).
I haven't submitted it upstream yet though because really the
virtio transport layer in QEMU needs to be refactored as a qdev
bus first.

-- PMM

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Sasha Levin]

On Wed, 2011-11-09 at 11:06 +0000, Pawel Moll wrote:
> On Wed, 2011-11-09 at 10:55 +0000, Sasha Levin wrote:
> > On Wed, 2011-11-09 at 10:47 +0000, Pawel Moll wrote:
> > > Yep, it's actually already in 3.2-rc1 (drivers/virtio/virtio_mmio.c) and
> > > in the spec (see Appendix X). And actually the control registers layout
> > > I used was originally based on the PCI "legacy" headers (slightly
> > > simplified), but evolved a bit since. My understanding is that the
> > > changes Michael is proposing affect the PCI device interface only so
> > > they shouldn't affect "my" interface.
> > 
> > I didn't know it's in already, might be interesting adding support to it
> > to x86 userspace tools.
> 
> Do you mean the qemu or the (non-qemu) KVM tools I know nothing
> about? ;-)

I'll get you in the loop when we add virtio-mmio :)

> 
> If qemu, Peter got it already running with virtio_mmio on ARM models
> (not upstream yet as far as I know), but I presume the code would we
> mostly non-ARM specific.
> 
> Cheers!
> 
> Pawel
> 
> 

-- 

Sasha.

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Pekka Enberg]

On Wed, 9 Nov 2011, Michael S. Tsirkin wrote:
>> KVM tool actually has support for 64bit features, we can probably remove
>> that when Pekka isn't looking :)
>
> It's not yet released so maybe it's not an issue yet.
> If it's too late I can re-add them to legacy too.
>
> Pekka, 64 features aren't yet used and we are discussing
> changing the layout for that field. Mind taking it out
> of kvm tool for now?

Sasha, why did we add 64-bit features to the KVM tool? Wasn't it part of 
the virtio spec? Does QEMU not use them? How badly will older versions of 
the KVM tool break if you drop 64-bit features?

 			Pekka

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Sasha Levin]

On Wed, 2011-11-09 at 14:25 +0200, Pekka Enberg wrote:
> On Wed, 9 Nov 2011, Michael S. Tsirkin wrote:
> >> KVM tool actually has support for 64bit features, we can probably remove
> >> that when Pekka isn't looking :)
> >
> > It's not yet released so maybe it's not an issue yet.
> > If it's too late I can re-add them to legacy too.
> >
> > Pekka, 64 features aren't yet used and we are discussing
> > changing the layout for that field. Mind taking it out
> > of kvm tool for now?
> 
> Sasha, why did we add 64-bit features to the KVM tool? Wasn't it part of 
> the virtio spec? Does QEMU not use them? How badly will older versions of 
> the KVM tool break if you drop 64-bit features?

We added 64-bit features to the tool because it just got into the spec
when we rewrote our virtio-pci handling - so we just implemented the
updated spec.

QEMU doesn't use them since while it did get into the spec, it was
intended to future-proof the limited feature bits, so no one really
needed them yet.

They don't exist in kernel code either, for same reason as above.

Nothing will break if we remove it since no one really used it, we were
probably the first and only implementation of the spec which considered
them :)

-- 

Sasha.

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Pekka Enberg]

On Wed, 9 Nov 2011, Sasha Levin wrote:
> They don't exist in kernel code either, for same reason as above.
>
> Nothing will break if we remove it since no one really used it, we were
> probably the first and only implementation of the spec which considered
> them :)

As long as we are able to run older versions of the KVM tool with newer 
kernels and vice versa, I see no reason why we can't drop 64-bit features 
from the KVM tool.

 			Pekka

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Avi Kivity]

On 11/09/2011 10:46 AM, Sasha Levin wrote:
> > Alternatively we can add new structures with new
> > structure IDs, pointed to from PCI configuration space.
> > 
> > As we don't yet have devices or drivers with 64 bit features,
> > I decided we don't need high feature bits in legacy space.
> > This also frees up feature bit 31 as we don't need it
> > to enable high feature bits anymore.
>
> KVM tool actually has support for 64bit features, we can probably remove
> that when Pekka isn't looking :)
>

What about the Windows drivers?

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Sasha Levin]

On Wed, 2011-11-09 at 14:38 +0200, Avi Kivity wrote:
> On 11/09/2011 10:46 AM, Sasha Levin wrote:
> > > Alternatively we can add new structures with new
> > > structure IDs, pointed to from PCI configuration space.
> > > 
> > > As we don't yet have devices or drivers with 64 bit features,
> > > I decided we don't need high feature bits in legacy space.
> > > This also frees up feature bit 31 as we don't need it
> > > to enable high feature bits anymore.
> >
> > KVM tool actually has support for 64bit features, we can probably remove
> > that when Pekka isn't looking :)
> >
> 
> What about the Windows drivers?

While 64-bit features were defined, there was no actual devices to use
them.

So even if Windows drivers had the ability to support 64-bit features,
no device ever needed it therefore no device ever activated it.

A potential issue might arise when we remap feature bit 31 to do
something else, and when activated it would fool Windows drivers to
think that 64-bit features are now activated, while they're not - but
thats not something that should happen since legacy layout shouldn't
have that field enabled as per the spec (no new features in legacy
code).

I was unable to check if it was actually implemented in the drivers
because
http://git.kernel.org/?p=virt/kvm/kvm-guest-drivers-windows.git;a=summary is not quite there (*cough*).

-- 

Sasha.

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Michael S. Tsirkin]

On Wed, Nov 09, 2011 at 02:48:58PM +0200, Sasha Levin wrote:
> On Wed, 2011-11-09 at 14:38 +0200, Avi Kivity wrote:
> > On 11/09/2011 10:46 AM, Sasha Levin wrote:
> > > > Alternatively we can add new structures with new
> > > > structure IDs, pointed to from PCI configuration space.
> > > > 
> > > > As we don't yet have devices or drivers with 64 bit features,
> > > > I decided we don't need high feature bits in legacy space.
> > > > This also frees up feature bit 31 as we don't need it
> > > > to enable high feature bits anymore.
> > >
> > > KVM tool actually has support for 64bit features, we can probably remove
> > > that when Pekka isn't looking :)
> > >
> > 
> > What about the Windows drivers?
> 
> While 64-bit features were defined, there was no actual devices to use
> them.
> 
> So even if Windows drivers had the ability to support 64-bit features,
> no device ever needed it therefore no device ever activated it.
> 
> A potential issue might arise when we remap feature bit 31 to do
> something else, and when activated it would fool Windows drivers to
> think that 64-bit features are now activated, while they're not - but
> thats not something that should happen since legacy layout shouldn't
> have that field enabled as per the spec (no new features in legacy
> code).
> 
> I was unable to check if it was actually implemented in the drivers
> because
> http://git.kernel.org/?p=virt/kvm/kvm-guest-drivers-windows.git;a=summary is not quite there (*cough*).

I'm pretty sure windows drivers didn't yet get to it.
This was out for a very short time.

> -- 
> 
> Sasha.
> 

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Michael S. Tsirkin]

On Wed, Nov 09, 2011 at 02:36:43PM +0200, Pekka Enberg wrote:
> On Wed, 9 Nov 2011, Sasha Levin wrote:
> >They don't exist in kernel code either, for same reason as above.
> >
> >Nothing will break if we remove it since no one really used it, we were
> >probably the first and only implementation of the spec which considered
> >them :)
> 
> As long as we are able to run older versions of the KVM tool with
> newer kernels and vice versa, I see no reason why we can't drop 64-bit
> features from the KVM tool.
> 
> 			Pekka

What do these older versions do? Do they *set* bit 31 in the
features register?

-- 
MST

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Michael S. Tsirkin]

On Wed, Nov 09, 2011 at 02:48:58PM +0200, Sasha Levin wrote:
> I was unable to check if it was actually implemented in the drivers
> because
> http://git.kernel.org/?p=virt/kvm/kvm-guest-drivers-windows.git;a=summary is not quite there (*cough*).

I found a mirror:
http://mirror.nexcess.net/kernel.org/scm/virt/kvm/kvm-guest-drivers-windows.git/
don't know how fresh it is.

> -- 
> 
> Sasha.
> 

[PATCHv2 RFC] virtio-spec: flexible configuration layout

[Michael S. Tsirkin]

Here's an updated spec patch. Changes from v1:
- add an example configuration
- scale tables so they fit within page
- clean up some remaining items mentioning feature bit 31

Description from v1:
Here's a spec change documenting what my C patch does
(almost - I tweaked the layout a bit, but the idea is the same).
Some more cleanups are needed but I thought I'd send it
for early flames/comments.

The idea is simple: we split functionally unrelated
register groups to independent structures, and let
the device place is anywhere using a capability
in PCI configuration space.

It can then go into MMIO space which is cheaper than PIO.

A legacy portion of the configuration is mirrored
in the first BAR, to keep legacy drivers working.
Any new fields can be added in existing structures
at the end, so they won't affect legacy.
Alternatively we can add new structures with new
structure IDs, pointed to from PCI configuration space.

As we don't yet have devices or drivers with 64 bit features,
I decided we don't need high feature bits in legacy space.
This also frees up feature bit 31 as we don't need it
to enable high feature bits anymore.
I don't think any released software actually uses it
(the kvm tool HEAD apparently does,
 but I think it's not too late to fix it).

As this solves the dynamic placement of MSIX vectors
and high feature bits,
I thought it's easier to just reserve space for that
programming than give it a separate structure. This
can be changed by a patch on top.

Note that data path is split from configuration.

PDF will follow.

Please comment.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

---

diff --git a/virtio-spec.lyx b/virtio-spec.lyx
index 6426f8f..2ce2553 100644
--- a/virtio-spec.lyx
+++ b/virtio-spec.lyx
@@ -3,6 +3,9 @@
 \begin_document
 \begin_header
 \textclass report
+\begin_preamble
+\usepackage{graphicx}
+\end_preamble
 \use_default_options false
 \maintain_unincluded_children false
 \language english
@@ -56,6 +59,7 @@
 \html_math_output 0
 \html_css_as_file 0
 \html_be_strict false
+\author 1986246365 "Michael S. Tsirkin" 
 \end_header
 
 \begin_body
@@ -570,7 +574,34 @@ Device Configuration
 \end_layout
 
 \begin_layout Standard
-To configure the device, we use the first I/O region of the PCI device.
+To configure the device, we 
+\change_inserted 1986246365 1320783354
+use I/O and/or memory regions of the PCI device.
+ These contain the virtio header registers, the notification register, the
+ ISR status register and device specific registers, as specified by Virtio
+ Structure PCI Capabilities
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320787659
+For backwards compatibility, devices should present part of these configuration
+ registers in a legacy configuration structure in the first I/O region of
+ the PCI device
+\change_unchanged
+
+\end_layout
+
+\end_inset
+
+.
+\end_layout
+
+\begin_layout Standard
+
+\change_deleted 1986246365 1320766512
+use the first I/O region of the PCI device.
  This contains a 
 \emph on
 virtio header
@@ -578,105 +609,3115 @@ virtio header
  followed by a 
 \emph on
 device-specific region.
+\change_unchanged
+
+\end_layout
+
+\begin_layout Standard
+There may be different widths of accesses to the I/O region; the 
+\begin_inset Quotes eld
+\end_inset
+
+natural
+\begin_inset Quotes erd
+\end_inset
+
+ access method for each field in the virtio header must be used (i.e.
+ 32-bit accesses for 32-bit fields, etc), but the device-specific region
+ can be accessed using any width accesses, and should obtain the same results.
+\end_layout
+
+\begin_layout Standard
+Note that this is possible because while the virtio header
+\change_inserted 1986246365 1320767249
+, notification and ISR status are
+\change_deleted 1986246365 1320767251
+ is
+\change_unchanged
+ PCI (i.e.
+ little) endian, the device-specific
+\change_deleted 1986246365 1320767335
+ region is
+\change_inserted 1986246365 1320767337
+registers are
+\change_unchanged
+ encoded in the native endian of the guest (where such distinction is applicable
+).
+\change_inserted 1986246365 1320767660
+
+\end_layout
+
+\begin_layout Subsection
+Device Initialization Sequence
+\begin_inset CommandInset label
+LatexCommand label
+name "sub:Device-Initialization-Sequence"
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+We start with an overview of device initialization, then expand on the details
+ of the device and how each step is preformed.
+\end_layout
+
+\begin_layout Enumerate
+Reset the device.
+ This is not required on initial start up.
+\end_layout
+
+\begin_layout Enumerate
+The ACKNOWLEDGE status bit is set: we have noticed the device.
+\end_layout
+
+\begin_layout Enumerate
+The DRIVER status bit is set: we know how to drive the device.
+\end_layout
+
+\begin_layout Enumerate
+
+\change_inserted 1986246365 1320838089
+PCI capability list scan, detecting virtio configuration layout using Virtio
+ Structure PCI capabilities.
+ 
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320838121
+Historically drivers skipped this step, using legacy configuration space
+ unconditionally.
+\change_unchanged
+
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Enumerate
+Device-specific setup, including reading the Device Feature Bits, discovery
+ of virtqueues for the device, optional MSI-X setup, and reading and possibly
+ writing the virtio configuration space.
+\end_layout
+
+\begin_layout Enumerate
+The subset of Device Feature Bits understood by the driver is written to
+ the device.
+\end_layout
+
+\begin_layout Enumerate
+The DRIVER_OK status bit is set.
+\end_layout
+
+\begin_layout Enumerate
+The device can now be used (ie.
+ buffers added to the virtqueues)
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+Historically, drivers have used the device before steps 5 and 6.
+ This is only allowed if the driver does not use any features which would
+ alter this early use of the device.
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+If any of these steps go irrecoverably wrong, the guest should set the FAILED
+ status bit to indicate that it has given up on the device (it can reset
+ the device later to restart if desired).
+\end_layout
+
+\begin_layout Standard
+We now cover the fields required for general setup in detail.
+\change_inserted 1986246365 1320766663
+
+\end_layout
+
+\begin_layout Subsection
+
+\change_inserted 1986246365 1320781584
+Virtio Structure PCI Capability
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320781061
+Virtio configuration layout includes virtio configuration header, notification
+ and ISR status and device configuration structures.
+ Each structure is mapped by a Base Address register (BAR) belonging to
+ the function, located beginning at 10h in Configuration Space.
+ 
+\emph on
+Structure ID 
+\emph default
+indentifies the structure
+\emph on
+, Size
+\emph default
+ indicates the structure size, in bytes, a 
+\emph on
+BAR Indicator register (BIR)
+\emph default
+ indicates which BAR, and 
+\emph on
+Offset
+\emph default
+ indicates where the structure begins relative to the base address associated
+ with the BAR.
+ The BAR is permitted to be either 32-bit or 64-bit, it can map Memory Space
+ or I/O Space.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320838522
+These registers are specified using vendor-specific PCI capability located
+ on capability list in PCI configuration space of the device.
+ This virtio structure capability uses little-endian format; all bits are
+ read-only:
+\end_layout
+
+\begin_layout Standard
+\begin_inset ERT
+status open
+
+\begin_layout Plain Layout
+
+
+\backslash
+resizebox{
+\backslash
+textwidth}{!}{%
+\end_layout
+
+\end_inset
+
+
+\change_inserted 1986246365 1320839073
+
+\begin_inset Tabular
+<lyxtabular version="3" rows="4" columns="34">
+<features tabularvalignment="bottom">
+<column alignment="center" valignment="top" width="0pt">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0pt">
+<row>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771774
+
+\size tiny
+Bit
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771777
+
+\size tiny
+31
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771779
+
+\size tiny
+30
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771781
+
+\size tiny
+29
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771783
+
+\size tiny
+28
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771786
+
+\size tiny
+27
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771810
+
+\size tiny
+26
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771813
+
+\size tiny
+25
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771815
+
+\size tiny
+24
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771817
+
+\size tiny
+23
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771819
+
+\size tiny
+22
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771821
+
+\size tiny
+21
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771823
+
+\size tiny
+20
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771825
+
+\size tiny
+19
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771829
+
+\size tiny
+18
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771831
+
+\size tiny
+17
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771833
+
+\size tiny
+16
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771835
+
+\size tiny
+15
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771840
+
+\size tiny
+14
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771841
+
+\size tiny
+13
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771843
+
+\size tiny
+12
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771845
+
+\size tiny
+11
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771847
+
+\size tiny
+10
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771850
+
+\size tiny
+9
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771852
+
+\size tiny
+8
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771854
+
+\size tiny
+7
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771856
+
+\size tiny
+6
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771858
+
+\size tiny
+5
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771859
+
+\size tiny
+4
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771861
+
+\size tiny
+3
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771862
+
+\size tiny
+2
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771864
+
+\size tiny
+1
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771888
+
+\size tiny
+0
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771997
+
+\size tiny
+Offset
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320781029
+
+\size larger
+Structure ID
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320850302
+
+\size larger
+CapLength = 12
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320850298
+
+\size larger
+NextCapPtr
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320850289
+
+\size larger
+CapID = 9
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320772004
+0
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320772549
+
+\size larger
+Size
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320850408
+
+\size larger
+BIR
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320772007
+4
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320772540
+
+\size larger
+Offset
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320772011
+8
+\end_layout
+
+\end_inset
+</cell>
+</row>
+</lyxtabular>
+
+\end_inset
+
+
+\change_unchanged
+
+\begin_inset ERT
+status open
+
+\begin_layout Plain Layout
+
+}
+\end_layout
+
+\end_inset
+
+
+\change_inserted 1986246365 1320839073
+
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320838992
+Purpose:
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320780912
+
+\emph on
+Capability ID
+\emph default
+, 
+\emph on
+Next Capability Pointer
+\emph default
+, 
+\emph on
+Capability Length
+\emph default
+ - these fields are specified by PCI local bus specification, Rev 3.0
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320785073
+
+\emph on
+Structure ID
+\emph default
+ - identifies the structure mapped.
+ If an ID representing a specific structure is absent, the structure is
+ located in the legacy virtio configuration in the first PCI BAR.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320782689
+
+\emph on
+Size
+\emph default
+ - size of the structure mapped.
+ This size might include padding, or fields unused by the driver.
+ Drivers are also recommended to only map part of configuration structure
+ large enough for device operation.
+ For example, a future device might present a large structure size of several
+ MBytes.
+ As current devices never utilize structures larger than 4KBytes in size,
+ driver can limit the mapped structure size to e.g.
+ 4KBytes to allow forward compatibility with such devices without loss of
+ functionality and without wasting resources.
+\change_unchanged
+
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320839833
+
+\emph on
+BIR - BAR Indicator Register
+\emph default
+ - Indicates which one of the BAR registers, located beginning at 10h in
+ Configuration Space, is used to map the structure into Memory or I/O Space.
+ Legal values are 0 - 5.
+ BAR offset is calculated as 
+\emph on
+BAR
+\emph default
+ = 10h + 4 * 
+\emph on
+BIR.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320781002
+
+\emph on
+Offset
+\emph default
+ - Used as an offset from the address contained by one of the function’s
+ Base Address registers to point to the base of the structure.
+ Depending on the structure, the lower bits are masked off (set to zero)
+ by software to form an aligned offset.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320785098
+A single device has multiple virtio capabilities on the PCI capability linked-li
+st.
+ To locate a specific register, driver scans the PCI capability list looking
+ for capabilities using the vendor specific 
+\emph on
+Capability ID
+\emph default
+, that is 
+\emph on
+Capability ID
+\emph default
+ value 
+\emph on
+9
+\emph default
+, according to the PCI spec.
+ For each capability, it verifies that the 
+\emph on
+Capability Length
+\emph default
+ field matches the expected value (12) - if it doesn't, this indicates a
+ layout change, which causes driver to ignore the specific PCI capability
+ and proceed to the next PCI capability in the list.
+ Next,
+\emph on
+ Structure ID
+\emph default
+ single-byte field is read at offset 3.
+ Unrecognized 
+\emph on
+Structure ID
+\emph default
+ values are ignored.
+ Low bits in the 
+\emph on
+Offset
+\emph default
+ register are masked to match alignment requirements, if any.
+ Device drivers are recommended to sanity check 
+\emph on
+BIR
+\emph default
+, 
+\emph on
+Offset
+\emph default
+, and 
+\emph on
+Size
+\emph default
+ fields, ignoring the capability or failing gracefully on error.
+ Drivers are also recommended to only map part of configuration structure
+ large enough for device operation.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320786035
+For backwards compatibility, devices should also present legacy configuration
+ space in the first I/O region of the PCI device, mirroring virtio configuration
+ structures in this space.
+ Drivers should fall back on this legacy structure if a specific Virtio
+ Structure capability is missing in the PCI capability list.
+ However, devices are not required to map the whole configuration structure
+ in legacy space - only the initial segment expected by legacy drivers needs
+ to be mapped.
+ The minimal size of such legacy segment size, for each structure, is listed
+ below.
+ Drivers are discouraged from using legacy space if a specific Structure
+ ID is present; this is to allow deprecating the legacy space in the future.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320850770
+Below is the list of supported Virtio structures, including Structure ID,
+ offset alignment, and where it can be located in the legacy configuration
+ space.
+ Note that legacy space layout changes as the MSI-X capability is enabled
+ and disabled
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320856794
+ie.
+ once you enable MSI-X on the device, some legacy fields move.
+ If you turn it off again, they move back! this does not affect the new
+ configuration 
+\end_layout
+
+\end_inset
+
+.
+ Alignment, offset and all sizes are in bytes.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320786868
+\begin_inset Tabular
+<lyxtabular version="3" rows="6" columns="6">
+<features tabularvalignment="middle">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<row>
+<cell multirow="3" alignment="left" valignment="middle" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Structure
+\change_inserted 1986246365 1320855560
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multirow="3" alignment="left" valignment="middle" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+ID
+\change_inserted 1986246365 1320855560
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Align-
+\end_layout
+
+\end_inset
+</cell>
+<cell multirow="3" alignment="left" valignment="middle" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320855560
+MinSize
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320785821
+Offset in
+\change_deleted 1986246365 1320785817
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320850780
+MinSize in
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell multirow="4" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multirow="4" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320855625
+ment
+\end_layout
+
+\end_inset
+</cell>
+<cell multirow="4" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320855593
+legacy space
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320855610
+ legacy space
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320785855
+Virtio Header
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320785860
+0x1
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786157
+4
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786225
+28 bytes
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786145
+0
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320855180
+24 or 20
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320855511
+\begin_inset CommandInset label
+LatexCommand label
+name "fn:24-or-20"
+
+\end_inset
+
+24 bytes if MSI-X is enabled, 20 bytes if it is disabled
+\change_unchanged
+
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786188
+Queue Notify
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786191
+0x2
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786267
+2 bytes
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786228
+2 bytes
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786505
+16 bytes
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786303
+2
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786878
+ISR Status
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786260
+0x3
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786270
+1 byte
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786273
+1 byte
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786534
+19 bytes
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786306
+1
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786847
+Device Header
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786474
+0x4
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786477
+1 byte
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320855405
+?
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320855417
+\begin_inset CommandInset label
+LatexCommand label
+name "fn:Device-specific"
+
+\end_inset
+
+Device specific
+\change_unchanged
+
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320855534
+24 or 20
+\begin_inset Formula $^{\text{\ref{fn:24-or-20}}}$
+\end_inset
+
+
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320855432
+?
+\begin_inset Formula $^{\text{\ref{fn:Device-specific}}}$
+\end_inset
+
+
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+</lyxtabular>
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320786973
+Drivers should assume that each structure size can be extended in the future.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320859762
+Queue Notify and ISR status are single-field registers.
+ Device Header has device specific structure.
+ Virtio Header structure is documented below.
+\end_layout
+
+\begin_layout Subsubsection
+
+\change_inserted 1986246365 1320867253
+Example Virtio Structure Layout
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320865758
+Let's assume that a virtio block device presents two Virtio Structure capabiliti
+es in PCI capability list:
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320863392
+Structure ID 0x1; BIR 1; Offset 0x1000; Size 0x100
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320863399
+Structure ID 0x4; BIR 1; Offset 0x1100; Size 0x200
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320863479
+Let's further assume that MSI-X is disabled.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320863498
+In this case, the driver should use:
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320863543
+Virtio Header at offset 0x1000 in BAR 1
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320863568
+Device Header at offset 0x1100 in BAR1
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320868074
+Queue Notify at offset 0x10 in BAR0 (legacy configuration space is used
+ since Structure ID is not present)
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320868077
+ISR status at offset 0x13 in BAR0 (legacy configuration space is used since
+ Structure ID is not present)
+\change_unchanged
+
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320867162
+For use by legacy drivers, the first 20 bytes in BAR0 mirror bytes 0x1000
+ to 0x1013 in BAR1, and the following 32 bytes in BAR0 should mirror bytes
+ 0x1100 to 111F in BAR1.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320867231
+Enabling MSI-X only affects the legacy drivers: with MSI-X enabled, the
+ first 24 bytes in BAR0 mirror bytes 0x1000 to 0x1017 in BAR1, and the following
+ 32 bytes in BAR0 should mirror bytes 0x1100 to 111F in BAR1.
+\change_unchanged
+
+\end_layout
+
+\begin_layout Subsection
+
+\change_inserted 1986246365 1320855988
+Virtio Header
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320858571
+The virtio header looks as follows:
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320858597
+\begin_inset Tabular
+<lyxtabular version="3" rows="13" columns="3">
+<features tabularvalignment="middle">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<row>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858603
+Bits
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858614
+R/W
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858619
+Purpose
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858841
+32
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858858
+R
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858870
+Device Features bits 0:31
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858873
+32
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858876
+R+W
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858885
+Guest Features bits 0:31
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858889
+32
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858891
+R+W
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858907
+Queue Address
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859001
+16
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859006
+R
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858953
+Queue Size
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859012
+16
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859008
+R+W
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858957
+Queue Select
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859015
+16
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858964
+Reserved
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859029
+8
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859032
+R+W
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858972
+Device Status
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859052
+8
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858989
+Reserved
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859063
+16
+\change_unchanged
+
 \end_layout
 
-\begin_layout Standard
-There may be different widths of accesses to the I/O region; the 
-\begin_inset Quotes eld
 \end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
 
-natural
-\begin_inset Quotes erd
-\end_inset
+\begin_layout Plain Layout
 
- access method for each field in the virtio header must be used (i.e.
- 32-bit accesses for 32-bit fields, etc), but the device-specific region
- can be accessed using any width accesses, and should obtain the same results.
-\end_layout
+\change_inserted 1986246365 1320859065
+R+W
+\change_unchanged
 
-\begin_layout Standard
-Note that this is possible because while the virtio header is PCI (i.e.
- little) endian, the device-specific region is encoded in the native endian
- of the guest (where such distinction is applicable).
 \end_layout
 
-\begin_layout Subsection
-Device Initialization Sequence
-\begin_inset CommandInset label
-LatexCommand label
-name "sub:Device-Initialization-Sequence"
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859081
+MSI-X Configuration Vector
+\change_unchanged
+
+\end_layout
 
 \end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
 
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859083
+16
+\change_unchanged
 
 \end_layout
 
-\begin_layout Standard
-We start with an overview of device initialization, then expand on the details
- of the device and how each step is preformed.
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859088
+R+W
+\change_unchanged
+
 \end_layout
 
-\begin_layout Enumerate
-Reset the device.
- This is not required on initial start up.
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859095
+MSI-X Queue Vector
+\change_unchanged
+
 \end_layout
 
-\begin_layout Enumerate
-The ACKNOWLEDGE status bit is set: we have noticed the device.
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859102
+32
+\change_unchanged
+
 \end_layout
 
-\begin_layout Enumerate
-The DRIVER status bit is set: we know how to drive the device.
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859106
+R
+\change_unchanged
+
 \end_layout
 
-\begin_layout Enumerate
-Device-specific setup, including reading the Device Feature Bits, discovery
- of virtqueues for the device, optional MSI-X setup, and reading and possibly
- writing the virtio configuration space.
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859116
+Device Features bits 32:63
+\change_unchanged
+
 \end_layout
 
-\begin_layout Enumerate
-The subset of Device Feature Bits understood by the driver is written to
- the device.
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859122
+32
+\change_unchanged
+
 \end_layout
 
-\begin_layout Enumerate
-The DRIVER_OK status bit is set.
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859119
+R+W
+\change_unchanged
+
 \end_layout
 
-\begin_layout Enumerate
-The device can now be used (ie.
- buffers added to the virtqueues)
-\begin_inset Foot
-status open
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
 
 \begin_layout Plain Layout
-Historically, drivers have used the device before steps 5 and 6.
- This is only allowed if the driver does not use any features which would
- alter this early use of the device.
+
+\change_inserted 1986246365 1320859130
+Guest Features bits 32:63
+\change_unchanged
+
 \end_layout
 
 \end_inset
+</cell>
+</row>
+</lyxtabular>
+
+\end_inset
 
 
 \end_layout
 
-\begin_layout Standard
-If any of these steps go irrecoverably wrong, the guest should set the FAILED
- status bit to indicate that it has given up on the device (it can reset
- the device later to restart if desired).
+\begin_layout Subsubsection
+
+\change_inserted 1986246365 1320856004
+Legacy 
+\change_unchanged
+Virtio Header
 \end_layout
 
 \begin_layout Standard
-We now cover the fields required for general setup in detail.
-\end_layout
 
-\begin_layout Subsection
-Virtio Header
+\change_inserted 1986246365 1320867482
+For backwards compatibility, devices should mirror the initial part of virtio
+ header and device header structures, Queue Notify and ISR Status registers
+ in legacy configuration space in the first I/O region of the PCI device.
+ Drivers should fall back on this legacy structure if a specific Virtio
+ Structure capability is missing in the PCI capability list.
+ Drivers are discouraged from using legacy space if a specific Structure
+ ID is present; this is to allow deprecating the legacy space in the future.
 \end_layout
 
 \begin_layout Standard
-The virtio header looks as follows:
+
+\change_inserted 1986246365 1320859848
+For reference, the
+\change_deleted 1986246365 1320856044
+The
+\change_unchanged
+ 
+\change_inserted 1986246365 1320859848
+legacy 
+\change_unchanged
+virtio header looks as follows
+\change_inserted 1986246365 1320859848
+; one observes that most register offsets match the new virtio header, to
+ make compatibility easier
+\change_unchanged
+:
 \end_layout
 
 \begin_layout Standard
@@ -1067,11 +4108,15 @@ Status
 If MSI-X is enabled for the device, two additional fields immediately follow
  this header:
 \begin_inset Foot
-status collapsed
+status open
 
 \begin_layout Plain Layout
 ie.
- once you enable MSI-X on the device, the other fields move.
+ once you enable MSI-X on the device, the other fields 
+\change_inserted 1986246365 1320867540
+in legacy configuration 
+\change_unchanged
+move.
  If you turn it off again, they move back!
 \end_layout
 
@@ -1219,11 +4264,15 @@ Vector
 \end_layout
 
 \begin_layout Standard
+
+\change_deleted 1986246365 1320867563
 Finally, if feature bits (VIRTIO_F_FEATURES_HI) this is immediately followed
  by two additional fields:
 \end_layout
 
 \begin_layout Standard
+
+\change_deleted 1986246365 1320867563
 \begin_inset Tabular
 <lyxtabular version="3" rows="4" columns="3">
 <features tabularvalignment="middle">
@@ -1359,11 +4408,16 @@ Features bits 32:63
 \end_inset
 
 
+\change_unchanged
+
 \end_layout
 
 \begin_layout Standard
-Immediately following these general headers, there may be device-specific
- headers:
+Immediately following these general headers, there may be 
+\change_inserted 1986246365 1320856347
+legacy 
+\change_unchanged
+device-specific headers:
 \end_layout
 
 \begin_layout Standard
@@ -1459,6 +4513,17 @@ Device Specific...
 \end_inset
 
 
+\change_inserted 1986246365 1320856108
+
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320856109
+Legacy device-specific headers are supported for backwards compatibility.
+ As such, they are not required to map the whole device configuration structure
+ - only the initial segment expected by legacy drivers needs to be mapped.
+ The size of such legacy segment size is device specific.
 \end_layout
 
 \begin_layout Subsubsection
@@ -1531,8 +4596,10 @@ name "sub:Feature-Bits"
 The least significant 31 bits of the first configuration field indicates
  the features that the device supports (the high bit is reserved, and will
  be used to indicate the presence of future feature bits elsewhere).
- If more than 31 feature bits are supported, the device indicates so by
- setting feature bit 31 (see 
+ 
+\change_deleted 1986246365 1320784929
+If more than 31 feature bits are supported, the device indicates so by setting
+ feature bit 31 (see 
 \begin_inset CommandInset ref
 LatexCommand ref
 reference "cha:Reserved-Feature-Bits"
@@ -1540,7 +4607,9 @@ reference "cha:Reserved-Feature-Bits"
 \end_inset
 
 ).
- The bits are allocated as follows:
+ 
+\change_unchanged
+The bits are allocated as follows:
 \end_layout
 
 \begin_layout Description
@@ -1615,10 +4684,14 @@ This allows for forwards and backwards compatibility: if the device is enhanced
 \end_layout
 
 \begin_layout Standard
+
+\change_deleted 1986246365 1320784940
 Access to feature bits 32 to 63 is enabled by Guest by setting feature bit
  31.
  If this bit is unset, Device must assume that all feature bits > 31 are
  unset.
+\change_unchanged
+
 \end_layout
 
 \begin_layout Subsubsection
@@ -1629,10 +4702,14 @@ Configuration/Queue Vectors
 When MSI-X capability is present and enabled in the device (through standard
  PCI configuration space) 4 bytes at byte offset 20 are used to map configuratio
 n change and queue interrupts to MSI-X vectors.
- In this case, the ISR Status field is unused, and device specific configuration
- starts at byte offset 24 in virtio header structure.
+ In this case, the ISR Status field is unused
+\change_deleted 1986246365 1320784981
+, and device specific configuration starts at byte offset 24 in virtio header
+ structure.
  When MSI-X capability is not enabled, device specific configuration starts
- at byte offset 20 in virtio header.
+ at byte offset 20 in virtio header
+\change_unchanged
+.
 \end_layout
 
 \begin_layout Standard
@@ -3829,7 +6906,13 @@ Appendix B: Reserved Feature Bits
 \end_layout
 
 \begin_layout Standard
-Currently there are five device-independent feature bits defined:
+Currently there are 
+\change_deleted 1986246365 1320867366
+five
+\change_inserted 1986246365 1320867367
+four
+\change_unchanged
+ device-independent feature bits defined:
 \end_layout
 
 \begin_layout Description
@@ -3926,9 +7009,13 @@ An experimental virtio PCI driver contained in Linux version 2.6.25 had this
 \end_layout
 
 \begin_layout Description
+
+\change_deleted 1986246365 1320867370
 VIRTIO_F_FEATURES_HIGH(31) This feature indicates that the device supports
  feature bits 32:63.
  If unset, feature bits 32:63 are unset.
+\change_unchanged
+
 \end_layout
 
 \begin_layout Chapter*
@@ -4186,6 +7273,24 @@ struct virtio_net_config {
 
 \end_layout
 
+\begin_layout Description
+
+\change_inserted 1986246365 1320867743
+Device
+\begin_inset space ~
+\end_inset
+
+legacy
+\begin_inset space ~
+\end_inset
+
+configuration
+\begin_inset space ~
+\end_inset
+
+size for this device is 8 bytes.
+\end_layout
+
 \begin_layout Section*
 Device Initialization
 \end_layout
@@ -5226,6 +8331,24 @@ struct virtio_blk_config {
 
 \end_layout
 
+\begin_layout Description
+
+\change_inserted 1986246365 1320867779
+Device
+\begin_inset space ~
+\end_inset
+
+legacy
+\begin_inset space ~
+\end_inset
+
+configuration
+\begin_inset space ~
+\end_inset
+
+size for this device is 32 bytes.
+\end_layout
+
 \begin_layout Section*
 Device Initialization
 \end_layout
@@ -5780,6 +8903,24 @@ struct virtio_console_config {
 
 \end_layout
 
+\begin_layout Description
+
+\change_inserted 1986246365 1320867806
+Device
+\begin_inset space ~
+\end_inset
+
+legacy
+\begin_inset space ~
+\end_inset
+
+configuration
+\begin_inset space ~
+\end_inset
+
+size for this device is 8 bytes.
+\end_layout
+
 \begin_layout Section*
 Device Initialization
 \end_layout
@@ -5994,6 +9135,28 @@ configuration
 \end_inset
 
 layout None currently defined.
+\change_inserted 1986246365 1320867840
+
+\end_layout
+
+\begin_layout Description
+
+\change_inserted 1986246365 1320867845
+Device
+\begin_inset space ~
+\end_inset
+
+legacy
+\begin_inset space ~
+\end_inset
+
+configuration
+\begin_inset space ~
+\end_inset
+
+size for this device is 0 bytes.
+\change_unchanged
+
 \end_layout
 
 \begin_layout Section*
@@ -6127,6 +9290,24 @@ struct virtio_balloon_config {
 
 \end_layout
 
+\begin_layout Description
+
+\change_inserted 1986246365 1320867865
+Device
+\begin_inset space ~
+\end_inset
+
+legacy
+\begin_inset space ~
+\end_inset
+
+configuration
+\begin_inset space ~
+\end_inset
+
+size for this device is 4 bytes.
+\end_layout
+
 \begin_layout Section*
 Device Initialization
 \end_layout

Re: [PATCHv2 RFC] virtio-spec: flexible configuration layout

[Sasha Levin]

On Wed, 2011-11-09 at 21:59 +0200, Michael S. Tsirkin wrote:

[snip]

> +\begin_layout Enumerate
> +Reset the device.
> + This is not required on initial start up.
> +\end_layout
> +
> +\begin_layout Enumerate
> +The ACKNOWLEDGE status bit is set: we have noticed the device.
> +\end_layout
> +
> +\begin_layout Enumerate
> +The DRIVER status bit is set: we know how to drive the device.
> +\end_layout
> +
> +\begin_layout Enumerate
> +
> +\change_inserted 1986246365 1320838089
> +PCI capability list scan, detecting virtio configuration layout using Virtio
> + Structure PCI capabilities.

Does the legacy space always gets mapped from BAR0?

If yes, It'll be a bit harder deprecating it in the future.

If not, this should actually be #1. Previous 3 bullets here were already
setting the status byte in the device config.

[snip]

> +<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
> +\begin_inset Text
> +
> +\begin_layout Plain Layout
> +
> +\change_inserted 1986246365 1320786225
> +28 bytes

This is the new virtio header min size, shouldn't it be 32? (20 + 4
(msi-x) + 4 (guest feature) + 4 (device features)).

-- 

Sasha.

Re: [PATCHv2 RFC] virtio-spec: flexible configuration layout

[Michael S. Tsirkin]

On Wed, Nov 09, 2011 at 10:24:47PM +0200, Sasha Levin wrote:
> On Wed, 2011-11-09 at 21:59 +0200, Michael S. Tsirkin wrote:
> 
> [snip]
> 
> > +\begin_layout Enumerate
> > +Reset the device.
> > + This is not required on initial start up.
> > +\end_layout
> > +
> > +\begin_layout Enumerate
> > +The ACKNOWLEDGE status bit is set: we have noticed the device.
> > +\end_layout
> > +
> > +\begin_layout Enumerate
> > +The DRIVER status bit is set: we know how to drive the device.
> > +\end_layout
> > +
> > +\begin_layout Enumerate
> > +
> > +\change_inserted 1986246365 1320838089
> > +PCI capability list scan, detecting virtio configuration layout using Virtio
> > + Structure PCI capabilities.
> 
> Does the legacy space always gets mapped from BAR0?
> 
> If yes,

Yes and this is repeated in several places. Not clear? How can this
be made clearer?

> It'll be a bit harder deprecating it in the future.

Harder than ... what ?
IMO there's no way to put legacy anywhere except the first BAR
without breaking existing guests.

> If not, this should actually be #1. Previous 3 bullets here were already
> setting the status byte in the device config.
> 
> [snip]
> 
> > +<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
> > +\begin_inset Text
> > +
> > +\begin_layout Plain Layout
> > +
> > +\change_inserted 1986246365 1320786225
> > +28 bytes
> 
> This is the new virtio header min size, shouldn't it be 32? (20 + 4
> (msi-x) + 4 (guest feature) + 4 (device features)).

Good catch, I'll correct this.

> -- 
> 
> Sasha.

Re: [PATCHv2 RFC] virtio-spec: flexible configuration layout

[Sasha Levin]

On Wed, 2011-11-09 at 22:52 +0200, Michael S. Tsirkin wrote:
> On Wed, Nov 09, 2011 at 10:24:47PM +0200, Sasha Levin wrote:
> > On Wed, 2011-11-09 at 21:59 +0200, Michael S. Tsirkin wrote:
> > 
> > [snip]
> > 
> > > +\begin_layout Enumerate
> > > +Reset the device.
> > > + This is not required on initial start up.
> > > +\end_layout
> > > +
> > > +\begin_layout Enumerate
> > > +The ACKNOWLEDGE status bit is set: we have noticed the device.
> > > +\end_layout
> > > +
> > > +\begin_layout Enumerate
> > > +The DRIVER status bit is set: we know how to drive the device.
> > > +\end_layout
> > > +
> > > +\begin_layout Enumerate
> > > +
> > > +\change_inserted 1986246365 1320838089
> > > +PCI capability list scan, detecting virtio configuration layout using Virtio
> > > + Structure PCI capabilities.
> > 
> > Does the legacy space always gets mapped from BAR0?
> > 
> > If yes,
> 
> Yes and this is repeated in several places. Not clear? How can this
> be made clearer?

Do you mean comments such as "For backwards compatibility, devices
should also present legacy configuration space in the first I/O region
of the PCI device"? What I understood from it is that the device should
have a legacy config in case it's used with an older guest, but I didn't
understand from it that the legacy config will be used even if new
layout is present.

> > It'll be a bit harder deprecating it in the future.
> 
> Harder than ... what ?

Harder than allowing devices not to present it at all if new layout
config is used. Right now the simple implementation is to use MMIO for
config and device specific, and let it fallback to legacy for ISR and
notifications (and therefore, this is probably how everybody will
implement it), which means that when you do want to deprecate legacy,
there will be extra work to be done then, instead of doing it now.

> IMO there's no way to put legacy anywhere except the first BAR
> without breaking existing guests.

It's not about where we put legacy, it's about how easy it is to drop
legacy entirely.

-- 

Sasha.

Re: [PATCHv2 RFC] virtio-spec: flexible configuration layout

[Sasha Levin]

On Wed, 2011-11-09 at 23:14 +0200, Michael S. Tsirkin wrote:
> On Wed, Nov 09, 2011 at 10:57:28PM +0200, Sasha Levin wrote:
> > On Wed, 2011-11-09 at 22:52 +0200, Michael S. Tsirkin wrote:
> > > On Wed, Nov 09, 2011 at 10:24:47PM +0200, Sasha Levin wrote:
> > > > On Wed, 2011-11-09 at 21:59 +0200, Michael S. Tsirkin wrote:
> > > > 
> > > > [snip]
> > > > 
> > > > > +\begin_layout Enumerate
> > > > > +Reset the device.
> > > > > + This is not required on initial start up.
> > > > > +\end_layout
> > > > > +
> > > > > +\begin_layout Enumerate
> > > > > +The ACKNOWLEDGE status bit is set: we have noticed the device.
> > > > > +\end_layout
> > > > > +
> > > > > +\begin_layout Enumerate
> > > > > +The DRIVER status bit is set: we know how to drive the device.
> > > > > +\end_layout
> > > > > +
> > > > > +\begin_layout Enumerate
> > > > > +
> > > > > +\change_inserted 1986246365 1320838089
> > > > > +PCI capability list scan, detecting virtio configuration layout using Virtio
> > > > > + Structure PCI capabilities.
> > > > 
> > > > Does the legacy space always gets mapped from BAR0?
> > > > 
> > > > If yes,
> > > 
> > > Yes and this is repeated in several places. Not clear? How can this
> > > be made clearer?
> > 
> > Do you mean comments such as "For backwards compatibility, devices
> > should also present legacy configuration space in the first I/O region
> > of the PCI device"? What I understood from it is that the device should
> > have a legacy config in case it's used with an older guest, but I didn't
> > understand from it that the legacy config will be used even if new
> > layout is present.
> 
> Yes, this is what I meant. New guest is required to use the new space
> and not the legacy one. So you dont need a legacy space for the at all.
> But practically, we'll need to support old guests for a long while.
> 
> > > > It'll be a bit harder deprecating it in the future.
> > > 
> > > Harder than ... what ?
> > 
> > Harder than allowing devices not to present it at all if new layout
> > config is used.
> 
> Yes, it's allowed if you know you have a new guest. It says
> explicitly that drivers are required to use new capabilities
> is they are there.
> 
> > Right now the simple implementation is to use MMIO for
> > config and device specific, and let it fallback to legacy for ISR and
> > notifications (and therefore, this is probably how everybody will
> > implement it), which means that when you do want to deprecate legacy,
> > there will be extra work to be done then, instead of doing it now.
> 
> If hypervisors don't implement the new layout then drivers will
> have to keep supporting the old one. I don't think we can do
> much about that.
> 
> > > IMO there's no way to put legacy anywhere except the first BAR
> > > without breaking existing guests.
> > 
> > It's not about where we put legacy, it's about how easy it is to drop
> > legacy entirely.
> 
> We can only do this after all guests and hypervisors are updated. When
> they are, we can drop legacy from drivers and hypervisors, and
> I don't see a way to make it easier.

Well, in that case, why does the PCI cap probing is #4 in the device
init list? Shouldn't we getting the layout and mapping it before we
write to the status byte?

-- 

Sasha.

Re: [PATCHv2 RFC] virtio-spec: flexible configuration layout

[Michael S. Tsirkin]

On Wed, Nov 09, 2011 at 10:57:28PM +0200, Sasha Levin wrote:
> On Wed, 2011-11-09 at 22:52 +0200, Michael S. Tsirkin wrote:
> > On Wed, Nov 09, 2011 at 10:24:47PM +0200, Sasha Levin wrote:
> > > On Wed, 2011-11-09 at 21:59 +0200, Michael S. Tsirkin wrote:
> > > 
> > > [snip]
> > > 
> > > > +\begin_layout Enumerate
> > > > +Reset the device.
> > > > + This is not required on initial start up.
> > > > +\end_layout
> > > > +
> > > > +\begin_layout Enumerate
> > > > +The ACKNOWLEDGE status bit is set: we have noticed the device.
> > > > +\end_layout
> > > > +
> > > > +\begin_layout Enumerate
> > > > +The DRIVER status bit is set: we know how to drive the device.
> > > > +\end_layout
> > > > +
> > > > +\begin_layout Enumerate
> > > > +
> > > > +\change_inserted 1986246365 1320838089
> > > > +PCI capability list scan, detecting virtio configuration layout using Virtio
> > > > + Structure PCI capabilities.
> > > 
> > > Does the legacy space always gets mapped from BAR0?
> > > 
> > > If yes,
> > 
> > Yes and this is repeated in several places. Not clear? How can this
> > be made clearer?
> 
> Do you mean comments such as "For backwards compatibility, devices
> should also present legacy configuration space in the first I/O region
> of the PCI device"? What I understood from it is that the device should
> have a legacy config in case it's used with an older guest, but I didn't
> understand from it that the legacy config will be used even if new
> layout is present.

Yes, this is what I meant. New guest is required to use the new space
and not the legacy one. So you dont need a legacy space for the at all.
But practically, we'll need to support old guests for a long while.

> > > It'll be a bit harder deprecating it in the future.
> > 
> > Harder than ... what ?
> 
> Harder than allowing devices not to present it at all if new layout
> config is used.

Yes, it's allowed if you know you have a new guest. It says
explicitly that drivers are required to use new capabilities
is they are there.

> Right now the simple implementation is to use MMIO for
> config and device specific, and let it fallback to legacy for ISR and
> notifications (and therefore, this is probably how everybody will
> implement it), which means that when you do want to deprecate legacy,
> there will be extra work to be done then, instead of doing it now.

If hypervisors don't implement the new layout then drivers will
have to keep supporting the old one. I don't think we can do
much about that.

> > IMO there's no way to put legacy anywhere except the first BAR
> > without breaking existing guests.
> 
> It's not about where we put legacy, it's about how easy it is to drop
> legacy entirely.

We can only do this after all guests and hypervisors are updated. When
they are, we can drop legacy from drivers and hypervisors, and
I don't see a way to make it easier.

> -- 
> 
> Sasha.

Re: [PATCHv2 RFC] virtio-spec: flexible configuration layout

[Michael S. Tsirkin]

On Wed, Nov 09, 2011 at 11:13:56PM +0200, Sasha Levin wrote:
> On Wed, 2011-11-09 at 23:14 +0200, Michael S. Tsirkin wrote:
> > On Wed, Nov 09, 2011 at 10:57:28PM +0200, Sasha Levin wrote:
> > > On Wed, 2011-11-09 at 22:52 +0200, Michael S. Tsirkin wrote:
> > > > On Wed, Nov 09, 2011 at 10:24:47PM +0200, Sasha Levin wrote:
> > > > > On Wed, 2011-11-09 at 21:59 +0200, Michael S. Tsirkin wrote:
> > > > > 
> > > > > [snip]
> > > > > 
> > > > > > +\begin_layout Enumerate
> > > > > > +Reset the device.
> > > > > > + This is not required on initial start up.
> > > > > > +\end_layout
> > > > > > +
> > > > > > +\begin_layout Enumerate
> > > > > > +The ACKNOWLEDGE status bit is set: we have noticed the device.
> > > > > > +\end_layout
> > > > > > +
> > > > > > +\begin_layout Enumerate
> > > > > > +The DRIVER status bit is set: we know how to drive the device.
> > > > > > +\end_layout
> > > > > > +
> > > > > > +\begin_layout Enumerate
> > > > > > +
> > > > > > +\change_inserted 1986246365 1320838089
> > > > > > +PCI capability list scan, detecting virtio configuration layout using Virtio
> > > > > > + Structure PCI capabilities.
> > > > > 
> > > > > Does the legacy space always gets mapped from BAR0?
> > > > > 
> > > > > If yes,
> > > > 
> > > > Yes and this is repeated in several places. Not clear? How can this
> > > > be made clearer?
> > > 
> > > Do you mean comments such as "For backwards compatibility, devices
> > > should also present legacy configuration space in the first I/O region
> > > of the PCI device"? What I understood from it is that the device should
> > > have a legacy config in case it's used with an older guest, but I didn't
> > > understand from it that the legacy config will be used even if new
> > > layout is present.
> > 
> > Yes, this is what I meant. New guest is required to use the new space
> > and not the legacy one. So you dont need a legacy space for the at all.
> > But practically, we'll need to support old guests for a long while.
> > 
> > > > > It'll be a bit harder deprecating it in the future.
> > > > 
> > > > Harder than ... what ?
> > > 
> > > Harder than allowing devices not to present it at all if new layout
> > > config is used.
> > 
> > Yes, it's allowed if you know you have a new guest. It says
> > explicitly that drivers are required to use new capabilities
> > is they are there.
> > 
> > > Right now the simple implementation is to use MMIO for
> > > config and device specific, and let it fallback to legacy for ISR and
> > > notifications (and therefore, this is probably how everybody will
> > > implement it), which means that when you do want to deprecate legacy,
> > > there will be extra work to be done then, instead of doing it now.
> > 
> > If hypervisors don't implement the new layout then drivers will
> > have to keep supporting the old one. I don't think we can do
> > much about that.
> > 
> > > > IMO there's no way to put legacy anywhere except the first BAR
> > > > without breaking existing guests.
> > > 
> > > It's not about where we put legacy, it's about how easy it is to drop
> > > legacy entirely.
> > 
> > We can only do this after all guests and hypervisors are updated. When
> > they are, we can drop legacy from drivers and hypervisors, and
> > I don't see a way to make it easier.
> 
> Well, in that case, why does the PCI cap probing is #4 in the device
> init list? Shouldn't we getting the layout and mapping it before we
> write to the status byte?

True, this is actually how it's done in the driver.
Good catch, I'll correct the text, thanks.

> -- 
> 
> Sasha.

[PATCHv3 RFC] virtio-spec: flexible configuration layout

[Michael S. Tsirkin]

Another update. I'll get to testing the linux guest patch now,
next version should hopefully be a non-RFC one, and be accompanied by
guest driver, assuming driver works fine.

Alignment negotiation is still TBD, it's easy to add but I'd like
to get agreement on the general approach first.
One drawback is that for device config header,
we can't reuse same memory for both legacy and new -
this is because the legacy offset changes dynamically,
and I'd like to avoid doing that with the new one.
This should not be a problem as device header can
go into MMIO which is not as space constrained as PIO.

Changes from v2:
- Clarify that 'first region' is BAR0.
- Clarify that devices should provide all capabilities
  (fallback is for compatibility).
- Remove a left-over mention of bit 31 use.
- Correct virtio header size.

Changes from v1:
- add an example configuration
- scale tables so they fit within page
- clean up some remaining items mentioning feature bit 31

Description from v1:
Here's a spec change documenting what my C patch does
(almost - I tweaked the layout a bit, but the idea is the same).
Some more cleanups are needed but I thought I'd send it
for early flames/comments.

The idea is simple: we split functionally unrelated
register groups to independent structures, and let
the device place is anywhere using a capability
in PCI configuration space.

It can then go into MMIO space which is cheaper than PIO.

A legacy portion of the configuration is mirrored
in the first BAR, to keep legacy drivers working.
Any new fields can be added in existing structures
at the end, so they won't affect legacy.
Alternatively we can add new structures with new
structure IDs, pointed to from PCI configuration space.

As we don't yet have devices or drivers with 64 bit features,
I decided we don't need high feature bits in legacy space.
This also frees up feature bit 31 as we don't need it
to enable high feature bits anymore.
I don't think any released software actually uses it
(the kvm tool HEAD apparently does,
 but I think it's not too late to fix it).

As this solves the dynamic placement of MSIX vectors
and high feature bits,
I thought it's easier to just reserve space for that
programming than give it a separate structure. This
can be changed by a patch on top.

Note that data path is split from configuration.

PDF will follow.

Please comment.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

---

diff --git a/virtio-spec.lyx b/virtio-spec.lyx
index 6426f8f..36ac61f 100644
--- a/virtio-spec.lyx
+++ b/virtio-spec.lyx
@@ -3,6 +3,9 @@
 \begin_document
 \begin_header
 \textclass report
+\begin_preamble
+\usepackage{graphicx}
+\end_preamble
 \use_default_options false
 \maintain_unincluded_children false
 \language english
@@ -56,6 +59,7 @@
 \html_math_output 0
 \html_css_as_file 0
 \html_be_strict false
+\author 1986246365 "Michael S. Tsirkin" 
 \end_header
 
 \begin_body
@@ -570,7 +574,34 @@ Device Configuration
 \end_layout
 
 \begin_layout Standard
-To configure the device, we use the first I/O region of the PCI device.
+To configure the device, we 
+\change_inserted 1986246365 1320783354
+use I/O and/or memory regions of the PCI device.
+ These contain the virtio header registers, the notification register, the
+ ISR status register and device specific registers, as specified by Virtio
+ Structure PCI Capabilities
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320920233
+For backwards compatibility, devices should present part of these configuration
+ registers in a legacy configuration structure in BAR0 in the first I/O
+ region of the PCI device
+\change_unchanged
+
+\end_layout
+
+\end_inset
+
+.
+\end_layout
+
+\begin_layout Standard
+
+\change_deleted 1986246365 1320766512
+use the first I/O region of the PCI device.
  This contains a 
 \emph on
 virtio header
@@ -578,105 +609,3120 @@ virtio header
  followed by a 
 \emph on
 device-specific region.
+\change_unchanged
+
+\end_layout
+
+\begin_layout Standard
+There may be different widths of accesses to the I/O region; the 
+\begin_inset Quotes eld
+\end_inset
+
+natural
+\begin_inset Quotes erd
+\end_inset
+
+ access method for each field in the virtio header must be used (i.e.
+ 32-bit accesses for 32-bit fields, etc), but the device-specific region
+ can be accessed using any width accesses, and should obtain the same results.
+\end_layout
+
+\begin_layout Standard
+Note that this is possible because while the virtio header
+\change_inserted 1986246365 1320767249
+, notification and ISR status are
+\change_deleted 1986246365 1320767251
+ is
+\change_unchanged
+ PCI (i.e.
+ little) endian, the device-specific
+\change_deleted 1986246365 1320767335
+ region is
+\change_inserted 1986246365 1320767337
+registers are
+\change_unchanged
+ encoded in the native endian of the guest (where such distinction is applicable
+).
+\change_inserted 1986246365 1320767660
+
+\end_layout
+
+\begin_layout Subsection
+Device Initialization Sequence
+\begin_inset CommandInset label
+LatexCommand label
+name "sub:Device-Initialization-Sequence"
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+We start with an overview of device initialization, then expand on the details
+ of the device and how each step is preformed.
+\end_layout
+
+\begin_layout Enumerate
+Reset the device.
+ This is not required on initial start up.
+\change_inserted 1986246365 1320918056
+
+\end_layout
+
+\begin_layout Enumerate
+
+\change_inserted 1986246365 1320918057
+PCI capability list scan, detecting virtio configuration layout using Virtio
+ Structure PCI capabilities.
+ 
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320918057
+Historically drivers skipped this step, using legacy configuration space
+ unconditionally.
+\end_layout
+
+\end_inset
+
+
+\change_unchanged
+
+\end_layout
+
+\begin_layout Enumerate
+The ACKNOWLEDGE status bit is set: we have noticed the device.
+\end_layout
+
+\begin_layout Enumerate
+The DRIVER status bit is set: we know how to drive the device.
+\end_layout
+
+\begin_layout Enumerate
+Device-specific setup, including reading the Device Feature Bits, discovery
+ of virtqueues for the device, optional MSI-X setup, and reading and possibly
+ writing the virtio configuration space.
+\end_layout
+
+\begin_layout Enumerate
+The subset of Device Feature Bits understood by the driver is written to
+ the device.
+\end_layout
+
+\begin_layout Enumerate
+The DRIVER_OK status bit is set.
+\end_layout
+
+\begin_layout Enumerate
+The device can now be used (ie.
+ buffers added to the virtqueues)
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+Historically, drivers have used the device before steps 5 and 6.
+ This is only allowed if the driver does not use any features which would
+ alter this early use of the device.
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+If any of these steps go irrecoverably wrong, the guest should set the FAILED
+ status bit to indicate that it has given up on the device (it can reset
+ the device later to restart if desired).
+\end_layout
+
+\begin_layout Standard
+We now cover the fields required for general setup in detail.
+\change_inserted 1986246365 1320766663
+
+\end_layout
+
+\begin_layout Subsection
+
+\change_inserted 1986246365 1320781584
+Virtio Structure PCI Capability
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320781061
+Virtio configuration layout includes virtio configuration header, notification
+ and ISR status and device configuration structures.
+ Each structure is mapped by a Base Address register (BAR) belonging to
+ the function, located beginning at 10h in Configuration Space.
+ 
+\emph on
+Structure ID 
+\emph default
+indentifies the structure
+\emph on
+, Size
+\emph default
+ indicates the structure size, in bytes, a 
+\emph on
+BAR Indicator register (BIR)
+\emph default
+ indicates which BAR, and 
+\emph on
+Offset
+\emph default
+ indicates where the structure begins relative to the base address associated
+ with the BAR.
+ The BAR is permitted to be either 32-bit or 64-bit, it can map Memory Space
+ or I/O Space.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320838522
+These registers are specified using vendor-specific PCI capability located
+ on capability list in PCI configuration space of the device.
+ This virtio structure capability uses little-endian format; all bits are
+ read-only:
+\end_layout
+
+\begin_layout Standard
+\begin_inset ERT
+status open
+
+\begin_layout Plain Layout
+
+
+\backslash
+resizebox{
+\backslash
+textwidth}{!}{%
+\end_layout
+
+\end_inset
+
+
+\change_inserted 1986246365 1320839073
+
+\begin_inset Tabular
+<lyxtabular version="3" rows="4" columns="34">
+<features tabularvalignment="bottom">
+<column alignment="center" valignment="top" width="0pt">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0pt">
+<row>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771774
+
+\size tiny
+Bit
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771777
+
+\size tiny
+31
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771779
+
+\size tiny
+30
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771781
+
+\size tiny
+29
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771783
+
+\size tiny
+28
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771786
+
+\size tiny
+27
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771810
+
+\size tiny
+26
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771813
+
+\size tiny
+25
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771815
+
+\size tiny
+24
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771817
+
+\size tiny
+23
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771819
+
+\size tiny
+22
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771821
+
+\size tiny
+21
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771823
+
+\size tiny
+20
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771825
+
+\size tiny
+19
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771829
+
+\size tiny
+18
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771831
+
+\size tiny
+17
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771833
+
+\size tiny
+16
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771835
+
+\size tiny
+15
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771840
+
+\size tiny
+14
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771841
+
+\size tiny
+13
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771843
+
+\size tiny
+12
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771845
+
+\size tiny
+11
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771847
+
+\size tiny
+10
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771850
+
+\size tiny
+9
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771852
+
+\size tiny
+8
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771854
+
+\size tiny
+7
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771856
+
+\size tiny
+6
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771858
+
+\size tiny
+5
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771859
+
+\size tiny
+4
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771861
+
+\size tiny
+3
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771862
+
+\size tiny
+2
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771864
+
+\size tiny
+1
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771888
+
+\size tiny
+0
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320771997
+
+\size tiny
+Offset
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320781029
+
+\size larger
+Structure ID
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320850302
+
+\size larger
+CapLength = 12
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320850298
+
+\size larger
+NextCapPtr
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320850289
+
+\size larger
+CapID = 9
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320772004
+0
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320772549
+
+\size larger
+Size
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320850408
+
+\size larger
+BIR
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320772007
+4
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320772540
+
+\size larger
+Offset
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320772011
+8
+\end_layout
+
+\end_inset
+</cell>
+</row>
+</lyxtabular>
+
+\end_inset
+
+
+\change_unchanged
+
+\begin_inset ERT
+status open
+
+\begin_layout Plain Layout
+
+}
+\end_layout
+
+\end_inset
+
+
+\change_inserted 1986246365 1320839073
+
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320838992
+Purpose:
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320780912
+
+\emph on
+Capability ID
+\emph default
+, 
+\emph on
+Next Capability Pointer
+\emph default
+, 
+\emph on
+Capability Length
+\emph default
+ - these fields are specified by PCI local bus specification, Rev 3.0
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320785073
+
+\emph on
+Structure ID
+\emph default
+ - identifies the structure mapped.
+ If an ID representing a specific structure is absent, the structure is
+ located in the legacy virtio configuration in the first PCI BAR.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320782689
+
+\emph on
+Size
+\emph default
+ - size of the structure mapped.
+ This size might include padding, or fields unused by the driver.
+ Drivers are also recommended to only map part of configuration structure
+ large enough for device operation.
+ For example, a future device might present a large structure size of several
+ MBytes.
+ As current devices never utilize structures larger than 4KBytes in size,
+ driver can limit the mapped structure size to e.g.
+ 4KBytes to allow forward compatibility with such devices without loss of
+ functionality and without wasting resources.
+\change_unchanged
+
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320839833
+
+\emph on
+BIR - BAR Indicator Register
+\emph default
+ - Indicates which one of the BAR registers, located beginning at 10h in
+ Configuration Space, is used to map the structure into Memory or I/O Space.
+ Legal values are 0 - 5.
+ BAR offset is calculated as 
+\emph on
+BAR
+\emph default
+ = 10h + 4 * 
+\emph on
+BIR.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320781002
+
+\emph on
+Offset
+\emph default
+ - Used as an offset from the address contained by one of the function’s
+ Base Address registers to point to the base of the structure.
+ Depending on the structure, the lower bits are masked off (set to zero)
+ by software to form an aligned offset.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320785098
+A single device has multiple virtio capabilities on the PCI capability linked-li
+st.
+ To locate a specific register, driver scans the PCI capability list looking
+ for capabilities using the vendor specific 
+\emph on
+Capability ID
+\emph default
+, that is 
+\emph on
+Capability ID
+\emph default
+ value 
+\emph on
+9
+\emph default
+, according to the PCI spec.
+ For each capability, it verifies that the 
+\emph on
+Capability Length
+\emph default
+ field matches the expected value (12) - if it doesn't, this indicates a
+ layout change, which causes driver to ignore the specific PCI capability
+ and proceed to the next PCI capability in the list.
+ Next,
+\emph on
+ Structure ID
+\emph default
+ single-byte field is read at offset 3.
+ Unrecognized 
+\emph on
+Structure ID
+\emph default
+ values are ignored.
+ Low bits in the 
+\emph on
+Offset
+\emph default
+ register are masked to match alignment requirements, if any.
+ Device drivers are recommended to sanity check 
+\emph on
+BIR
+\emph default
+, 
+\emph on
+Offset
+\emph default
+, and 
+\emph on
+Size
+\emph default
+ fields, ignoring the capability or failing gracefully on error.
+ Drivers are also recommended to only map part of configuration structure
+ large enough for device operation.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320920511
+For backwards compatibility, devices should also present legacy configuration
+ space in BAR0 in the first I/O region of the PCI device, mirroring virtio
+ configuration structures in this space.
+ For forwards compatibility, drivers should fall back on this legacy structure
+ if a specific Virtio Structure capability is missing in the PCI capability
+ list.
+ However, devices are not required to map the whole configuration structure
+ in legacy space - only the initial segment expected by legacy drivers needs
+ to be mapped.
+ Devices are further required to implement all Structure IDs for the structures
+ they present, as opposed to relying on driver legacy fallback.
+ The minimal size of such legacy segment size, for each structure, is listed
+ below.
+ Drivers are discouraged from using legacy space if a specific Structure
+ ID is present; this is to allow deprecating the legacy space in the future.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320850770
+Below is the list of supported Virtio structures, including Structure ID,
+ offset alignment, and where it can be located in the legacy configuration
+ space.
+ Note that legacy space layout changes as the MSI-X capability is enabled
+ and disabled
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320856794
+ie.
+ once you enable MSI-X on the device, some legacy fields move.
+ If you turn it off again, they move back! this does not affect the new
+ configuration 
+\end_layout
+
+\end_inset
+
+.
+ Alignment, offset and all sizes are in bytes.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320786868
+\begin_inset Tabular
+<lyxtabular version="3" rows="6" columns="6">
+<features tabularvalignment="middle">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<row>
+<cell multirow="3" alignment="left" valignment="middle" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Structure
+\change_inserted 1986246365 1320855560
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multirow="3" alignment="left" valignment="middle" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+ID
+\change_inserted 1986246365 1320855560
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Align-
+\end_layout
+
+\end_inset
+</cell>
+<cell multirow="3" alignment="left" valignment="middle" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320855560
+MinSize
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320785821
+Offset in
+\change_deleted 1986246365 1320785817
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320850780
+MinSize in
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell multirow="4" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multirow="4" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320855625
+ment
+\end_layout
+
+\end_inset
+</cell>
+<cell multirow="4" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320855593
+legacy space
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320855610
+ legacy space
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320785855
+Virtio Header
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320785860
+0x1
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786157
+4
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320918196
+32 bytes
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786145
+0
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320855180
+24 or 20
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320855511
+\begin_inset CommandInset label
+LatexCommand label
+name "fn:24-or-20"
+
+\end_inset
+
+24 bytes if MSI-X is enabled, 20 bytes if it is disabled
+\change_unchanged
+
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786188
+Queue Notify
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786191
+0x2
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786267
+2 bytes
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786228
+2 bytes
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786505
+16 bytes
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786303
+2
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786878
+ISR Status
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786260
+0x3
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786270
+1 byte
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786273
+1 byte
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786534
+19 bytes
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786306
+1
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786847
+Device Header
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786474
+0x4
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320786477
+1 byte
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320918121
+...
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320855417
+\begin_inset CommandInset label
+LatexCommand label
+name "fn:Device-specific"
+
+\end_inset
+
+Device specific
+\change_unchanged
+
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320855534
+24 or 20
+\begin_inset Formula $^{\text{\ref{fn:24-or-20}}}$
+\end_inset
+
+
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320918125
+...
+\begin_inset Formula $^{\text{\ref{fn:Device-specific}}}$
+\end_inset
+
+
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+</lyxtabular>
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320786973
+Drivers should assume that each structure size can be extended in the future.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320859762
+Queue Notify and ISR status are single-field registers.
+ Device Header has device specific structure.
+ Virtio Header structure is documented below.
+\end_layout
+
+\begin_layout Subsubsection
+
+\change_inserted 1986246365 1320867253
+Example Virtio Structure Layout
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320865758
+Let's assume that a virtio block device presents two Virtio Structure capabiliti
+es in PCI capability list:
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320863392
+Structure ID 0x1; BIR 1; Offset 0x1000; Size 0x100
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320863399
+Structure ID 0x4; BIR 1; Offset 0x1100; Size 0x200
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320863479
+Let's further assume that MSI-X is disabled.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320863498
+In this case, the driver should use:
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320863543
+Virtio Header at offset 0x1000 in BAR 1
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320863568
+Device Header at offset 0x1100 in BAR1
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320868074
+Queue Notify at offset 0x10 in BAR0 (legacy configuration space is used
+ since Structure ID is not present)
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320868077
+ISR status at offset 0x13 in BAR0 (legacy configuration space is used since
+ Structure ID is not present)
+\change_unchanged
+
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320867162
+For use by legacy drivers, the first 20 bytes in BAR0 mirror bytes 0x1000
+ to 0x1013 in BAR1, and the following 32 bytes in BAR0 should mirror bytes
+ 0x1100 to 111F in BAR1.
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320867231
+Enabling MSI-X only affects the legacy drivers: with MSI-X enabled, the
+ first 24 bytes in BAR0 mirror bytes 0x1000 to 0x1017 in BAR1, and the following
+ 32 bytes in BAR0 should mirror bytes 0x1100 to 111F in BAR1.
+\change_unchanged
+
+\end_layout
+
+\begin_layout Subsection
+
+\change_inserted 1986246365 1320855988
+Virtio Header
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320858571
+The virtio header looks as follows:
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320858597
+\begin_inset Tabular
+<lyxtabular version="3" rows="13" columns="3">
+<features tabularvalignment="middle">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<column alignment="center" valignment="top" width="0">
+<row>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858603
+Bits
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858614
+R/W
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858619
+Purpose
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858841
+32
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858858
+R
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858870
+Device Features bits 0:31
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858873
+32
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858876
+R+W
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858885
+Guest Features bits 0:31
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858889
+32
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858891
+R+W
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858907
+Queue Address
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859001
+16
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859006
+R
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858953
+Queue Size
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859012
+16
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859008
+R+W
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858957
+Queue Select
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859015
+16
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858964
+Reserved
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859029
+8
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859032
+R+W
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858972
+Device Status
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859052
+8
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320858989
+Reserved
+\change_unchanged
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859063
+16
+\change_unchanged
+
 \end_layout
 
-\begin_layout Standard
-There may be different widths of accesses to the I/O region; the 
-\begin_inset Quotes eld
 \end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
 
-natural
-\begin_inset Quotes erd
-\end_inset
+\begin_layout Plain Layout
 
- access method for each field in the virtio header must be used (i.e.
- 32-bit accesses for 32-bit fields, etc), but the device-specific region
- can be accessed using any width accesses, and should obtain the same results.
-\end_layout
+\change_inserted 1986246365 1320859065
+R+W
+\change_unchanged
 
-\begin_layout Standard
-Note that this is possible because while the virtio header is PCI (i.e.
- little) endian, the device-specific region is encoded in the native endian
- of the guest (where such distinction is applicable).
 \end_layout
 
-\begin_layout Subsection
-Device Initialization Sequence
-\begin_inset CommandInset label
-LatexCommand label
-name "sub:Device-Initialization-Sequence"
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859081
+MSI-X Configuration Vector
+\change_unchanged
+
+\end_layout
 
 \end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
 
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859083
+16
+\change_unchanged
 
 \end_layout
 
-\begin_layout Standard
-We start with an overview of device initialization, then expand on the details
- of the device and how each step is preformed.
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859088
+R+W
+\change_unchanged
+
 \end_layout
 
-\begin_layout Enumerate
-Reset the device.
- This is not required on initial start up.
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859095
+MSI-X Queue Vector
+\change_unchanged
+
 \end_layout
 
-\begin_layout Enumerate
-The ACKNOWLEDGE status bit is set: we have noticed the device.
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859102
+32
+\change_unchanged
+
 \end_layout
 
-\begin_layout Enumerate
-The DRIVER status bit is set: we know how to drive the device.
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859106
+R
+\change_unchanged
+
 \end_layout
 
-\begin_layout Enumerate
-Device-specific setup, including reading the Device Feature Bits, discovery
- of virtqueues for the device, optional MSI-X setup, and reading and possibly
- writing the virtio configuration space.
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859116
+Device Features bits 32:63
+\change_unchanged
+
 \end_layout
 
-\begin_layout Enumerate
-The subset of Device Feature Bits understood by the driver is written to
- the device.
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859122
+32
+\change_unchanged
+
 \end_layout
 
-\begin_layout Enumerate
-The DRIVER_OK status bit is set.
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\change_inserted 1986246365 1320859119
+R+W
+\change_unchanged
+
 \end_layout
 
-\begin_layout Enumerate
-The device can now be used (ie.
- buffers added to the virtqueues)
-\begin_inset Foot
-status open
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
 
 \begin_layout Plain Layout
-Historically, drivers have used the device before steps 5 and 6.
- This is only allowed if the driver does not use any features which would
- alter this early use of the device.
+
+\change_inserted 1986246365 1320859130
+Guest Features bits 32:63
+\change_unchanged
+
 \end_layout
 
 \end_inset
+</cell>
+</row>
+</lyxtabular>
+
+\end_inset
 
 
 \end_layout
 
-\begin_layout Standard
-If any of these steps go irrecoverably wrong, the guest should set the FAILED
- status bit to indicate that it has given up on the device (it can reset
- the device later to restart if desired).
+\begin_layout Subsubsection
+
+\change_inserted 1986246365 1320856004
+Legacy 
+\change_unchanged
+Virtio Header
 \end_layout
 
 \begin_layout Standard
-We now cover the fields required for general setup in detail.
-\end_layout
 
-\begin_layout Subsection
-Virtio Header
+\change_inserted 1986246365 1320867482
+For backwards compatibility, devices should mirror the initial part of virtio
+ header and device header structures, Queue Notify and ISR Status registers
+ in legacy configuration space in the first I/O region of the PCI device.
+ Drivers should fall back on this legacy structure if a specific Virtio
+ Structure capability is missing in the PCI capability list.
+ Drivers are discouraged from using legacy space if a specific Structure
+ ID is present; this is to allow deprecating the legacy space in the future.
 \end_layout
 
 \begin_layout Standard
-The virtio header looks as follows:
+
+\change_inserted 1986246365 1320859848
+For reference, the
+\change_deleted 1986246365 1320856044
+The
+\change_unchanged
+ 
+\change_inserted 1986246365 1320859848
+legacy 
+\change_unchanged
+virtio header looks as follows
+\change_inserted 1986246365 1320859848
+; one observes that most register offsets match the new virtio header, to
+ make compatibility easier
+\change_unchanged
+:
 \end_layout
 
 \begin_layout Standard
@@ -1067,11 +4113,15 @@ Status
 If MSI-X is enabled for the device, two additional fields immediately follow
  this header:
 \begin_inset Foot
-status collapsed
+status open
 
 \begin_layout Plain Layout
 ie.
- once you enable MSI-X on the device, the other fields move.
+ once you enable MSI-X on the device, the other fields 
+\change_inserted 1986246365 1320867540
+in legacy configuration 
+\change_unchanged
+move.
  If you turn it off again, they move back!
 \end_layout
 
@@ -1219,11 +4269,15 @@ Vector
 \end_layout
 
 \begin_layout Standard
+
+\change_deleted 1986246365 1320867563
 Finally, if feature bits (VIRTIO_F_FEATURES_HI) this is immediately followed
  by two additional fields:
 \end_layout
 
 \begin_layout Standard
+
+\change_deleted 1986246365 1320867563
 \begin_inset Tabular
 <lyxtabular version="3" rows="4" columns="3">
 <features tabularvalignment="middle">
@@ -1359,11 +4413,16 @@ Features bits 32:63
 \end_inset
 
 
+\change_unchanged
+
 \end_layout
 
 \begin_layout Standard
-Immediately following these general headers, there may be device-specific
- headers:
+Immediately following these general headers, there may be 
+\change_inserted 1986246365 1320856347
+legacy 
+\change_unchanged
+device-specific headers:
 \end_layout
 
 \begin_layout Standard
@@ -1459,6 +4518,17 @@ Device Specific...
 \end_inset
 
 
+\change_inserted 1986246365 1320856108
+
+\end_layout
+
+\begin_layout Standard
+
+\change_inserted 1986246365 1320856109
+Legacy device-specific headers are supported for backwards compatibility.
+ As such, they are not required to map the whole device configuration structure
+ - only the initial segment expected by legacy drivers needs to be mapped.
+ The size of such legacy segment size is device specific.
 \end_layout
 
 \begin_layout Subsubsection
@@ -1531,8 +4601,10 @@ name "sub:Feature-Bits"
 The least significant 31 bits of the first configuration field indicates
  the features that the device supports (the high bit is reserved, and will
  be used to indicate the presence of future feature bits elsewhere).
- If more than 31 feature bits are supported, the device indicates so by
- setting feature bit 31 (see 
+ 
+\change_deleted 1986246365 1320784929
+If more than 31 feature bits are supported, the device indicates so by setting
+ feature bit 31 (see 
 \begin_inset CommandInset ref
 LatexCommand ref
 reference "cha:Reserved-Feature-Bits"
@@ -1540,7 +4612,9 @@ reference "cha:Reserved-Feature-Bits"
 \end_inset
 
 ).
- The bits are allocated as follows:
+ 
+\change_unchanged
+The bits are allocated as follows:
 \end_layout
 
 \begin_layout Description
@@ -1615,10 +4689,14 @@ This allows for forwards and backwards compatibility: if the device is enhanced
 \end_layout
 
 \begin_layout Standard
+
+\change_deleted 1986246365 1320784940
 Access to feature bits 32 to 63 is enabled by Guest by setting feature bit
  31.
  If this bit is unset, Device must assume that all feature bits > 31 are
  unset.
+\change_unchanged
+
 \end_layout
 
 \begin_layout Subsubsection
@@ -1629,10 +4707,14 @@ Configuration/Queue Vectors
 When MSI-X capability is present and enabled in the device (through standard
  PCI configuration space) 4 bytes at byte offset 20 are used to map configuratio
 n change and queue interrupts to MSI-X vectors.
- In this case, the ISR Status field is unused, and device specific configuration
- starts at byte offset 24 in virtio header structure.
+ In this case, the ISR Status field is unused
+\change_deleted 1986246365 1320784981
+, and device specific configuration starts at byte offset 24 in virtio header
+ structure.
  When MSI-X capability is not enabled, device specific configuration starts
- at byte offset 20 in virtio header.
+ at byte offset 20 in virtio header
+\change_unchanged
+.
 \end_layout
 
 \begin_layout Standard
@@ -3829,7 +6911,13 @@ Appendix B: Reserved Feature Bits
 \end_layout
 
 \begin_layout Standard
-Currently there are five device-independent feature bits defined:
+Currently there are 
+\change_deleted 1986246365 1320867366
+five
+\change_inserted 1986246365 1320867367
+four
+\change_unchanged
+ device-independent feature bits defined:
 \end_layout
 
 \begin_layout Description
@@ -3926,9 +7014,13 @@ An experimental virtio PCI driver contained in Linux version 2.6.25 had this
 \end_layout
 
 \begin_layout Description
+
+\change_deleted 1986246365 1320867370
 VIRTIO_F_FEATURES_HIGH(31) This feature indicates that the device supports
  feature bits 32:63.
  If unset, feature bits 32:63 are unset.
+\change_unchanged
+
 \end_layout
 
 \begin_layout Chapter*
@@ -4186,6 +7278,24 @@ struct virtio_net_config {
 
 \end_layout
 
+\begin_layout Description
+
+\change_inserted 1986246365 1320867743
+Device
+\begin_inset space ~
+\end_inset
+
+legacy
+\begin_inset space ~
+\end_inset
+
+configuration
+\begin_inset space ~
+\end_inset
+
+size for this device is 8 bytes.
+\end_layout
+
 \begin_layout Section*
 Device Initialization
 \end_layout
@@ -5226,6 +8336,24 @@ struct virtio_blk_config {
 
 \end_layout
 
+\begin_layout Description
+
+\change_inserted 1986246365 1320867779
+Device
+\begin_inset space ~
+\end_inset
+
+legacy
+\begin_inset space ~
+\end_inset
+
+configuration
+\begin_inset space ~
+\end_inset
+
+size for this device is 32 bytes.
+\end_layout
+
 \begin_layout Section*
 Device Initialization
 \end_layout
@@ -5780,6 +8908,24 @@ struct virtio_console_config {
 
 \end_layout
 
+\begin_layout Description
+
+\change_inserted 1986246365 1320867806
+Device
+\begin_inset space ~
+\end_inset
+
+legacy
+\begin_inset space ~
+\end_inset
+
+configuration
+\begin_inset space ~
+\end_inset
+
+size for this device is 8 bytes.
+\end_layout
+
 \begin_layout Section*
 Device Initialization
 \end_layout
@@ -5994,6 +9140,28 @@ configuration
 \end_inset
 
 layout None currently defined.
+\change_inserted 1986246365 1320867840
+
+\end_layout
+
+\begin_layout Description
+
+\change_inserted 1986246365 1320867845
+Device
+\begin_inset space ~
+\end_inset
+
+legacy
+\begin_inset space ~
+\end_inset
+
+configuration
+\begin_inset space ~
+\end_inset
+
+size for this device is 0 bytes.
+\change_unchanged
+
 \end_layout
 
 \begin_layout Section*
@@ -6127,6 +9295,24 @@ struct virtio_balloon_config {
 
 \end_layout
 
+\begin_layout Description
+
+\change_inserted 1986246365 1320867865
+Device
+\begin_inset space ~
+\end_inset
+
+legacy
+\begin_inset space ~
+\end_inset
+
+configuration
+\begin_inset space ~
+\end_inset
+
+size for this device is 4 bytes.
+\end_layout
+
 \begin_layout Section*
 Device Initialization
 \end_layout

Re: [PATCHv2 RFC] virtio-spec: flexible configuration layout

[Rusty Russell]

On Wed, 09 Nov 2011 22:57:28 +0200, Sasha Levin <levinsasha928@gmail.com> wrote:
> On Wed, 2011-11-09 at 22:52 +0200, Michael S. Tsirkin wrote:
> > On Wed, Nov 09, 2011 at 10:24:47PM +0200, Sasha Levin wrote:
> > > It'll be a bit harder deprecating it in the future.
> > 
> > Harder than ... what ?
> 
> Harder than allowing devices not to present it at all if new layout
> config is used. Right now the simple implementation is to use MMIO for
> config and device specific, and let it fallback to legacy for ISR and
> notifications (and therefore, this is probably how everybody will
> implement it), which means that when you do want to deprecate legacy,
> there will be extra work to be done then, instead of doing it now.

Indeed, I'd like to see two changes to your proposal:

(1) It should be all or nothing.  If a driver can find the virtio header
    capability, it should only use the capabilties.  Otherwise, it
    should fall back to legacy.  Your draft suggests a mix is possible;
    I prefer a clean failure (ie. one day don't present a BAR 0 *at
    all*, so ancient drivers just fail to load.).

(2) There's no huge win in keeping the same layout.  Let's make some
    cleanups.  There are more users ahead of us then behind us (I
    hope!).

But I think this is the right direction!

Thanks,
Rusty.

Re: [PATCHv2 RFC] virtio-spec: flexible configuration layout

[Sasha Levin]

On Fri, Nov 11, 2011 at 6:24 AM, Rusty Russell <rusty@rustcorp.com.au> wrote:
> On Wed, 09 Nov 2011 22:57:28 +0200, Sasha Levin <levinsasha928@gmail.com> wrote:
>> On Wed, 2011-11-09 at 22:52 +0200, Michael S. Tsirkin wrote:
>> > On Wed, Nov 09, 2011 at 10:24:47PM +0200, Sasha Levin wrote:
>> > > It'll be a bit harder deprecating it in the future.
>> >
>> > Harder than ... what ?
>>
>> Harder than allowing devices not to present it at all if new layout
>> config is used. Right now the simple implementation is to use MMIO for
>> config and device specific, and let it fallback to legacy for ISR and
>> notifications (and therefore, this is probably how everybody will
>> implement it), which means that when you do want to deprecate legacy,
>> there will be extra work to be done then, instead of doing it now.
>
> Indeed, I'd like to see two changes to your proposal:
>
> (1) It should be all or nothing.  If a driver can find the virtio header
>    capability, it should only use the capabilties.  Otherwise, it
>    should fall back to legacy.  Your draft suggests a mix is possible;
>    I prefer a clean failure (ie. one day don't present a BAR 0 *at
>    all*, so ancient drivers just fail to load.).
>
> (2) There's no huge win in keeping the same layout.  Let's make some
>    cleanups.  There are more users ahead of us then behind us (I
>    hope!).

Actually, if we already do cleanups, here are two more suggestions:

1. Make 64bit features a one big 64bit block, instead of having 32bits
in one place and 32 in another.
2. Remove the reserved fields out of the config (the ones that were
caused by moving the ISR and the notifications out).

> But I think this is the right direction!
>
> Thanks,
> Rusty.
>

Also, an unrelated questions: With PIO, requests were ordered, which
means that if we wrote to the queue selector and then read from a
queue register we would read the correct queue info.
Is the same thing assured to us with MMIO? If we write to a queue
selector and immediately read from queue info would we be reading the
right info, or is there the slight chance that it would get reordered
and we would be reading queue info first and writing to the selector
later?

Re: [PATCHv2 RFC] virtio-spec: flexible configuration layout

[Michael S. Tsirkin]

On Fri, Nov 11, 2011 at 09:39:13AM +0200, Sasha Levin wrote:
> On Fri, Nov 11, 2011 at 6:24 AM, Rusty Russell <rusty@rustcorp.com.au> wrote:
> > On Wed, 09 Nov 2011 22:57:28 +0200, Sasha Levin <levinsasha928@gmail.com> wrote:
> >> On Wed, 2011-11-09 at 22:52 +0200, Michael S. Tsirkin wrote:
> >> > On Wed, Nov 09, 2011 at 10:24:47PM +0200, Sasha Levin wrote:
> >> > > It'll be a bit harder deprecating it in the future.
> >> >
> >> > Harder than ... what ?
> >>
> >> Harder than allowing devices not to present it at all if new layout
> >> config is used. Right now the simple implementation is to use MMIO for
> >> config and device specific, and let it fallback to legacy for ISR and
> >> notifications (and therefore, this is probably how everybody will
> >> implement it), which means that when you do want to deprecate legacy,
> >> there will be extra work to be done then, instead of doing it now.
> >
> > Indeed, I'd like to see two changes to your proposal:
> >
> > (1) It should be all or nothing.  If a driver can find the virtio header
> >    capability, it should only use the capabilties.  Otherwise, it
> >    should fall back to legacy.  Your draft suggests a mix is possible;
> >    I prefer a clean failure (ie. one day don't present a BAR 0 *at
> >    all*, so ancient drivers just fail to load.).
> >
> > (2) There's no huge win in keeping the same layout.  Let's make some
> >    cleanups.  There are more users ahead of us then behind us (I
> >    hope!).
> 
> Actually, if we already do cleanups, here are two more suggestions:
> 
> 1. Make 64bit features a one big 64bit block, instead of having 32bits
> in one place and 32 in another.
> 2. Remove the reserved fields out of the config (the ones that were
> caused by moving the ISR and the notifications out).
> 
> > But I think this is the right direction!
> >
> > Thanks,
> > Rusty.
> >
> 
> Also, an unrelated questions: With PIO, requests were ordered, which
> means that if we wrote to the queue selector and then read from a
> queue register we would read the correct queue info.
> Is the same thing assured to us with MMIO?

For real PCI, reads do not bypass writes in PCI. However this
is only true if both are MMIO or both PIO reads.
I don't think the ordering of MMIO versus PIO is guaranteed.

On KVM, the kernel doesn't do anything to guarantee ordering.
So you get the natural ordering of the CPU.

> If we write to a queue
> selector and immediately read from queue info would we be reading the
> right info, or is there the slight chance that it would get reordered
> and we would be reading queue info first and writing to the selector
> later?

The thing to realize is that write to queue selector with KVM is in the
end performed by host. And reading queue info means that host will be
reading the queue selector. So this is a write followed by read
from the same address. AFAIK no CPUs can reorder such accesses,
so you get the right info.

-- 
MST

Re: [PATCHv2 RFC] virtio-spec: flexible configuration layout

[Michael S. Tsirkin]

On Fri, Nov 11, 2011 at 02:54:31PM +1030, Rusty Russell wrote:
> On Wed, 09 Nov 2011 22:57:28 +0200, Sasha Levin <levinsasha928@gmail.com> wrote:
> > On Wed, 2011-11-09 at 22:52 +0200, Michael S. Tsirkin wrote:
> > > On Wed, Nov 09, 2011 at 10:24:47PM +0200, Sasha Levin wrote:
> > > > It'll be a bit harder deprecating it in the future.
> > > 
> > > Harder than ... what ?
> > 
> > Harder than allowing devices not to present it at all if new layout
> > config is used. Right now the simple implementation is to use MMIO for
> > config and device specific, and let it fallback to legacy for ISR and
> > notifications (and therefore, this is probably how everybody will
> > implement it), which means that when you do want to deprecate legacy,
> > there will be extra work to be done then, instead of doing it now.
> 
> Indeed, I'd like to see two changes to your proposal:
> 
> (1) It should be all or nothing.  If a driver can find the virtio header
>     capability, it should only use the capabilties.  Otherwise, it
>     should fall back to legacy.  Your draft suggests a mix is possible;
>     I prefer a clean failure (ie. one day don't present a BAR 0 *at
>     all*, so ancient drivers just fail to load.).
> (2) There's no huge win in keeping the same layout.  Let's make some
>     cleanups.  There are more users ahead of us then behind us (I
>     hope!).
> But I think this is the right direction!
> 
> Thanks,
> Rusty.

I'll do these changes, thanks!

Re: [PATCHv2 RFC] virtio-spec: flexible configuration layout

[Pawel Moll]

> > Also, an unrelated questions: With PIO, requests were ordered, which
> > means that if we wrote to the queue selector and then read from a
> > queue register we would read the correct queue info.
> > Is the same thing assured to us with MMIO?
> 
> For real PCI, reads do not bypass writes in PCI. However this
> is only true if both are MMIO or both PIO reads.
> I don't think the ordering of MMIO versus PIO is guaranteed.
> 
> On KVM, the kernel doesn't do anything to guarantee ordering.
> So you get the natural ordering of the CPU.
> 
> > If we write to a queue
> > selector and immediately read from queue info would we be reading the
> > right info, or is there the slight chance that it would get reordered
> > and we would be reading queue info first and writing to the selector
> > later?
> 
> The thing to realize is that write to queue selector with KVM is in the
> end performed by host. And reading queue info means that host will be
> reading the queue selector. So this is a write followed by read
> from the same address. AFAIK no CPUs can reorder such accesses,
> so you get the right info.

(As far as I understand all the complexity ;-) Memory mapped using
ioremap()-like functions should preserve access order. In case of ARM
architecture such memory region is defined (at the page tables level) as
a "device memory" (contrary to "normal memory") and the processor will
not try to be clever about it. I know next-to-nothing about x86, but I
suppose similar idea applies.

Cheers!

Paweł

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Ronen Hod]

On 11/09/2011 05:19 PM, Michael S. Tsirkin wrote:
> On Wed, Nov 09, 2011 at 02:48:58PM +0200, Sasha Levin wrote:
>> On Wed, 2011-11-09 at 14:38 +0200, Avi Kivity wrote:
>>> On 11/09/2011 10:46 AM, Sasha Levin wrote:
>>>>> Alternatively we can add new structures with new
>>>>> structure IDs, pointed to from PCI configuration space.
>>>>>
>>>>> As we don't yet have devices or drivers with 64 bit features,
>>>>> I decided we don't need high feature bits in legacy space.
>>>>> This also frees up feature bit 31 as we don't need it
>>>>> to enable high feature bits anymore.
>>>> KVM tool actually has support for 64bit features, we can probably remove
>>>> that when Pekka isn't looking :)
>>>>
>>> What about the Windows drivers?
>> While 64-bit features were defined, there was no actual devices to use
>> them.
>>
>> So even if Windows drivers had the ability to support 64-bit features,
>> no device ever needed it therefore no device ever activated it.
>>
>> A potential issue might arise when we remap feature bit 31 to do
>> something else, and when activated it would fool Windows drivers to
>> think that 64-bit features are now activated, while they're not - but
>> thats not something that should happen since legacy layout shouldn't
>> have that field enabled as per the spec (no new features in legacy
>> code).
>>
>> I was unable to check if it was actually implemented in the drivers
>> because
>> http://git.kernel.org/?p=virt/kvm/kvm-guest-drivers-windows.git;a=summary is not quite there (*cough*).

https://github.com/YanVugenfirer/kvm-guest-drivers-windows

> I'm pretty sure windows drivers didn't yet get to it.
> This was out for a very short time.
>
>> -- 
>>
>> Sasha.
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCHv2 RFC] virtio-spec: flexible configuration layout

[Michael S. Tsirkin]

On Fri, Nov 11, 2011 at 02:54:31PM +1030, Rusty Russell wrote:
> On Wed, 09 Nov 2011 22:57:28 +0200, Sasha Levin <levinsasha928@gmail.com> wrote:
> > On Wed, 2011-11-09 at 22:52 +0200, Michael S. Tsirkin wrote:
> > > On Wed, Nov 09, 2011 at 10:24:47PM +0200, Sasha Levin wrote:
> > > > It'll be a bit harder deprecating it in the future.
> > > 
> > > Harder than ... what ?
> > 
> > Harder than allowing devices not to present it at all if new layout
> > config is used. Right now the simple implementation is to use MMIO for
> > config and device specific, and let it fallback to legacy for ISR and
> > notifications (and therefore, this is probably how everybody will
> > implement it), which means that when you do want to deprecate legacy,
> > there will be extra work to be done then, instead of doing it now.
> 
> Indeed, I'd like to see two changes to your proposal:
> 
> (1) It should be all or nothing.  If a driver can find the virtio header
>     capability, it should only use the capabilties.  Otherwise, it
>     should fall back to legacy.

Okay, but going forward, if we add more capabilities, we probably won't
want to require them and fail to load if not there.  That's really why I
wanted to make the failover ignore any capability separately - to make
this future proof.  I'm not terribly fixated on this, it just seemed a
bit more symmetrical to treat all capabilities in the same way. Hmm?

>     Your draft suggests a mix is possible;
>     I prefer a clean failure (ie. one day don't present a BAR 0 *at
>     all*, so ancient drivers just fail to load.).

Just to clarify, as written in draft this is possible with the current
spec proposal.  So I'm guessing there's some other motivation that you
had in mind?

> (2) There's no huge win in keeping the same layout.  Let's make some
>     cleanups.

About this last point - what cleanups do you have in mind?  Just move
some registers around?  I guess we could put feature bits near each
other, and move device status towards the end to avoid wasting 3 bytes.
The win seems minimal, but the change does make legacy hypervisor
support in guests more cumbersome, as we need to spread coditional code
around instead of localizing it in the initialization path.

>    There are more users ahead of us then behind us (I
>     hope!).

In that case isn't it safe to assume we'll find some uses
for the reserved registers?

> But I think this is the right direction!
> 
> Thanks,
> Rusty.

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Vadim Rozenfeld]

On Wed, 2011-11-09 at 14:38 +0200, Avi Kivity wrote:
> On 11/09/2011 10:46 AM, Sasha Levin wrote:
> > > Alternatively we can add new structures with new
> > > structure IDs, pointed to from PCI configuration space.
> > > 
> > > As we don't yet have devices or drivers with 64 bit features,
> > > I decided we don't need high feature bits in legacy space.
> > > This also frees up feature bit 31 as we don't need it
> > > to enable high feature bits anymore.
> >
> > KVM tool actually has support for 64bit features, we can probably remove
> > that when Pekka isn't looking :)
> >
> 
> What about the Windows drivers?
> 
Probably some small adjustments will be required,
but I don't see any major problem here.

Vadim. 

Re: [PATCHv2 RFC] virtio-spec: flexible configuration layout

[Michael S. Tsirkin]

On Sun, Nov 13, 2011 at 05:14:27PM +0200, Michael S. Tsirkin wrote:
> On Fri, Nov 11, 2011 at 02:54:31PM +1030, Rusty Russell wrote:
> > On Wed, 09 Nov 2011 22:57:28 +0200, Sasha Levin <levinsasha928@gmail.com> wrote:
> > > On Wed, 2011-11-09 at 22:52 +0200, Michael S. Tsirkin wrote:
> > > > On Wed, Nov 09, 2011 at 10:24:47PM +0200, Sasha Levin wrote:
> > > > > It'll be a bit harder deprecating it in the future.
> > > > 
> > > > Harder than ... what ?
> > > 
> > > Harder than allowing devices not to present it at all if new layout
> > > config is used. Right now the simple implementation is to use MMIO for
> > > config and device specific, and let it fallback to legacy for ISR and
> > > notifications (and therefore, this is probably how everybody will
> > > implement it), which means that when you do want to deprecate legacy,
> > > there will be extra work to be done then, instead of doing it now.
> > 
> > Indeed, I'd like to see two changes to your proposal:
> > 
> > (1) It should be all or nothing.  If a driver can find the virtio header
> >     capability, it should only use the capabilties.  Otherwise, it
> >     should fall back to legacy.
> 
> Okay, but going forward, if we add more capabilities, we probably won't
> want to require them and fail to load if not there.  That's really why I
> wanted to make the failover ignore any capability separately - to make
> this future proof.  I'm not terribly fixated on this, it just seemed a
> bit more symmetrical to treat all capabilities in the same way. Hmm?
> 
> >     Your draft suggests a mix is possible;
> >     I prefer a clean failure (ie. one day don't present a BAR 0 *at
> >     all*, so ancient drivers just fail to load.).
> 
> Just to clarify, as written in draft this is possible with the current
> spec proposal.  So I'm guessing there's some other motivation that you
> had in mind?
> 
> > (2) There's no huge win in keeping the same layout.  Let's make some
> >     cleanups.
> 
> About this last point - what cleanups do you have in mind?  Just move
> some registers around?  I guess we could put feature bits near each
> other, and move device status towards the end to avoid wasting 3 bytes.
> The win seems minimal, but the change does make legacy hypervisor
> support in guests more cumbersome, as we need to spread coditional code
> around instead of localizing it in the initialization path.
> 
> >    There are more users ahead of us then behind us (I
> >     hope!).
> 
> In that case isn't it safe to assume we'll find some uses
> for the reserved registers?
> 
> > But I think this is the right direction!
> > 
> > Thanks,
> > Rusty.

Note: waiting on response to the above before I tweak the
spec again.

-- 
MST

Re: [PATCHv2 RFC] virtio-spec: flexible configuration layout

[Rusty Russell]

On Sun, 13 Nov 2011 17:14:27 +0200, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Fri, Nov 11, 2011 at 02:54:31PM +1030, Rusty Russell wrote:
> > Indeed, I'd like to see two changes to your proposal:
> > 
> > (1) It should be all or nothing.  If a driver can find the virtio header
> >     capability, it should only use the capabilties.  Otherwise, it
> >     should fall back to legacy.
> 
> Okay, but going forward, if we add more capabilities, we probably won't
> want to require them and fail to load if not there.  That's really why I
> wanted to make the failover ignore any capability separately - to make
> this future proof.  I'm not terribly fixated on this, it just seemed a
> bit more symmetrical to treat all capabilities in the same way. Hmm?

Sure, a future capbility may not exist.  But once a driver finds that
virtio header structure in the capability, it should *never* fall back
to the legacy area.  ie. it can expect that Queue Notify, ISR Status and
Device Header all exist.

ie. either use legacy mode, or use capabilities.  Never both.

> 
> >     Your draft suggests a mix is possible;
> >     I prefer a clean failure (ie. one day don't present a BAR 0 *at
> >     all*, so ancient drivers just fail to load.).
> 
> Just to clarify, as written in draft this is possible with the current
> spec proposal.  So I'm guessing there's some other motivation that you
> had in mind?

At the moment you give a hybrid model where both are used.  In five
years' time, that's going to be particularly ugly.
> 
> > (2) There's no huge win in keeping the same layout.  Let's make some
> >     cleanups.
> 
> About this last point - what cleanups do you have in mind?  Just move
> some registers around?  I guess we could put feature bits near each
> other, and move device status towards the end to avoid wasting 3
> bytes.

> The win seems minimal, but the change does make legacy hypervisor
> support in guests more cumbersome, as we need to spread coditional code
> around instead of localizing it in the initialization path.

But the separation between "legacy" and "modern" will be sharper, making
it easier to excise the legacy portion later.

And in five years' time, people implementing virtio will really thank us
that they can completely ignore the legacy header.

> >    There are more users ahead of us then behind us (I
> >     hope!).
> 
> In that case isn't it safe to assume we'll find some uses
> for the reserved registers?

How would we tell?  If we use a new capability struct for it, it's
obvious.  Otherwise, you're going to need to steal more feature bits.

Cheers,
Rusty.
PS.  Sorry, was off sick for a few days.

Re: [PATCHv2 RFC] virtio-spec: flexible configuration layout

[Rusty Russell]

On Fri, 11 Nov 2011 09:39:13 +0200, Sasha Levin <levinsasha928@gmail.com> wrote:
> On Fri, Nov 11, 2011 at 6:24 AM, Rusty Russell <rusty@rustcorp.com.au> wrote:
> > (2) There's no huge win in keeping the same layout.  Let's make some
> >    cleanups.  There are more users ahead of us then behind us (I
> >    hope!).
> 
> Actually, if we already do cleanups, here are two more suggestions:
> 
> 1. Make 64bit features a one big 64bit block, instead of having 32bits
> in one place and 32 in another.
> 2. Remove the reserved fields out of the config (the ones that were
> caused by moving the ISR and the notifications out).

Yes, those were exactly what I was thinking.  I left it vague because
there might be others you can see if we're prepared to abandon the
current format.

Cheers,
Rusty.

Re: [PATCHv2 RFC] virtio-spec: flexible configuration layout

[Michael S. Tsirkin]

On Wed, Nov 16, 2011 at 10:28:05AM +1030, Rusty Russell wrote:
> On Sun, 13 Nov 2011 17:14:27 +0200, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > On Fri, Nov 11, 2011 at 02:54:31PM +1030, Rusty Russell wrote:
> > > Indeed, I'd like to see two changes to your proposal:
> > > 
> > > (1) It should be all or nothing.  If a driver can find the virtio header
> > >     capability, it should only use the capabilties.  Otherwise, it
> > >     should fall back to legacy.
> > 
> > Okay, but going forward, if we add more capabilities, we probably won't
> > want to require them and fail to load if not there.  That's really why I
> > wanted to make the failover ignore any capability separately - to make
> > this future proof.  I'm not terribly fixated on this, it just seemed a
> > bit more symmetrical to treat all capabilities in the same way. Hmm?
> 
> Sure, a future capbility may not exist.  But once a driver finds that
> virtio header structure in the capability, it should *never* fall back
> to the legacy area.  ie. it can expect that Queue Notify, ISR Status and
> Device Header all exist.
> 
> ie. either use legacy mode, or use capabilities.  Never both.
> 
> > 
> > >     Your draft suggests a mix is possible;
> > >     I prefer a clean failure (ie. one day don't present a BAR 0 *at
> > >     all*, so ancient drivers just fail to load.).
> > 
> > Just to clarify, as written in draft this is possible with the current
> > spec proposal.  So I'm guessing there's some other motivation that you
> > had in mind?
> 
> At the moment you give a hybrid model where both are used.  In five
> years' time, that's going to be particularly ugly.
> > 
> > > (2) There's no huge win in keeping the same layout.  Let's make some
> > >     cleanups.
> > 
> > About this last point - what cleanups do you have in mind?  Just move
> > some registers around?  I guess we could put feature bits near each
> > other, and move device status towards the end to avoid wasting 3
> > bytes.
> 
> > The win seems minimal, but the change does make legacy hypervisor
> > support in guests more cumbersome, as we need to spread coditional code
> > around instead of localizing it in the initialization path.
> 
> But the separation between "legacy" and "modern" will be sharper, making
> it easier to excise the legacy portion later.
> 
> And in five years' time, people implementing virtio will really thank us
> that they can completely ignore the legacy header.

OK, I get it I think.

> > >    There are more users ahead of us then behind us (I
> > >     hope!).
> > 
> > In that case isn't it safe to assume we'll find some uses
> > for the reserved registers?
> 
> How would we tell?  If we use a new capability struct for it, it's
> obvious.  Otherwise, you're going to need to steal more feature bits.

Yes, exactly, if as you suggest, we disable legacy header
when there is a capability - we can use reserved registers
for other stuff.

> Cheers,
> Rusty.
> PS.  Sorry, was off sick for a few days.

Re: [PATCHv2 RFC] virtio-spec: flexible configuration layout

[Michael S. Tsirkin]

On Wed, Nov 16, 2011 at 10:28:52AM +1030, Rusty Russell wrote:
> On Fri, 11 Nov 2011 09:39:13 +0200, Sasha Levin <levinsasha928@gmail.com> wrote:
> > On Fri, Nov 11, 2011 at 6:24 AM, Rusty Russell <rusty@rustcorp.com.au> wrote:
> > > (2) There's no huge win in keeping the same layout.  Let's make some
> > >    cleanups.  There are more users ahead of us then behind us (I
> > >    hope!).
> > 
> > Actually, if we already do cleanups, here are two more suggestions:
> > 
> > 1. Make 64bit features a one big 64bit block, instead of having 32bits
> > in one place and 32 in another.
> > 2. Remove the reserved fields out of the config (the ones that were
> > caused by moving the ISR and the notifications out).
> 
> Yes, those were exactly what I was thinking.  I left it vague because
> there might be others you can see if we're prepared to abandon the
> current format.
> 
> Cheers,
> Rusty.

Yes but driver code doesn't get any cleaner by moving the fields.
And in fact, the legacy support makes the code messier.
What are the advantages?

-- 
MST

Re: [PATCHv2 RFC] virtio-spec: flexible configuration layout

[Sasha Levin]

On Wed, 2011-11-16 at 09:21 +0200, Michael S. Tsirkin wrote:
> On Wed, Nov 16, 2011 at 10:28:52AM +1030, Rusty Russell wrote:
> > On Fri, 11 Nov 2011 09:39:13 +0200, Sasha Levin <levinsasha928@gmail.com> wrote:
> > > On Fri, Nov 11, 2011 at 6:24 AM, Rusty Russell <rusty@rustcorp.com.au> wrote:
> > > > (2) There's no huge win in keeping the same layout.  Let's make some
> > > >    cleanups.  There are more users ahead of us then behind us (I
> > > >    hope!).
> > > 
> > > Actually, if we already do cleanups, here are two more suggestions:
> > > 
> > > 1. Make 64bit features a one big 64bit block, instead of having 32bits
> > > in one place and 32 in another.
> > > 2. Remove the reserved fields out of the config (the ones that were
> > > caused by moving the ISR and the notifications out).
> > 
> > Yes, those were exactly what I was thinking.  I left it vague because
> > there might be others you can see if we're prepared to abandon the
> > current format.
> > 
> > Cheers,
> > Rusty.
> 
> Yes but driver code doesn't get any cleaner by moving the fields.
> And in fact, the legacy support makes the code messier.
> What are the advantages?
> 

What about splitting the parts which handle legacy code and new code?
It'll make it easier playing with the new spec more freely and will also
make it easier removing legacy code in the future since you'll need to
simply delete a chunk of code instead of removing legacy bits out of
working code with a surgical knife.

-- 

Sasha.

Re: [PATCHv2 RFC] virtio-spec: flexible configuration layout

[Michael S. Tsirkin]

On Wed, Nov 16, 2011 at 10:17:39AM +0200, Sasha Levin wrote:
> On Wed, 2011-11-16 at 09:21 +0200, Michael S. Tsirkin wrote:
> > On Wed, Nov 16, 2011 at 10:28:52AM +1030, Rusty Russell wrote:
> > > On Fri, 11 Nov 2011 09:39:13 +0200, Sasha Levin <levinsasha928@gmail.com> wrote:
> > > > On Fri, Nov 11, 2011 at 6:24 AM, Rusty Russell <rusty@rustcorp.com.au> wrote:
> > > > > (2) There's no huge win in keeping the same layout.  Let's make some
> > > > >    cleanups.  There are more users ahead of us then behind us (I
> > > > >    hope!).
> > > > 
> > > > Actually, if we already do cleanups, here are two more suggestions:
> > > > 
> > > > 1. Make 64bit features a one big 64bit block, instead of having 32bits
> > > > in one place and 32 in another.
> > > > 2. Remove the reserved fields out of the config (the ones that were
> > > > caused by moving the ISR and the notifications out).
> > > 
> > > Yes, those were exactly what I was thinking.  I left it vague because
> > > there might be others you can see if we're prepared to abandon the
> > > current format.
> > > 
> > > Cheers,
> > > Rusty.
> > 
> > Yes but driver code doesn't get any cleaner by moving the fields.
> > And in fact, the legacy support makes the code messier.
> > What are the advantages?
> > 

The advantages question is what should really balance out the overhead.

> What about splitting the parts which handle legacy code and new code?

Well, I considered that. Something along the lines of
#define VIRTIO_NEW_MSI_CONFIG_VECTOR        18
And so on for all registers.

This seems to add a significant maintainance burden because of code
duplication. Note that, for example, vector programming is affected.
Multiply that by the number of guest OSes.


> It'll make it easier playing with the new spec more freely

I'm really worried about maintaing drivers long term.
Ease of experimentation is secondary for me.

> and will also
> make it easier removing legacy code in the future since you'll need to
> simply delete a chunk of code instead of removing legacy bits out of
> working code with a surgical knife.

It's unlikely to be a single chunk: we'd have structures and macros
which are separate. So at least 3 chunks.

Just for fun, here's what's involved in removing legacy map
support on top of my patch. As you see there are 4 chunks:
structure decl, map, unmap, and msix enable/disable.
And finding them was as simple as looking for legacy_map.


---

diff --git a/drivers/virtio/virtio_pci.c b/drivers/virtio/virtio_pci.c
index d242fcc..6c4d2faf 100644
--- a/drivers/virtio/virtio_pci.c
+++ b/drivers/virtio/virtio_pci.c
@@ -64,9 +64,6 @@ struct virtio_pci_device
 
 	/* Various IO mappings: used for resource tracking only. */
 
-	/* Legacy BAR0: typically PIO. */
-	void __iomem *legacy_map;
-
 	/* Mappings specified by device capabilities: typically in MMIO */
 	void __iomem *isr_map;
 	void __iomem *notify_map;
@@ -81,11 +78,7 @@ struct virtio_pci_device
 static void virtio_pci_set_msix_enabled(struct virtio_pci_device *vp_dev, int enabled)
 {
 	vp_dev->msix_enabled = enabled;
-	if (vp_dev->device_map)
-		vp_dev->ioaddr_device = vp_dev->device_map;
-	else
-		vp_dev->ioaddr_device = vp_dev->legacy_map +
-			VIRTIO_PCI_CONFIG(vp_dev);
+	vp_dev->ioaddr_device = vp_dev->device_map;
 }
 
 static void __iomem *virtio_pci_map_cfg(struct virtio_pci_device *vp_dev, u8 cap_id,
@@ -147,8 +140,6 @@ err:
 
 static void virtio_pci_iounmap(struct virtio_pci_device *vp_dev)
 {
-	if (vp_dev->legacy_map)
-		pci_iounmap(vp_dev->pci_dev, vp_dev->legacy_map);
 	if (vp_dev->isr_map)
 		pci_iounmap(vp_dev->pci_dev, vp_dev->isr_map);
 	if (vp_dev->notify_map)
@@ -176,36 +167,15 @@ static int virtio_pci_iomap(struct virtio_pci_device *vp_dev)
 
 	if (!vp_dev->notify_map || !vp_dev->common_map ||
 	    !vp_dev->device_map) {
-		/*
-		 * If not all capabilities present, map legacy PIO.
-		 * Legacy access is at BAR 0. We never need to map
-		 * more than 256 bytes there, since legacy config space
-		 * used PIO which has this size limit.
-		 * */
-		vp_dev->legacy_map = pci_iomap(vp_dev->pci_dev, 0, 256);
-		if (!vp_dev->legacy_map) {
-			dev_err(&vp_dev->vdev.dev, "Unable to map legacy PIO");
-			goto err;
-		}
+		dev_err(&vp_dev->vdev.dev, "Unable to map IO");
+		goto err;
 	}
 
-	/* Prefer MMIO if available. If not, fallback to legacy PIO. */
-	if (vp_dev->common_map)
-		vp_dev->ioaddr = vp_dev->common_map;
-	else
-		vp_dev->ioaddr = vp_dev->legacy_map;
+	vp_dev->ioaddr = vp_dev->common_map;
 
-	if (vp_dev->device_map)
-		vp_dev->ioaddr_device = vp_dev->device_map;
-	else
-		vp_dev->ioaddr_device = vp_dev->legacy_map +
-			VIRTIO_PCI_CONFIG(vp_dev);
+	vp_dev->ioaddr_device = vp_dev->device_map;
 
-	if (vp_dev->notify_map)
-		vp_dev->ioaddr_notify = vp_dev->notify_map;
-	else
-		vp_dev->ioaddr_notify = vp_dev->legacy_map +
-			VIRTIO_PCI_QUEUE_NOTIFY;
+	vp_dev->ioaddr_notify = vp_dev->notify_map;
 
 	return 0;
 err:

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Michael S. Tsirkin]

On Wed, Nov 09, 2011 at 02:36:43PM +0200, Pekka Enberg wrote:
> On Wed, 9 Nov 2011, Sasha Levin wrote:
> >They don't exist in kernel code either, for same reason as above.
> >
> >Nothing will break if we remove it since no one really used it, we were
> >probably the first and only implementation of the spec which considered
> >them :)
> 
> As long as we are able to run older versions of the KVM tool with
> newer kernels and vice versa, I see no reason why we can't drop
> 64-bit features from the KVM tool.
> 
> 			Pekka

So what happened? Did you guys do this?  Need to know what to do to make
progress.  IIUC Rusty removed the new fields in 0.9.3.
Does your tool still use them? Did any version of the tool released by
distros do so?

-- 
MST

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Sasha Levin]

On Mon, 2012-06-18 at 14:54 +0300, Michael S. Tsirkin wrote:
> On Wed, Nov 09, 2011 at 02:36:43PM +0200, Pekka Enberg wrote:
> > On Wed, 9 Nov 2011, Sasha Levin wrote:
> > >They don't exist in kernel code either, for same reason as above.
> > >
> > >Nothing will break if we remove it since no one really used it, we were
> > >probably the first and only implementation of the spec which considered
> > >them :)
> > 
> > As long as we are able to run older versions of the KVM tool with
> > newer kernels and vice versa, I see no reason why we can't drop
> > 64-bit features from the KVM tool.
> > 
> > 			Pekka
> 
> So what happened? Did you guys do this?  Need to know what to do to make
> progress.  IIUC Rusty removed the new fields in 0.9.3.
> Does your tool still use them? Did any version of the tool released by
> distros do so?

Yup, they were removed quite a while ago.

Re: [PATCH RFC] virtio-spec: flexible configuration layout

[Michael S. Tsirkin]

On Mon, Jun 18, 2012 at 02:05:17PM +0200, Sasha Levin wrote:
> On Mon, 2012-06-18 at 14:54 +0300, Michael S. Tsirkin wrote:
> > On Wed, Nov 09, 2011 at 02:36:43PM +0200, Pekka Enberg wrote:
> > > On Wed, 9 Nov 2011, Sasha Levin wrote:
> > > >They don't exist in kernel code either, for same reason as above.
> > > >
> > > >Nothing will break if we remove it since no one really used it, we were
> > > >probably the first and only implementation of the spec which considered
> > > >them :)
> > > 
> > > As long as we are able to run older versions of the KVM tool with
> > > newer kernels and vice versa, I see no reason why we can't drop
> > > 64-bit features from the KVM tool.
> > > 
> > > 			Pekka
> > 
> > So what happened? Did you guys do this?  Need to know what to do to make
> > progress.  IIUC Rusty removed the new fields in 0.9.3.
> > Does your tool still use them? Did any version of the tool released by
> > distros do so?
> 
> Yup, they were removed quite a while ago.

Thanks, that's nice. So we can reuse that bit if we need it
and it won't break your code.

-- 
MST