[PATCH v4 0/8] Add enlightenments for vGPU

[PATCH v4 0/8] Add enlightenments for vGPU

[Yu Zhang]

This patch set includes necessary code changes when i915 driver
runs inside a VM. Though ideally we can run an unmodified i915
driver in VM, adding such enlightenments can greatly reduce the
virtualization complexity in orders of magnitude. Code changes
for the host side, which includes the actual Intel GVT-g
implementation, will be sent out in other patches.

The primary change introduced here is to implement so-called
"address space ballooning" technique. XenGT partitions global
graphics memory among multiple VMs, so each VM can directly
access a portion of the memory without hypervisor's intervention,
e.g. filling textures or queuing commands. However with the
partitioning an unmodified i915 driver would assume a smaller
graphics memory starting from address ZERO, so requires XenGT
core module (vgt) to translate the graphics address between
'guest view' and 'host view', for all registers and command
opcodes which contain a graphics memory address. To reduce the
complexity, XenGT introduces "address space ballooning", by
telling the exact partitioning knowledge to each guest i915
driver, which then reserves and prevents non-allocated portions
from allocation. Then vgt module only needs to scan and validate
graphics addresses without complexity of translation.

Note: The partitioning of global graphics memory may break some
applications, with large objects in the aperture, because current
userspace assumes half of the aperture usable. That would need
separate fix either in user space (e.g. remove assumption in mesa)
or in kernel (with some faulting mechanism).

The partitioning knowledge is conveyed through a reserved MMIO
range, called PVINFO, which will be architecturally reserved in
future hardware generations. Another information carried through
PVINFO is about the number of fence registers. As a global resource,
XenGT also partitions them among VMs.

Other changes are trivial as optimizations, to either reduce the
trap overhead or disable power management features which don't
make sense in a virtualized environment.


Yu Zhang (8):
  drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
  drm/i915: Adds graphic address space ballooning logic
  drm/i915: Partition the fence registers for vGPU in i915 driver
  drm/i915: Disable framebuffer compression for i915 driver in VM
  drm/i915: Add the display switch logic for vGPU in i915 driver
  drm/i915: Disable power management for i915 driver in VM
  drm/i915: Create vGPU specific MMIO operations to reduce traps
  drm/i915: Support alias ppgtt in VM if ppgtt is enabled

 Documentation/DocBook/drm.tmpl      |   5 +
 drivers/gpu/drm/i915/Makefile       |   3 +
 drivers/gpu/drm/i915/i915_dma.c     |   8 ++
 drivers/gpu/drm/i915/i915_drv.h     |  10 ++
 drivers/gpu/drm/i915/i915_gem.c     |   5 +
 drivers/gpu/drm/i915/i915_gem_gtt.c |  33 ++++-
 drivers/gpu/drm/i915/i915_vgpu.c    | 264 ++++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_vgpu.h    |  91 +++++++++++++
 drivers/gpu/drm/i915/intel_fbc.c    |   4 +
 drivers/gpu/drm/i915/intel_pm.c     |   4 +
 drivers/gpu/drm/i915/intel_uncore.c |  34 +++++
 11 files changed, 458 insertions(+), 3 deletions(-)
 create mode 100644 drivers/gpu/drm/i915/i915_vgpu.c
 create mode 100644 drivers/gpu/drm/i915/i915_vgpu.h

-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.

[Yu Zhang]

Introduce a PV INFO structure, to facilitate the Intel GVT-g
technology, which is a GPU virtualization solution with mediated
pass-through. This page contains the shared information between
i915 driver and the host emulator. For now, this structure utilizes
an area of 4K bytes on HSW GPU's unused MMIO space. Future hardware
will have the reserved window architecturally defined, and layout
of the page will be added in future BSpec.

The i915 driver load routine detects if it is running in a VM by
reading the contents of this PV INFO page. Thereafter a flag,
vgpu.active is set, and intel_vgpu_active() is used by checking
this flag to conclude if GPU is virtualized with Intel GVT-g. By
now, intel_vgpu_active() will return true, only when the driver
is running as a guest in the Intel GVT-g enhanced environment on
HSW platform.

v2:
take Chris' comments:
        - call the i915_check_vgpu() in intel_uncore_init()
        - sanitize i915_check_vgpu() by adding BUILD_BUG_ON() and debug info
take Daniel's comments:
        - put the definition of PV INFO into a new header - i915_vgt_if.h
other changes:
        - access mmio regs by readq/readw in i915_check_vgpu()

v3:
take Daniel's comments:
        - move the i915/vgt interfaces into a new i915_vgpu.c
        - update makefile
        - add kerneldoc to functions which are non-static
        - add a DOC: section describing some of the high-level design
        - update drm docbook
other changes:
        - rename i915_vgt_if.h to i915_vgpu.h

v4:
take Tvrtko's comments:
        - fix a typo in commit message
        - add debug message when vgt version mismatches
        - rename low_gmadr/high_gmadr to mappable/non-mappable in PV INFO
          structure

Signed-off-by: Yu Zhang <yu.c.zhang@linux.intel.com>
Signed-off-by: Jike Song <jike.song@intel.com>
Signed-off-by: Eddie Dong <eddie.dong@intel.com>
---
 Documentation/DocBook/drm.tmpl      |  5 +++
 drivers/gpu/drm/i915/Makefile       |  3 ++
 drivers/gpu/drm/i915/i915_drv.h     | 10 +++++
 drivers/gpu/drm/i915/i915_vgpu.c    | 86 +++++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_vgpu.h    | 85 ++++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/intel_uncore.c |  3 ++
 6 files changed, 192 insertions(+)
 create mode 100644 drivers/gpu/drm/i915/i915_vgpu.c
 create mode 100644 drivers/gpu/drm/i915/i915_vgpu.h

diff --git a/Documentation/DocBook/drm.tmpl b/Documentation/DocBook/drm.tmpl
index 03f1985..249f0c9 100644
--- a/Documentation/DocBook/drm.tmpl
+++ b/Documentation/DocBook/drm.tmpl
@@ -3979,6 +3979,11 @@ int num_ioctls;</synopsis>
 !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_disable_interrupts
 !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_enable_interrupts
       </sect2>
+      <sect2>
+        <title>Intel GVT-g Guest Support(vGPU)</title>
+!Pdrivers/gpu/drm/i915/i915_vgpu.c Intel GVT-g guest support
+!Idrivers/gpu/drm/i915/i915_vgpu.c
+      </sect2>
     </sect1>
     <sect1>
       <title>Display Hardware Handling</title>
diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile
index f019225..f025e7f 100644
--- a/drivers/gpu/drm/i915/Makefile
+++ b/drivers/gpu/drm/i915/Makefile
@@ -83,6 +83,9 @@ i915-y += dvo_ch7017.o \
 	  intel_sdvo.o \
 	  intel_tv.o
 
+# virtual gpu code
+i915-y += i915_vgpu.o
+
 # legacy horrors
 i915-y += i915_dma.o \
 	  i915_ums.o
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index c0b8644..9bc0c9e 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1640,6 +1640,10 @@ struct i915_workarounds {
 	u32 count;
 };
 
+struct i915_virtual_gpu {
+	bool active;
+};
+
 struct drm_i915_private {
 	struct drm_device *dev;
 	struct kmem_cache *slab;
@@ -1652,6 +1656,8 @@ struct drm_i915_private {
 
 	struct intel_uncore uncore;
 
+	struct i915_virtual_gpu vgpu;
+
 	struct intel_gmbus gmbus[GMBUS_NUM_PORTS];
 
 
@@ -2583,6 +2589,10 @@ void intel_uncore_forcewake_get(struct drm_i915_private *dev_priv,
 void intel_uncore_forcewake_put(struct drm_i915_private *dev_priv,
 				enum forcewake_domains domains);
 void assert_forcewakes_inactive(struct drm_i915_private *dev_priv);
+static inline bool intel_vgpu_active(struct drm_device *dev)
+{
+	return to_i915(dev)->vgpu.active;
+}
 
 void
 i915_enable_pipestat(struct drm_i915_private *dev_priv, enum pipe pipe,
diff --git a/drivers/gpu/drm/i915/i915_vgpu.c b/drivers/gpu/drm/i915/i915_vgpu.c
new file mode 100644
index 0000000..995a600
--- /dev/null
+++ b/drivers/gpu/drm/i915/i915_vgpu.c
@@ -0,0 +1,86 @@
+/*
+ * Copyright(c) 2011-2015 Intel Corporation. All rights reserved.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#include "intel_drv.h"
+#include "i915_vgpu.h"
+
+/**
+ * DOC: Intel GVT-g guest support
+ *
+ * Intel GVT-g is a graphics virtualization technology which shares the
+ * GPU among multiple virtual machines on a time-sharing basis. Each
+ * virtual machine is presented a virtual GPU (vGPU), which has equivalent
+ * features as the underlying physical GPU (pGPU), so i915 driver can run
+ * seamlessly in a virtual machine. This file provides vGPU specific
+ * optimizations when running in a virtual machine, to reduce the complexity
+ * of vGPU emulation and to improve the overall performance.
+ *
+ * A primary function introduced here is so-called "address space ballooning"
+ * technique. Intel GVT-g partitions global graphics memory among multiple VMs,
+ * so each VM can directly access a portion of the memory without hypervisor's
+ * intervention, e.g. filling textures or queuing commands. However with the
+ * partitioning an unmodified i915 driver would assume a smaller graphics
+ * memory starting from address ZERO, then requires vGPU emulation module to
+ * translate the graphics address between 'guest view' and 'host view', for
+ * all registers and command opcodes which contain a graphics memory address.
+ * To reduce the complexity, Intel GVT-g introduces "address space ballooning",
+ * by telling the exact partitioning knowledge to each guest i915 driver, which
+ * then reserves and prevents non-allocated portions from allocation. Thus vGPU
+ * emulation module only needs to scan and validate graphics addresses without
+ * complexity of address translation.
+ *
+ */
+
+/**
+ * i915_check_vgpu - detect virtual GPU
+ * @dev: drm device *
+ *
+ * This function is called at the initialization stage, to detect whether
+ * running on a vGPU.
+ */
+void i915_check_vgpu(struct drm_device *dev)
+{
+	struct drm_i915_private *dev_priv = to_i915(dev);
+	uint64_t magic;
+	uint32_t version;
+
+	BUILD_BUG_ON(sizeof(struct vgt_if) != VGT_PVINFO_SIZE);
+
+	if (!IS_HASWELL(dev))
+		return;
+
+	magic = readq(dev_priv->regs + vgtif_reg(magic));
+	if (magic != VGT_MAGIC)
+		return;
+
+	version = INTEL_VGT_IF_VERSION_ENCODE(
+		readw(dev_priv->regs + vgtif_reg(version_major)),
+		readw(dev_priv->regs + vgtif_reg(version_minor)));
+	if (version != INTEL_VGT_IF_VERSION) {
+		DRM_INFO("VGT interface version mismatch!\n");
+		return;
+	}
+
+	dev_priv->vgpu.active = true;
+	DRM_INFO("Virtual GPU for Intel GVT-g detected.\n");
+}
diff --git a/drivers/gpu/drm/i915/i915_vgpu.h b/drivers/gpu/drm/i915/i915_vgpu.h
new file mode 100644
index 0000000..aa72e14
--- /dev/null
+++ b/drivers/gpu/drm/i915/i915_vgpu.h
@@ -0,0 +1,85 @@
+/*
+ * Copyright(c) 2011-2015 Intel Corporation. All rights reserved.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#ifndef _I915_VGPU_H_
+#define _I915_VGPU_H_
+
+/* The MMIO offset of the shared info between guest and host emulator */
+#define VGT_PVINFO_PAGE	0x78000
+#define VGT_PVINFO_SIZE	0x1000
+
+/*
+ * The following structure pages are defined in GEN MMIO space
+ * for virtualization. (One page for now)
+ */
+#define VGT_MAGIC         0x4776544776544776	/* 'vGTvGTvG' */
+#define VGT_VERSION_MAJOR 1
+#define VGT_VERSION_MINOR 0
+
+#define INTEL_VGT_IF_VERSION_ENCODE(major, minor) ((major) << 16 | (minor))
+#define INTEL_VGT_IF_VERSION \
+	INTEL_VGT_IF_VERSION_ENCODE(VGT_VERSION_MAJOR, VGT_VERSION_MINOR)
+
+struct vgt_if {
+	uint64_t magic;		/* VGT_MAGIC */
+	uint16_t version_major;
+	uint16_t version_minor;
+	uint32_t vgt_id;	/* ID of vGT instance */
+	uint32_t rsv1[12];	/* pad to offset 0x40 */
+	/*
+	 *  Data structure to describe the balooning info of resources.
+	 *  Each VM can only have one portion of continuous area for now.
+	 *  (May support scattered resource in future)
+	 *  (starting from offset 0x40)
+	 */
+	struct {
+		/* Aperture register balooning */
+		struct {
+			uint32_t base;
+			uint32_t size;
+		} mappable_gmadr;	/* aperture */
+		/* GMADR register balooning */
+		struct {
+			uint32_t base;
+			uint32_t size;
+		} nonmappable_gmadr;	/* non aperture */
+		/* allowed fence registers */
+		uint32_t fence_num;
+		uint32_t rsv2[3];
+	} avail_rs;		/* available/assigned resource */
+	uint32_t rsv3[0x200 - 24];	/* pad to half page */
+	/*
+	 * The bottom half page is for response from Gfx driver to hypervisor.
+	 * Set to reserved fields temporarily by now.
+	 */
+	uint32_t rsv4;
+	uint32_t display_ready;	/* ready for display owner switch */
+	uint32_t rsv5[0x200 - 2];	/* pad to one page */
+} __packed;
+
+#define vgtif_reg(x) \
+	(VGT_PVINFO_PAGE + (long)&((struct vgt_if *)NULL)->x)
+
+extern void i915_check_vgpu(struct drm_device *dev);
+
+#endif /* _I915_VGPU_H_ */
diff --git a/drivers/gpu/drm/i915/intel_uncore.c b/drivers/gpu/drm/i915/intel_uncore.c
index c47a3ba..d95419d 100644
--- a/drivers/gpu/drm/i915/intel_uncore.c
+++ b/drivers/gpu/drm/i915/intel_uncore.c
@@ -23,6 +23,7 @@
 
 #include "i915_drv.h"
 #include "intel_drv.h"
+#include "i915_vgpu.h"
 
 #include <linux/pm_runtime.h>
 
@@ -1082,6 +1083,8 @@ void intel_uncore_init(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
 
+	i915_check_vgpu(dev);
+
 	intel_uncore_ellc_detect(dev);
 	intel_uncore_fw_domains_init(dev);
 	__intel_uncore_early_sanitize(dev, false);
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 2/8] drm/i915: Adds graphic address space ballooning logic

[Yu Zhang]

With Intel GVT-g, the global graphic memory space is partitioned by
multiple vGPU instances in different VMs. The ballooning code is called
in i915_gem_setup_global_gtt(), utilizing the drm mm allocator APIs to
mark the graphic address space which are partitioned out to other vGPUs
as reserved. With ballooning, host side does not need to translate a
grahpic address from guest view to host view. By now, current implementation
only support the static ballooning, but in the future, with more cooperation
from guest driver, the same interfaces can be extended to grow/shrink the
guest graphic memory dynamically.

v2:
take Chris and Daniel's comments:
	- no guard page between different VMs
	- use drm_mm_reserve_node() to do the reservation for ballooning,
	instead of the previous drm_mm_insert_node_in_range_generic()

v3:
take Daniel's comments:
	- move ballooning functions into i915_vgpu.c
	- add kerneldoc to ballooning functions

v4:
take Tvrtko's comments:
	- more accurate comments and commit message

Signed-off-by: Yu Zhang <yu.c.zhang@linux.intel.com>
Signed-off-by: Jike Song <jike.song@intel.com>
Signed-off-by: Zhi Wang <zhi.a.wang@intel.com>
Signed-off-by: Eddie Dong <eddie.dong@intel.com>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c |  17 +++-
 drivers/gpu/drm/i915/i915_vgpu.c    | 178 ++++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_vgpu.h    |   2 +
 3 files changed, 194 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index b48b586..057f905 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -27,6 +27,7 @@
 #include <drm/drmP.h>
 #include <drm/i915_drm.h>
 #include "i915_drv.h"
+#include "i915_vgpu.h"
 #include "i915_trace.h"
 #include "intel_drv.h"
 
@@ -1756,6 +1757,16 @@ static int i915_gem_setup_global_gtt(struct drm_device *dev,
 
 	/* Subtract the guard page ... */
 	drm_mm_init(&ggtt_vm->mm, start, end - start - PAGE_SIZE);
+
+	dev_priv->gtt.base.start = start;
+	dev_priv->gtt.base.total = end - start;
+
+	if (intel_vgpu_active(dev)) {
+		ret = intel_vgt_balloon(dev);
+		if (ret)
+			return ret;
+	}
+
 	if (!HAS_LLC(dev))
 		dev_priv->gtt.base.mm.color_adjust = i915_gtt_color_adjust;
 
@@ -1775,9 +1786,6 @@ static int i915_gem_setup_global_gtt(struct drm_device *dev,
 		vma->bound |= GLOBAL_BIND;
 	}
 
-	dev_priv->gtt.base.start = start;
-	dev_priv->gtt.base.total = end - start;
-
 	/* Clear any non-preallocated blocks */
 	drm_mm_for_each_hole(entry, &ggtt_vm->mm, hole_start, hole_end) {
 		DRM_DEBUG_KMS("clearing unused GTT space: [%lx, %lx]\n",
@@ -1829,6 +1837,9 @@ void i915_global_gtt_cleanup(struct drm_device *dev)
 	}
 
 	if (drm_mm_initialized(&vm->mm)) {
+		if (intel_vgpu_active(dev))
+			intel_vgt_deballoon();
+
 		drm_mm_takedown(&vm->mm);
 		list_del(&vm->global_link);
 	}
diff --git a/drivers/gpu/drm/i915/i915_vgpu.c b/drivers/gpu/drm/i915/i915_vgpu.c
index 995a600..5eee75b 100644
--- a/drivers/gpu/drm/i915/i915_vgpu.c
+++ b/drivers/gpu/drm/i915/i915_vgpu.c
@@ -84,3 +84,181 @@ void i915_check_vgpu(struct drm_device *dev)
 	dev_priv->vgpu.active = true;
 	DRM_INFO("Virtual GPU for Intel GVT-g detected.\n");
 }
+
+struct _balloon_info_ {
+	/*
+	 * There are up to 2 regions per mappable/unmappable graphic
+	 * memory that might be ballooned. Here, index 0/1 is for mappable
+	 * graphic memory, 2/3 for unmappable graphic memory.
+	 */
+	struct drm_mm_node space[4];
+};
+
+static struct _balloon_info_ bl_info;
+
+/**
+ * intel_vgt_deballoon - deballoon reserved graphics address trunks
+ *
+ * This function is called to deallocate the ballooned-out graphic memory, when
+ * driver is unloaded or when ballooning fails.
+ */
+void intel_vgt_deballoon(void)
+{
+	int i;
+
+	DRM_DEBUG("VGT deballoon.\n");
+
+	for (i = 0; i < 4; i++) {
+		if (bl_info.space[i].allocated)
+			drm_mm_remove_node(&bl_info.space[i]);
+	}
+
+	memset(&bl_info, 0, sizeof(bl_info));
+}
+
+static int vgt_balloon_space(struct drm_mm *mm,
+			     struct drm_mm_node *node,
+			     unsigned long start, unsigned long end)
+{
+	unsigned long size = end - start;
+
+	if (start == end)
+		return -EINVAL;
+
+	DRM_INFO("balloon space: range [ 0x%lx - 0x%lx ] %lu KiB.\n",
+		 start, end, size / 1024);
+
+	node->start = start;
+	node->size = size;
+
+	return drm_mm_reserve_node(mm, node);
+}
+
+/**
+ * intel_vgt_balloon - balloon out reserved graphics address trunks
+ * @dev: drm device
+ *
+ * This function is called at the initialization stage, to balloon out the
+ * graphic address space allocated to other vGPUs, by marking these spaces as
+ * reserved. The ballooning related knowledge(starting address and size of
+ * the mappable/unmappable graphic memory) is described in the vgt_if structure
+ * in a reserved mmio range.
+ *
+ * To give an example, the drawing below depicts one typical scenario after
+ * ballooning. Here the vGPU1 has 2 pieces of graphic address spaces ballooned
+ * out each for the mappable and the non-mappable part. From the vGPU1 point of
+ * view, the total size is the same as the physical one, with the start address
+ * of its graphic space being zero. Yet there are some portions ballooned out(
+ * the shadow part, which are marked as reserved by drm allocator). From the
+ * host point of view, the graphic address space is partitioned by multiple
+ * vGPUs in different VMs.
+ *
+ *                        vGPU1 view         Host view
+ *             0 ------> +-----------+     +-----------+
+ *               ^       |///////////|     |   vGPU3   |
+ *               |       |///////////|     +-----------+
+ *               |       |///////////|     |   vGPU2   |
+ *               |       +-----------+     +-----------+
+ *        mappable GM    | available | ==> |   vGPU1   |
+ *               |       +-----------+     +-----------+
+ *               |       |///////////|     |           |
+ *               v       |///////////|     |   Host    |
+ *               +=======+===========+     +===========+
+ *               ^       |///////////|     |   vGPU3   |
+ *               |       |///////////|     +-----------+
+ *               |       |///////////|     |   vGPU2   |
+ *               |       +-----------+     +-----------+
+ *      unmappable GM    | available | ==> |   vGPU1   |
+ *               |       +-----------+     +-----------+
+ *               |       |///////////|     |           |
+ *               |       |///////////|     |   Host    |
+ *               v       |///////////|     |           |
+ * total GM size ------> +-----------+     +-----------+
+ *
+ * Returns:
+ * zero on success, non-zero if configuration invalid or ballooning failed
+ */
+int intel_vgt_balloon(struct drm_device *dev)
+{
+	struct drm_i915_private *dev_priv = to_i915(dev);
+	struct i915_address_space *ggtt_vm = &dev_priv->gtt.base;
+	unsigned long ggtt_vm_end = ggtt_vm->start + ggtt_vm->total;
+
+	unsigned long mappable_base, mappable_size, mappable_end;
+	unsigned long unmappable_base, unmappable_size, unmappable_end;
+	int ret;
+
+	mappable_base = I915_READ(vgtif_reg(avail_rs.mappable_gmadr.base));
+	mappable_size = I915_READ(vgtif_reg(avail_rs.mappable_gmadr.size));
+	unmappable_base = I915_READ(vgtif_reg(avail_rs.nonmappable_gmadr.base));
+	unmappable_size = I915_READ(vgtif_reg(avail_rs.nonmappable_gmadr.size));
+
+	mappable_end = mappable_base + mappable_size;
+	unmappable_end = unmappable_base + unmappable_size;
+
+	DRM_INFO("VGT ballooning configuration:\n");
+	DRM_INFO("Mappable graphic memory: base 0x%lx size %ldKiB\n",
+		 mappable_base, mappable_size / 1024);
+	DRM_INFO("Unmappable graphic memory: base 0x%lx size %ldKiB\n",
+		 unmappable_base, unmappable_size / 1024);
+
+	if (mappable_base < ggtt_vm->start ||
+	    mappable_end > dev_priv->gtt.mappable_end ||
+	    unmappable_base < dev_priv->gtt.mappable_end ||
+	    unmappable_end > ggtt_vm_end) {
+		DRM_ERROR("Invalid ballooning configuration!\n");
+		return -EINVAL;
+	}
+
+	/* Unmappable graphic memory ballooning */
+	if (unmappable_base > dev_priv->gtt.mappable_end) {
+		ret = vgt_balloon_space(&ggtt_vm->mm,
+					&bl_info.space[2],
+					dev_priv->gtt.mappable_end,
+					unmappable_base);
+
+		if (ret)
+			goto err;
+	}
+
+	/*
+	 * No need to partition out the last physical page,
+	 * because it is reserved to the guard page.
+	 */
+	if (unmappable_end < ggtt_vm_end - PAGE_SIZE) {
+		ret = vgt_balloon_space(&ggtt_vm->mm,
+					&bl_info.space[3],
+					unmappable_end,
+					ggtt_vm_end - PAGE_SIZE);
+		if (ret)
+			goto err;
+	}
+
+	/* Mappable graphic memory ballooning */
+	if (mappable_base > ggtt_vm->start) {
+		ret = vgt_balloon_space(&ggtt_vm->mm,
+					&bl_info.space[0],
+					ggtt_vm->start, mappable_base);
+
+		if (ret)
+			goto err;
+	}
+
+	if (mappable_end < dev_priv->gtt.mappable_end) {
+		ret = vgt_balloon_space(&ggtt_vm->mm,
+					&bl_info.space[1],
+					mappable_end,
+					dev_priv->gtt.mappable_end);
+
+		if (ret)
+			goto err;
+	}
+
+	DRM_INFO("VGT balloon successfully\n");
+	return 0;
+
+err:
+	DRM_ERROR("VGT balloon fail\n");
+	intel_vgt_deballoon();
+	return ret;
+}
diff --git a/drivers/gpu/drm/i915/i915_vgpu.h b/drivers/gpu/drm/i915/i915_vgpu.h
index aa72e14..3ed01a7 100644
--- a/drivers/gpu/drm/i915/i915_vgpu.h
+++ b/drivers/gpu/drm/i915/i915_vgpu.h
@@ -81,5 +81,7 @@ struct vgt_if {
 	(VGT_PVINFO_PAGE + (long)&((struct vgt_if *)NULL)->x)
 
 extern void i915_check_vgpu(struct drm_device *dev);
+extern int intel_vgt_balloon(struct drm_device *dev);
+extern void intel_vgt_deballoon(void);
 
 #endif /* _I915_VGPU_H_ */
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver

[Yu Zhang]

With Intel GVT-g, the fence registers are partitioned by multiple
vGPU instances in different VMs. Routine i915_gem_load() is modified
to reset the num_fence_regs, when the driver detects it's running in
a VM. Accesses to the fence registers from vGPU will be trapped and
remapped by the host side. And the allocated fence number is provided
in PV INFO page structure. By now, the value of fence number is fixed,
but in the future we can relax this limitation, to allocate the fence
registers dynamically from host side.

Signed-off-by: Yu Zhang <yu.c.zhang@linux.intel.com>
Signed-off-by: Jike Song <jike.song@intel.com>
Signed-off-by: Eddie Dong <eddie.dong@intel.com>
---
 drivers/gpu/drm/i915/i915_gem.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index c26d36c..1765989 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -29,6 +29,7 @@
 #include <drm/drm_vma_manager.h>
 #include <drm/i915_drm.h>
 #include "i915_drv.h"
+#include "i915_vgpu.h"
 #include "i915_trace.h"
 #include "intel_drv.h"
 #include <linux/oom.h>
@@ -4987,6 +4988,10 @@ i915_gem_load(struct drm_device *dev)
 	else
 		dev_priv->num_fence_regs = 8;
 
+	if (intel_vgpu_active(dev))
+		dev_priv->num_fence_regs =
+				I915_READ(vgtif_reg(avail_rs.fence_num));
+
 	/* Initialize fence registers to zero */
 	INIT_LIST_HEAD(&dev_priv->mm.fence_list);
 	i915_gem_restore_fences(dev);
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 4/8] drm/i915: Disable framebuffer compression for i915 driver in VM

[Yu Zhang]

Framebuffer compression is disabled when driver detects it's
running in a Intel GVT-g enlightened VM, because FBC is not
emulated and there is no stolen memory for a vGPU.

v2:
take Chris' comments:
        - move the code into intel_update_fbc()

v4:
take Tvrtko's comments:
        - rebase the code into intel_fbc_update()

Signed-off-by: Yu Zhang <yu.c.zhang@linux.intel.com>
Signed-off-by: Jike Song <jike.song@intel.com>
Signed-off-by: Zhiyuan Lv <zhiyuan.lv@intel.com>
---
 drivers/gpu/drm/i915/intel_fbc.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/gpu/drm/i915/intel_fbc.c b/drivers/gpu/drm/i915/intel_fbc.c
index 7341e87..ee65731 100644
--- a/drivers/gpu/drm/i915/intel_fbc.c
+++ b/drivers/gpu/drm/i915/intel_fbc.c
@@ -505,6 +505,10 @@ void intel_fbc_update(struct drm_device *dev)
 	if (!HAS_FBC(dev))
 		return;
 
+	/* disable framebuffer compression in vGPU */
+	if (intel_vgpu_active(dev))
+		i915.enable_fbc = 0;
+
 	if (i915.enable_fbc < 0) {
 		if (set_no_fbc_reason(dev_priv, FBC_CHIP_DEFAULT))
 			DRM_DEBUG_KMS("disabled per chip default\n");
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 5/8] drm/i915: Add the display switch logic for vGPU in i915 driver

[Yu Zhang]

Display switch logic is added to notify the host side that
current vGPU have a valid surface to show. It does so by
writing the display_ready field in PV INFO page, and then
will be handled in the host side. This is useful to avoid
trickiness when the VM's framebuffer is being accessed in
the middle of VM modesetting, e.g. compositing the framebuffer
in the host side.

v2:
        - move the notification code outside the 'else' in load sequence
        - remove the notification code in intel_crtc_set_config()

v4:
        - code rebase, no need to define another dev_priv
        - use #define instead of enum for display readiness

Signed-off-by: Yu Zhang <yu.c.zhang@linux.intel.com>
Signed-off-by: Jike Song <jike.song@intel.com>
Signed-off-by: Zhiyuan Lv <zhiyuan.lv@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/gpu/drm/i915/i915_dma.c  | 8 ++++++++
 drivers/gpu/drm/i915/i915_vgpu.h | 4 ++++
 2 files changed, 12 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_dma.c b/drivers/gpu/drm/i915/i915_dma.c
index 1a46787..5804aa5 100644
--- a/drivers/gpu/drm/i915/i915_dma.c
+++ b/drivers/gpu/drm/i915/i915_dma.c
@@ -36,6 +36,7 @@
 #include "intel_drv.h"
 #include <drm/i915_drm.h>
 #include "i915_drv.h"
+#include "i915_vgpu.h"
 #include "i915_trace.h"
 #include <linux/pci.h>
 #include <linux/console.h>
@@ -842,6 +843,13 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags)
 		}
 	}
 
+	/*
+	 * Notify a valid surface after modesetting,
+	 * when running inside a VM.
+	 */
+	if (intel_vgpu_active(dev))
+		I915_WRITE(vgtif_reg(display_ready), VGT_DRV_DISPLAY_READY);
+
 	i915_setup_sysfs(dev);
 
 	if (INTEL_INFO(dev)->num_pipes) {
diff --git a/drivers/gpu/drm/i915/i915_vgpu.h b/drivers/gpu/drm/i915/i915_vgpu.h
index 3ed01a7..0db9ccf 100644
--- a/drivers/gpu/drm/i915/i915_vgpu.h
+++ b/drivers/gpu/drm/i915/i915_vgpu.h
@@ -80,6 +80,10 @@ struct vgt_if {
 #define vgtif_reg(x) \
 	(VGT_PVINFO_PAGE + (long)&((struct vgt_if *)NULL)->x)
 
+/* vGPU display status to be used by the host side */
+#define VGT_DRV_DISPLAY_NOT_READY 0
+#define VGT_DRV_DISPLAY_READY     1  /* ready for display switch */
+
 extern void i915_check_vgpu(struct drm_device *dev);
 extern int intel_vgt_balloon(struct drm_device *dev);
 extern void intel_vgt_deballoon(void);
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 6/8] drm/i915: Disable power management for i915 driver in VM

[Yu Zhang]

With Intel GVT-g, GPU power management is controlled by
host driver, so there is no need to provide virtualized
GPU PM support. In the future it might be useful to gather
VM input for freq boost, but now let's disable it simply.

v2:
take Chris' comments:
        - do not special case this to gen6+

Signed-off-by: Yu Zhang <yu.c.zhang@linux.intel.com>
Signed-off-by: Jike Song <jike.song@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/gpu/drm/i915/intel_pm.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/gpu/drm/i915/intel_pm.c b/drivers/gpu/drm/i915/intel_pm.c
index 3c64810..177ff97 100644
--- a/drivers/gpu/drm/i915/intel_pm.c
+++ b/drivers/gpu/drm/i915/intel_pm.c
@@ -5623,6 +5623,10 @@ void intel_enable_gt_powersave(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
 
+	/* Powersaving is controlled by the host when inside a VM */
+	if (intel_vgpu_active(dev))
+		return;
+
 	if (IS_IRONLAKE_M(dev)) {
 		mutex_lock(&dev->struct_mutex);
 		ironlake_enable_drps(dev);
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 7/8] drm/i915: Create vGPU specific MMIO operations to reduce traps

[Yu Zhang]

In the virtualized environment, forcewake operations are not
necessary for the driver, because mmio accesses will be trapped
and emulated by the host side, and real forcewake operations are
also done in the host. New mmio access handlers are added to directly
call the __raw_i915_read/write, therefore will reduce many traps and
increase the overall performance for drivers running in the VM with
Intel GVT-g enhancement.

v2:
take Chris' comments:
        - register the mmio hooks in intel_uncore_init()
v3:
take Daniel's comments:
        - use macros to assign mmio write functions for vGPU
v4:
take Tvrtko's comments:
        - also use mmio hooks for read operations

Signed-off-by: Yu Zhang <yu.c.zhang@linux.intel.com>
Signed-off-by: Jike Song <jike.song@intel.com>
Signed-off-by: Kevin Tian <kevin.tian@intel.com>k
---
 drivers/gpu/drm/i915/intel_uncore.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/drivers/gpu/drm/i915/intel_uncore.c b/drivers/gpu/drm/i915/intel_uncore.c
index d95419d..1b8d8ce 100644
--- a/drivers/gpu/drm/i915/intel_uncore.c
+++ b/drivers/gpu/drm/i915/intel_uncore.c
@@ -641,6 +641,14 @@ static inline void __force_wake_get(struct drm_i915_private *dev_priv,
 		dev_priv->uncore.funcs.force_wake_get(dev_priv, fw_domains);
 }
 
+#define __vgpu_read(x) \
+static u##x \
+vgpu_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
+	GEN6_READ_HEADER(x); \
+	val = __raw_i915_read##x(dev_priv, reg); \
+	GEN6_READ_FOOTER; \
+}
+
 #define __gen6_read(x) \
 static u##x \
 gen6_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
@@ -704,6 +712,10 @@ gen9_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
 	GEN6_READ_FOOTER; \
 }
 
+__vgpu_read(8)
+__vgpu_read(16)
+__vgpu_read(32)
+__vgpu_read(64)
 __gen9_read(8)
 __gen9_read(16)
 __gen9_read(32)
@@ -725,6 +737,7 @@ __gen6_read(64)
 #undef __chv_read
 #undef __vlv_read
 #undef __gen6_read
+#undef __vgpu_read
 #undef GEN6_READ_FOOTER
 #undef GEN6_READ_HEADER
 
@@ -808,6 +821,14 @@ hsw_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace)
 	GEN6_WRITE_FOOTER; \
 }
 
+#define __vgpu_write(x) \
+static void vgpu_write##x(struct drm_i915_private *dev_priv, \
+			  off_t reg, u##x val, bool trace) { \
+	GEN6_WRITE_HEADER; \
+	__raw_i915_write##x(dev_priv, reg, val); \
+	GEN6_WRITE_FOOTER; \
+}
+
 static const u32 gen8_shadowed_regs[] = {
 	FORCEWAKE_MT,
 	GEN6_RPNSWREQ,
@@ -925,12 +946,17 @@ __gen6_write(8)
 __gen6_write(16)
 __gen6_write(32)
 __gen6_write(64)
+__vgpu_write(8)
+__vgpu_write(16)
+__vgpu_write(32)
+__vgpu_write(64)
 
 #undef __gen9_write
 #undef __chv_write
 #undef __gen8_write
 #undef __hsw_write
 #undef __gen6_write
+#undef __vgpu_write
 #undef GEN6_WRITE_FOOTER
 #undef GEN6_WRITE_HEADER
 
@@ -1133,6 +1159,11 @@ void intel_uncore_init(struct drm_device *dev)
 		break;
 	}
 
+	if (intel_vgpu_active(dev)) {
+		ASSIGN_WRITE_MMIO_VFUNCS(vgpu);
+		ASSIGN_READ_MMIO_VFUNCS(vgpu);
+	}
+
 	i915_check_and_clear_faults(dev);
 }
 #undef ASSIGN_WRITE_MMIO_VFUNCS
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH v4 8/8] drm/i915: Support alias ppgtt in VM if ppgtt is enabled

[Yu Zhang]

The current Intel GVT-g only supports alias ppgtt. And the
emulation is done in the host by first trapping PP_DIR_BASE
mmio accesses. Updating PP_DIR_BASE by using instructions such
as MI_LOAD_REGISTER_IMM are hard to detect and are not supported
in current code. Therefore this patch also adds a new callback
routine - vgpu_mm_switch() to set the PP_DIR_BASE by mmio writes.

v2:
take Chris' comments:
        - move the code into sanitize_enable_ppgtt()
v4:
take Tvrtko's comments:
        - fix the parenthesis alignment warning

Signed-off-by: Yu Zhang <yu.c.zhang@linux.intel.com>
Signed-off-by: Jike Song <jike.song@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 057f905..e54b2a0 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -104,6 +104,9 @@ static int sanitize_enable_ppgtt(struct drm_device *dev, int enable_ppgtt)
 	has_aliasing_ppgtt = INTEL_INFO(dev)->gen >= 6;
 	has_full_ppgtt = INTEL_INFO(dev)->gen >= 7;
 
+	if (intel_vgpu_active(dev))
+		has_full_ppgtt = false; /* emulation is too hard */
+
 	/*
 	 * We don't allow disabling PPGTT for gen9+ as it's a requirement for
 	 * execlists, the sole mechanism available to submit work.
@@ -798,6 +801,16 @@ static int hsw_mm_switch(struct i915_hw_ppgtt *ppgtt,
 	return 0;
 }
 
+static int vgpu_mm_switch(struct i915_hw_ppgtt *ppgtt,
+			  struct intel_engine_cs *ring)
+{
+	struct drm_i915_private *dev_priv = to_i915(ppgtt->base.dev);
+
+	I915_WRITE(RING_PP_DIR_DCLV(ring), PP_DIR_DCLV_2G);
+	I915_WRITE(RING_PP_DIR_BASE(ring), get_pd_offset(ppgtt));
+	return 0;
+}
+
 static int gen7_mm_switch(struct i915_hw_ppgtt *ppgtt,
 			  struct intel_engine_cs *ring)
 {
@@ -1127,6 +1140,9 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	} else
 		BUG();
 
+	if (intel_vgpu_active(dev))
+		ppgtt->switch_mm = vgpu_mm_switch;
+
 	ret = gen6_ppgtt_alloc(ppgtt);
 	if (ret)
 		return ret;
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 0/8] Add enlightenments for vGPU

[Tvrtko Ursulin]

On 02/10/2015 11:05 AM, Yu Zhang wrote:
> This patch set includes necessary code changes when i915 driver
> runs inside a VM. Though ideally we can run an unmodified i915
> driver in VM, adding such enlightenments can greatly reduce the
> virtualization complexity in orders of magnitude. Code changes
> for the host side, which includes the actual Intel GVT-g
> implementation, will be sent out in other patches.
>
> The primary change introduced here is to implement so-called
> "address space ballooning" technique. XenGT partitions global
> graphics memory among multiple VMs, so each VM can directly
> access a portion of the memory without hypervisor's intervention,
> e.g. filling textures or queuing commands. However with the
> partitioning an unmodified i915 driver would assume a smaller
> graphics memory starting from address ZERO, so requires XenGT
> core module (vgt) to translate the graphics address between
> 'guest view' and 'host view', for all registers and command
> opcodes which contain a graphics memory address. To reduce the
> complexity, XenGT introduces "address space ballooning", by
> telling the exact partitioning knowledge to each guest i915
> driver, which then reserves and prevents non-allocated portions
> from allocation. Then vgt module only needs to scan and validate
> graphics addresses without complexity of translation.
>
> Note: The partitioning of global graphics memory may break some
> applications, with large objects in the aperture, because current
> userspace assumes half of the aperture usable. That would need
> separate fix either in user space (e.g. remove assumption in mesa)
> or in kernel (with some faulting mechanism).
>
> The partitioning knowledge is conveyed through a reserved MMIO
> range, called PVINFO, which will be architecturally reserved in
> future hardware generations. Another information carried through
> PVINFO is about the number of fence registers. As a global resource,
> XenGT also partitions them among VMs.
>
> Other changes are trivial as optimizations, to either reduce the
> trap overhead or disable power management features which don't
> make sense in a virtualized environment.
>
>
> Yu Zhang (8):
>    drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
>    drm/i915: Adds graphic address space ballooning logic
>    drm/i915: Partition the fence registers for vGPU in i915 driver
>    drm/i915: Disable framebuffer compression for i915 driver in VM
>    drm/i915: Add the display switch logic for vGPU in i915 driver
>    drm/i915: Disable power management for i915 driver in VM
>    drm/i915: Create vGPU specific MMIO operations to reduce traps
>    drm/i915: Support alias ppgtt in VM if ppgtt is enabled

All my comments have been addressed (and I especially like the ASCII 
diagram of the memory space!) so you can put my r-b on all the patches 
from this series:

Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

Regards,

Tvrtko
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 0/8] Add enlightenments for vGPU

[Yu, Zhang]

On 2/10/2015 8:11 PM, Tvrtko Ursulin wrote:
> On 02/10/2015 11:05 AM, Yu Zhang wrote:
>> This patch set includes necessary code changes when i915 driver
>> runs inside a VM. Though ideally we can run an unmodified i915
>> driver in VM, adding such enlightenments can greatly reduce the
>> virtualization complexity in orders of magnitude. Code changes
>> for the host side, which includes the actual Intel GVT-g
>> implementation, will be sent out in other patches.
>>
>> The primary change introduced here is to implement so-called
>> "address space ballooning" technique. XenGT partitions global
>> graphics memory among multiple VMs, so each VM can directly
>> access a portion of the memory without hypervisor's intervention,
>> e.g. filling textures or queuing commands. However with the
>> partitioning an unmodified i915 driver would assume a smaller
>> graphics memory starting from address ZERO, so requires XenGT
>> core module (vgt) to translate the graphics address between
>> 'guest view' and 'host view', for all registers and command
>> opcodes which contain a graphics memory address. To reduce the
>> complexity, XenGT introduces "address space ballooning", by
>> telling the exact partitioning knowledge to each guest i915
>> driver, which then reserves and prevents non-allocated portions
>> from allocation. Then vgt module only needs to scan and validate
>> graphics addresses without complexity of translation.
>>
>> Note: The partitioning of global graphics memory may break some
>> applications, with large objects in the aperture, because current
>> userspace assumes half of the aperture usable. That would need
>> separate fix either in user space (e.g. remove assumption in mesa)
>> or in kernel (with some faulting mechanism).
>>
>> The partitioning knowledge is conveyed through a reserved MMIO
>> range, called PVINFO, which will be architecturally reserved in
>> future hardware generations. Another information carried through
>> PVINFO is about the number of fence registers. As a global resource,
>> XenGT also partitions them among VMs.
>>
>> Other changes are trivial as optimizations, to either reduce the
>> trap overhead or disable power management features which don't
>> make sense in a virtualized environment.
>>
>>
>> Yu Zhang (8):
>>    drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
>>    drm/i915: Adds graphic address space ballooning logic
>>    drm/i915: Partition the fence registers for vGPU in i915 driver
>>    drm/i915: Disable framebuffer compression for i915 driver in VM
>>    drm/i915: Add the display switch logic for vGPU in i915 driver
>>    drm/i915: Disable power management for i915 driver in VM
>>    drm/i915: Create vGPU specific MMIO operations to reduce traps
>>    drm/i915: Support alias ppgtt in VM if ppgtt is enabled
>
> All my comments have been addressed (and I especially like the ASCII
> diagram of the memory space!) so you can put my r-b on all the patches
> from this series:
>
> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Thank you very much, Tvrtko. This is great. :-)

Btw, should I resend another version patch series, which add the 
"Reviewed-by" line, or will Daniel directly merge these patches and add 
the "Reviewed-by" at that time? I'm willing to take whatever actions 
necessary, but I'm not familiar with this process. Thanks again. :)

Yu
>
> Regards,
>
> Tvrtko
>
>
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 0/8] Add enlightenments for vGPU

[Daniel Vetter]

On Tue, Feb 10, 2015 at 12:11:02PM +0000, Tvrtko Ursulin wrote:
> On 02/10/2015 11:05 AM, Yu Zhang wrote:
> >This patch set includes necessary code changes when i915 driver
> >runs inside a VM. Though ideally we can run an unmodified i915
> >driver in VM, adding such enlightenments can greatly reduce the
> >virtualization complexity in orders of magnitude. Code changes
> >for the host side, which includes the actual Intel GVT-g
> >implementation, will be sent out in other patches.
> >
> >The primary change introduced here is to implement so-called
> >"address space ballooning" technique. XenGT partitions global
> >graphics memory among multiple VMs, so each VM can directly
> >access a portion of the memory without hypervisor's intervention,
> >e.g. filling textures or queuing commands. However with the
> >partitioning an unmodified i915 driver would assume a smaller
> >graphics memory starting from address ZERO, so requires XenGT
> >core module (vgt) to translate the graphics address between
> >'guest view' and 'host view', for all registers and command
> >opcodes which contain a graphics memory address. To reduce the
> >complexity, XenGT introduces "address space ballooning", by
> >telling the exact partitioning knowledge to each guest i915
> >driver, which then reserves and prevents non-allocated portions
> >from allocation. Then vgt module only needs to scan and validate
> >graphics addresses without complexity of translation.
> >
> >Note: The partitioning of global graphics memory may break some
> >applications, with large objects in the aperture, because current
> >userspace assumes half of the aperture usable. That would need
> >separate fix either in user space (e.g. remove assumption in mesa)
> >or in kernel (with some faulting mechanism).
> >
> >The partitioning knowledge is conveyed through a reserved MMIO
> >range, called PVINFO, which will be architecturally reserved in
> >future hardware generations. Another information carried through
> >PVINFO is about the number of fence registers. As a global resource,
> >XenGT also partitions them among VMs.
> >
> >Other changes are trivial as optimizations, to either reduce the
> >trap overhead or disable power management features which don't
> >make sense in a virtualized environment.
> >
> >
> >Yu Zhang (8):
> >   drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
> >   drm/i915: Adds graphic address space ballooning logic
> >   drm/i915: Partition the fence registers for vGPU in i915 driver
> >   drm/i915: Disable framebuffer compression for i915 driver in VM
> >   drm/i915: Add the display switch logic for vGPU in i915 driver
> >   drm/i915: Disable power management for i915 driver in VM
> >   drm/i915: Create vGPU specific MMIO operations to reduce traps
> >   drm/i915: Support alias ppgtt in VM if ppgtt is enabled
> 
> All my comments have been addressed (and I especially like the ASCII diagram
> of the memory space!) so you can put my r-b on all the patches from this
> series:
> 
> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

All merged, thanks for patches&review.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 0/8] Add enlightenments for vGPU

[Yu, Zhang]

On 2/11/2015 4:06 PM, Daniel Vetter wrote:
> On Tue, Feb 10, 2015 at 12:11:02PM +0000, Tvrtko Ursulin wrote:
>> On 02/10/2015 11:05 AM, Yu Zhang wrote:
>>> This patch set includes necessary code changes when i915 driver
>>> runs inside a VM. Though ideally we can run an unmodified i915
>>> driver in VM, adding such enlightenments can greatly reduce the
>>> virtualization complexity in orders of magnitude. Code changes
>>> for the host side, which includes the actual Intel GVT-g
>>> implementation, will be sent out in other patches.
>>>
>>> The primary change introduced here is to implement so-called
>>> "address space ballooning" technique. XenGT partitions global
>>> graphics memory among multiple VMs, so each VM can directly
>>> access a portion of the memory without hypervisor's intervention,
>>> e.g. filling textures or queuing commands. However with the
>>> partitioning an unmodified i915 driver would assume a smaller
>>> graphics memory starting from address ZERO, so requires XenGT
>>> core module (vgt) to translate the graphics address between
>>> 'guest view' and 'host view', for all registers and command
>>> opcodes which contain a graphics memory address. To reduce the
>>> complexity, XenGT introduces "address space ballooning", by
>>> telling the exact partitioning knowledge to each guest i915
>>> driver, which then reserves and prevents non-allocated portions
>> >from allocation. Then vgt module only needs to scan and validate
>>> graphics addresses without complexity of translation.
>>>
>>> Note: The partitioning of global graphics memory may break some
>>> applications, with large objects in the aperture, because current
>>> userspace assumes half of the aperture usable. That would need
>>> separate fix either in user space (e.g. remove assumption in mesa)
>>> or in kernel (with some faulting mechanism).
>>>
>>> The partitioning knowledge is conveyed through a reserved MMIO
>>> range, called PVINFO, which will be architecturally reserved in
>>> future hardware generations. Another information carried through
>>> PVINFO is about the number of fence registers. As a global resource,
>>> XenGT also partitions them among VMs.
>>>
>>> Other changes are trivial as optimizations, to either reduce the
>>> trap overhead or disable power management features which don't
>>> make sense in a virtualized environment.
>>>
>>>
>>> Yu Zhang (8):
>>>    drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
>>>    drm/i915: Adds graphic address space ballooning logic
>>>    drm/i915: Partition the fence registers for vGPU in i915 driver
>>>    drm/i915: Disable framebuffer compression for i915 driver in VM
>>>    drm/i915: Add the display switch logic for vGPU in i915 driver
>>>    drm/i915: Disable power management for i915 driver in VM
>>>    drm/i915: Create vGPU specific MMIO operations to reduce traps
>>>    drm/i915: Support alias ppgtt in VM if ppgtt is enabled
>>
>> All my comments have been addressed (and I especially like the ASCII diagram
>> of the memory space!) so you can put my r-b on all the patches from this
>> series:
>>
>> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
>
> All merged, thanks for patches&review.
> -Daniel
Got it. Thank you, Daniel! :)

Yu
>
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH v4 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.

[Dave Airlie]

> +/*
> + * The following structure pages are defined in GEN MMIO space
> + * for virtualization. (One page for now)
> + */
> +#define VGT_MAGIC         0x4776544776544776   /* 'vGTvGTvG' */

one of my dusty 32-bit compilers dislikes this,

 CC [M]  drivers/gpu/drm/i915/i915_vgpu.o
/home/airlied/kernel/drm-2.6/drivers/gpu/drm/i915/i915_vgpu.c: In
function ‘i915_check_vgpu’:
/home/airlied/kernel/drm-2.6/drivers/gpu/drm/i915/i915_vgpu.c:73:
warning: integer constant is too large for ‘long’ type

Dave.
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx