* [PATCH v2] Persistent grant maps for xen blk drivers
@ 2012-10-24 16:58 Roger Pau Monne
2012-10-29 13:57 ` Konrad Rzeszutek Wilk
2012-10-30 17:01 ` Konrad Rzeszutek Wilk
0 siblings, 2 replies; 13+ messages in thread
From: Roger Pau Monne @ 2012-10-24 16:58 UTC (permalink / raw)
To: xen-devel; +Cc: Roger Pau Monne, Oliver Chick, konrad.wilk, linux-kernel
This patch implements persistent grants for the xen-blk{front,back}
mechanism. The effect of this change is to reduce the number of unmap
operations performed, since they cause a (costly) TLB shootdown. This
allows the I/O performance to scale better when a large number of VMs
are performing I/O.
Previously, the blkfront driver was supplied a bvec[] from the request
queue. This was granted to dom0; dom0 performed the I/O and wrote
directly into the grant-mapped memory and unmapped it; blkfront then
removed foreign access for that grant. The cost of unmapping scales
badly with the number of CPUs in Dom0. An experiment showed that when
Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
(at which point 650,000 IOPS are being performed in total). If more
than 5 guests are used, the performance declines. By 10 guests, only
400,000 IOPS are being performed.
This patch improves performance by only unmapping when the connection
between blkfront and back is broken.
On startup blkfront notifies blkback that it is using persistent
grants, and blkback will do the same. If blkback is not capable of
persistent mapping, blkfront will still use the same grants, since it
is compatible with the previous protocol, and simplifies the code
complexity in blkfront.
To perform a read, in persistent mode, blkfront uses a separate pool
of pages that it maps to dom0. When a request comes in, blkfront
transmutes the request so that blkback will write into one of these
free pages. Blkback keeps note of which grefs it has already
mapped. When a new ring request comes to blkback, it looks to see if
it has already mapped that page. If so, it will not map it again. If
the page hasn't been previously mapped, it is mapped now, and a record
is kept of this mapping. Blkback proceeds as usual. When blkfront is
notified that blkback has completed a request, it memcpy's from the
shared memory, into the bvec supplied. A record that the {gref, page}
tuple is mapped, and not inflight is kept.
Writes are similar, except that the memcpy is peformed from the
supplied bvecs, into the shared pages, before the request is put onto
the ring.
Blkback stores a mapping of grefs=>{page mapped to by gref} in
a red-black tree. As the grefs are not known apriori, and provide no
guarantees on their ordering, we have to perform a search
through this tree to find the page, for every gref we receive. This
operation takes O(log n) time in the worst case. In blkfront grants
are stored using a single linked list.
The maximum number of grants that blkback will persistenly map is
currently set to RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, to
prevent a malicios guest from attempting a DoS, by supplying fresh
grefs, causing the Dom0 kernel to map excessively. If a guest
is using persistent grants and exceeds the maximum number of grants to
map persistenly the newly passed grefs will be mapped and unmaped.
Using this approach, we can have requests that mix persistent and
non-persistent grants, and we need to handle them correctly.
This allows us to set the maximum number of persistent grants to a
lower value than RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, although
setting it will lead to unpredictable performance.
In writing this patch, the question arrises as to if the additional
cost of performing memcpys in the guest (to/from the pool of granted
pages) outweigh the gains of not performing TLB shootdowns. The answer
to that question is `no'. There appears to be very little, if any
additional cost to the guest of using persistent grants. There is
perhaps a small saving, from the reduced number of hypercalls
performed in granting, and ending foreign access.
Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
Cc: <konrad.wilk@oracle.com>
Cc: <linux-kernel@vger.kernel.org>
---
Changes since v1:
* Changed the unmap_seg array to a bitmap.
* Only report using persistent grants in blkfront if blkback supports
it.
* Reword some comments.
* Fix a bug when setting the handler, index j was not incremented
correctly.
* Check that the tree of grants in blkback is not empty before
iterating over it when doing the cleanup.
* Rebase on top of linux-net.
---
Benchmarks showing the impact of this patch in blk performance can be
found at:
http://xenbits.xensource.com/people/royger/persistent_grants/
---
drivers/block/xen-blkback/blkback.c | 292 ++++++++++++++++++++++++++++++++---
drivers/block/xen-blkback/common.h | 17 ++
drivers/block/xen-blkback/xenbus.c | 23 +++-
drivers/block/xen-blkfront.c | 197 ++++++++++++++++++++----
4 files changed, 474 insertions(+), 55 deletions(-)
diff --git a/drivers/block/xen-blkback/blkback.c b/drivers/block/xen-blkback/blkback.c
index 280a138..663d42d 100644
--- a/drivers/block/xen-blkback/blkback.c
+++ b/drivers/block/xen-blkback/blkback.c
@@ -39,6 +39,7 @@
#include <linux/list.h>
#include <linux/delay.h>
#include <linux/freezer.h>
+#include <linux/bitmap.h>
#include <xen/events.h>
#include <xen/page.h>
@@ -79,6 +80,7 @@ struct pending_req {
unsigned short operation;
int status;
struct list_head free_list;
+ DECLARE_BITMAP(unmap_seg, BLKIF_MAX_SEGMENTS_PER_REQUEST);
};
#define BLKBACK_INVALID_HANDLE (~0)
@@ -99,6 +101,36 @@ struct xen_blkbk {
static struct xen_blkbk *blkbk;
/*
+ * Maximum number of grant pages that can be mapped in blkback.
+ * BLKIF_MAX_SEGMENTS_PER_REQUEST * RING_SIZE is the maximum number of
+ * pages that blkback will persistently map.
+ * Currently, this is:
+ * RING_SIZE = 32 (for all known ring types)
+ * BLKIF_MAX_SEGMENTS_PER_REQUEST = 11
+ * sizeof(struct persistent_gnt) = 48
+ * So the maximum memory used to store the grants is:
+ * 32 * 11 * 48 = 16896 bytes
+ */
+static inline unsigned int max_mapped_grant_pages(enum blkif_protocol protocol)
+{
+ switch (protocol) {
+ case BLKIF_PROTOCOL_NATIVE:
+ return __CONST_RING_SIZE(blkif, PAGE_SIZE) *
+ BLKIF_MAX_SEGMENTS_PER_REQUEST;
+ case BLKIF_PROTOCOL_X86_32:
+ return __CONST_RING_SIZE(blkif_x86_32, PAGE_SIZE) *
+ BLKIF_MAX_SEGMENTS_PER_REQUEST;
+ case BLKIF_PROTOCOL_X86_64:
+ return __CONST_RING_SIZE(blkif_x86_64, PAGE_SIZE) *
+ BLKIF_MAX_SEGMENTS_PER_REQUEST;
+ default:
+ BUG();
+ }
+ return 0;
+}
+
+
+/*
* Little helpful macro to figure out the index and virtual address of the
* pending_pages[..]. For each 'pending_req' we have have up to
* BLKIF_MAX_SEGMENTS_PER_REQUEST (11) pages. The seg would be from 0 through
@@ -129,6 +161,57 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif,
static void make_response(struct xen_blkif *blkif, u64 id,
unsigned short op, int st);
+#define foreach_grant(pos, rbtree, node) \
+ for ((pos) = container_of(rb_first((rbtree)), typeof(*(pos)), node); \
+ &(pos)->node != NULL; \
+ (pos) = container_of(rb_next(&(pos)->node), typeof(*(pos)), node))
+
+
+static void add_persistent_gnt(struct rb_root *root,
+ struct persistent_gnt *persistent_gnt)
+{
+ struct rb_node **new = &(root->rb_node), *parent = NULL;
+ struct persistent_gnt *this;
+
+ /* Figure out where to put new node */
+ while (*new) {
+ this = container_of(*new, struct persistent_gnt, node);
+
+ parent = *new;
+ if (persistent_gnt->gnt < this->gnt)
+ new = &((*new)->rb_left);
+ else if (persistent_gnt->gnt > this->gnt)
+ new = &((*new)->rb_right);
+ else {
+ pr_alert(DRV_PFX " trying to add a gref that's already in the tree\n");
+ BUG();
+ }
+ }
+
+ /* Add new node and rebalance tree. */
+ rb_link_node(&(persistent_gnt->node), parent, new);
+ rb_insert_color(&(persistent_gnt->node), root);
+}
+
+static struct persistent_gnt *get_persistent_gnt(struct rb_root *root,
+ grant_ref_t gref)
+{
+ struct persistent_gnt *data;
+ struct rb_node *node = root->rb_node;
+
+ while (node) {
+ data = container_of(node, struct persistent_gnt, node);
+
+ if (gref < data->gnt)
+ node = node->rb_left;
+ else if (gref > data->gnt)
+ node = node->rb_right;
+ else
+ return data;
+ }
+ return NULL;
+}
+
/*
* Retrieve from the 'pending_reqs' a free pending_req structure to be used.
*/
@@ -275,6 +358,11 @@ int xen_blkif_schedule(void *arg)
{
struct xen_blkif *blkif = arg;
struct xen_vbd *vbd = &blkif->vbd;
+ struct gnttab_unmap_grant_ref unmap[BLKIF_MAX_SEGMENTS_PER_REQUEST];
+ struct page *pages[BLKIF_MAX_SEGMENTS_PER_REQUEST];
+ struct persistent_gnt *persistent_gnt;
+ int ret = 0;
+ int segs_to_unmap = 0;
xen_blkif_get(blkif);
@@ -302,6 +390,36 @@ int xen_blkif_schedule(void *arg)
print_stats(blkif);
}
+ /* Free all persistent grant pages */
+ if (!RB_EMPTY_ROOT(&blkif->persistent_gnts)) {
+ foreach_grant(persistent_gnt, &blkif->persistent_gnts, node) {
+ BUG_ON(persistent_gnt->handle ==
+ BLKBACK_INVALID_HANDLE);
+ gnttab_set_unmap_op(&unmap[segs_to_unmap],
+ (unsigned long) pfn_to_kaddr(page_to_pfn(
+ persistent_gnt->page)),
+ GNTMAP_host_map,
+ persistent_gnt->handle);
+
+ pages[segs_to_unmap] = persistent_gnt->page;
+ rb_erase(&persistent_gnt->node,
+ &blkif->persistent_gnts);
+ kfree(persistent_gnt);
+ blkif->persistent_gnt_c--;
+
+ if (++segs_to_unmap == BLKIF_MAX_SEGMENTS_PER_REQUEST ||
+ !rb_next(&persistent_gnt->node)) {
+ ret = gnttab_unmap_refs(unmap, NULL, pages,
+ segs_to_unmap);
+ BUG_ON(ret);
+ segs_to_unmap = 0;
+ }
+ }
+ }
+
+ BUG_ON(blkif->persistent_gnt_c != 0);
+ BUG_ON(!RB_EMPTY_ROOT(&blkif->persistent_gnts));
+
if (log_stats)
print_stats(blkif);
@@ -328,6 +446,8 @@ static void xen_blkbk_unmap(struct pending_req *req)
int ret;
for (i = 0; i < req->nr_pages; i++) {
+ if (!test_bit(i, req->unmap_seg))
+ continue;
handle = pending_handle(req, i);
if (handle == BLKBACK_INVALID_HANDLE)
continue;
@@ -344,12 +464,26 @@ static void xen_blkbk_unmap(struct pending_req *req)
static int xen_blkbk_map(struct blkif_request *req,
struct pending_req *pending_req,
- struct seg_buf seg[])
+ struct seg_buf seg[],
+ struct page *pages[])
{
struct gnttab_map_grant_ref map[BLKIF_MAX_SEGMENTS_PER_REQUEST];
- int i;
+ struct persistent_gnt *persistent_gnts[BLKIF_MAX_SEGMENTS_PER_REQUEST];
+ struct page *pages_to_gnt[BLKIF_MAX_SEGMENTS_PER_REQUEST];
+ struct persistent_gnt *persistent_gnt = NULL;
+ struct xen_blkif *blkif = pending_req->blkif;
+ phys_addr_t addr = 0;
+ int i, j;
+ bool new_map;
int nseg = req->u.rw.nr_segments;
+ int segs_to_map = 0;
int ret = 0;
+ int use_persistent_gnts;
+
+ use_persistent_gnts = (blkif->vbd.feature_gnt_persistent);
+
+ BUG_ON(blkif->persistent_gnt_c >
+ max_mapped_grant_pages(pending_req->blkif->blk_protocol));
/*
* Fill out preq.nr_sects with proper amount of sectors, and setup
@@ -359,36 +493,143 @@ static int xen_blkbk_map(struct blkif_request *req,
for (i = 0; i < nseg; i++) {
uint32_t flags;
- flags = GNTMAP_host_map;
- if (pending_req->operation != BLKIF_OP_READ)
- flags |= GNTMAP_readonly;
- gnttab_set_map_op(&map[i], vaddr(pending_req, i), flags,
- req->u.rw.seg[i].gref,
- pending_req->blkif->domid);
+ if (use_persistent_gnts)
+ persistent_gnt = get_persistent_gnt(
+ &blkif->persistent_gnts,
+ req->u.rw.seg[i].gref);
+
+ if (persistent_gnt) {
+ /*
+ * We are using persistent grants and
+ * the grant is already mapped
+ */
+ new_map = 0;
+ } else if (use_persistent_gnts &&
+ blkif->persistent_gnt_c <
+ max_mapped_grant_pages(blkif->blk_protocol)) {
+ /*
+ * We are using persistent grants, the grant is
+ * not mapped but we have room for it
+ */
+ new_map = 1;
+ persistent_gnt = kzalloc(
+ sizeof(struct persistent_gnt),
+ GFP_KERNEL);
+ if (!persistent_gnt)
+ return -ENOMEM;
+ persistent_gnt->page = alloc_page(GFP_KERNEL);
+ if (!persistent_gnt->page) {
+ kfree(persistent_gnt);
+ return -ENOMEM;
+ }
+ persistent_gnt->gnt = req->u.rw.seg[i].gref;
+
+ pages_to_gnt[segs_to_map] =
+ persistent_gnt->page;
+ addr = (unsigned long) pfn_to_kaddr(
+ page_to_pfn(persistent_gnt->page));
+
+ add_persistent_gnt(&blkif->persistent_gnts,
+ persistent_gnt);
+ blkif->persistent_gnt_c++;
+ pr_debug(DRV_PFX " grant %u added to the tree of persistent grants, using %u/%u\n",
+ persistent_gnt->gnt, blkif->persistent_gnt_c,
+ max_mapped_grant_pages(blkif->blk_protocol));
+ } else {
+ /*
+ * We are either using persistent grants and
+ * hit the maximum limit of grants mapped,
+ * or we are not using persistent grants.
+ */
+ if (use_persistent_gnts &&
+ !blkif->vbd.overflow_max_grants) {
+ blkif->vbd.overflow_max_grants = 1;
+ pr_alert(DRV_PFX " domain %u, device %#x is using maximum number of persistent grants\n",
+ blkif->domid, blkif->vbd.handle);
+ }
+ new_map = 1;
+ pages[i] = blkbk->pending_page(pending_req, i);
+ addr = vaddr(pending_req, i);
+ pages_to_gnt[segs_to_map] =
+ blkbk->pending_page(pending_req, i);
+ }
+
+ if (persistent_gnt) {
+ pages[i] = persistent_gnt->page;
+ persistent_gnts[i] = persistent_gnt;
+ } else {
+ persistent_gnts[i] = NULL;
+ }
+
+ if (new_map) {
+ flags = GNTMAP_host_map;
+ if (!persistent_gnt &&
+ (pending_req->operation != BLKIF_OP_READ))
+ flags |= GNTMAP_readonly;
+ gnttab_set_map_op(&map[segs_to_map++], addr,
+ flags, req->u.rw.seg[i].gref,
+ blkif->domid);
+ }
}
- ret = gnttab_map_refs(map, NULL, &blkbk->pending_page(pending_req, 0), nseg);
- BUG_ON(ret);
+ if (segs_to_map) {
+ ret = gnttab_map_refs(map, NULL, pages_to_gnt, segs_to_map);
+ BUG_ON(ret);
+ }
/*
* Now swizzle the MFN in our domain with the MFN from the other domain
* so that when we access vaddr(pending_req,i) it has the contents of
* the page from the other domain.
*/
- for (i = 0; i < nseg; i++) {
- if (unlikely(map[i].status != 0)) {
- pr_debug(DRV_PFX "invalid buffer -- could not remap it\n");
- map[i].handle = BLKBACK_INVALID_HANDLE;
- ret |= 1;
+ bitmap_zero(pending_req->unmap_seg, BLKIF_MAX_SEGMENTS_PER_REQUEST);
+ for (i = 0, j = 0; i < nseg; i++) {
+ if (!persistent_gnts[i] || !persistent_gnts[i]->handle) {
+ /* This is a newly mapped grant */
+ BUG_ON(j >= segs_to_map);
+ if (unlikely(map[j].status != 0)) {
+ pr_debug(DRV_PFX "invalid buffer -- could not remap it\n");
+ map[j].handle = BLKBACK_INVALID_HANDLE;
+ ret |= 1;
+ if (persistent_gnts[i]) {
+ rb_erase(&persistent_gnts[i]->node,
+ &blkif->persistent_gnts);
+ blkif->persistent_gnt_c--;
+ kfree(persistent_gnts[i]);
+ persistent_gnts[i] = NULL;
+ }
+ }
+ }
+ if (persistent_gnts[i]) {
+ if (!persistent_gnts[i]->handle) {
+ /*
+ * If this is a new persistent grant
+ * save the handler
+ */
+ persistent_gnts[i]->handle = map[j].handle;
+ persistent_gnts[i]->dev_bus_addr =
+ map[j++].dev_bus_addr;
+ }
+ pending_handle(pending_req, i) =
+ persistent_gnts[i]->handle;
+
+ if (ret)
+ continue;
+
+ seg[i].buf = persistent_gnts[i]->dev_bus_addr |
+ (req->u.rw.seg[i].first_sect << 9);
+ } else {
+ pending_handle(pending_req, i) = map[j].handle;
+ bitmap_set(pending_req->unmap_seg, i, 1);
+
+ if (ret) {
+ j++;
+ continue;
+ }
+
+ seg[i].buf = map[j++].dev_bus_addr |
+ (req->u.rw.seg[i].first_sect << 9);
}
-
- pending_handle(pending_req, i) = map[i].handle;
-
- if (ret)
- continue;
-
- seg[i].buf = map[i].dev_bus_addr |
- (req->u.rw.seg[i].first_sect << 9);
}
return ret;
}
@@ -591,6 +832,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif,
int operation;
struct blk_plug plug;
bool drain = false;
+ struct page *pages[BLKIF_MAX_SEGMENTS_PER_REQUEST];
switch (req->operation) {
case BLKIF_OP_READ:
@@ -677,7 +919,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif,
* the hypercall to unmap the grants - that is all done in
* xen_blkbk_unmap.
*/
- if (xen_blkbk_map(req, pending_req, seg))
+ if (xen_blkbk_map(req, pending_req, seg, pages))
goto fail_flush;
/*
@@ -689,7 +931,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif,
for (i = 0; i < nseg; i++) {
while ((bio == NULL) ||
(bio_add_page(bio,
- blkbk->pending_page(pending_req, i),
+ pages[i],
seg[i].nsec << 9,
seg[i].buf & ~PAGE_MASK) == 0)) {
diff --git a/drivers/block/xen-blkback/common.h b/drivers/block/xen-blkback/common.h
index 9ad3b5e..ae7951f 100644
--- a/drivers/block/xen-blkback/common.h
+++ b/drivers/block/xen-blkback/common.h
@@ -34,6 +34,7 @@
#include <linux/vmalloc.h>
#include <linux/wait.h>
#include <linux/io.h>
+#include <linux/rbtree.h>
#include <asm/setup.h>
#include <asm/pgalloc.h>
#include <asm/hypervisor.h>
@@ -160,10 +161,22 @@ struct xen_vbd {
sector_t size;
bool flush_support;
bool discard_secure;
+
+ unsigned int feature_gnt_persistent:1;
+ unsigned int overflow_max_grants:1;
};
struct backend_info;
+
+struct persistent_gnt {
+ struct page *page;
+ grant_ref_t gnt;
+ grant_handle_t handle;
+ uint64_t dev_bus_addr;
+ struct rb_node node;
+};
+
struct xen_blkif {
/* Unique identifier for this interface. */
domid_t domid;
@@ -190,6 +203,10 @@ struct xen_blkif {
struct task_struct *xenblkd;
unsigned int waiting_reqs;
+ /* tree to store persistent grants */
+ struct rb_root persistent_gnts;
+ unsigned int persistent_gnt_c;
+
/* statistics */
unsigned long st_print;
int st_rd_req;
diff --git a/drivers/block/xen-blkback/xenbus.c b/drivers/block/xen-blkback/xenbus.c
index 4f66171..b225026 100644
--- a/drivers/block/xen-blkback/xenbus.c
+++ b/drivers/block/xen-blkback/xenbus.c
@@ -118,6 +118,7 @@ static struct xen_blkif *xen_blkif_alloc(domid_t domid)
atomic_set(&blkif->drain, 0);
blkif->st_print = jiffies;
init_waitqueue_head(&blkif->waiting_to_free);
+ blkif->persistent_gnts.rb_node = NULL;
return blkif;
}
@@ -673,6 +674,13 @@ again:
xen_blkbk_barrier(xbt, be, be->blkif->vbd.flush_support);
+ err = xenbus_printf(xbt, dev->nodename, "feature-persistent", "%u", 1);
+ if (err) {
+ xenbus_dev_fatal(dev, err, "writing %s/feature-persistent",
+ dev->nodename);
+ goto abort;
+ }
+
err = xenbus_printf(xbt, dev->nodename, "sectors", "%llu",
(unsigned long long)vbd_sz(&be->blkif->vbd));
if (err) {
@@ -721,6 +729,7 @@ static int connect_ring(struct backend_info *be)
struct xenbus_device *dev = be->dev;
unsigned long ring_ref;
unsigned int evtchn;
+ unsigned int pers_grants;
char protocol[64] = "";
int err;
@@ -750,8 +759,18 @@ static int connect_ring(struct backend_info *be)
xenbus_dev_fatal(dev, err, "unknown fe protocol %s", protocol);
return -1;
}
- pr_info(DRV_PFX "ring-ref %ld, event-channel %d, protocol %d (%s)\n",
- ring_ref, evtchn, be->blkif->blk_protocol, protocol);
+ err = xenbus_gather(XBT_NIL, dev->otherend,
+ "feature-persistent-grants", "%u",
+ &pers_grants, NULL);
+ if (err)
+ pers_grants = 0;
+
+ be->blkif->vbd.feature_gnt_persistent = pers_grants;
+ be->blkif->vbd.overflow_max_grants = 0;
+
+ pr_info(DRV_PFX "ring-ref %ld, event-channel %d, protocol %d (%s) %s\n",
+ ring_ref, evtchn, be->blkif->blk_protocol, protocol,
+ pers_grants ? "persistent grants" : "");
/* Map the shared frame, irq etc. */
err = xen_blkif_map(be->blkif, ring_ref, evtchn);
diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c
index 007db89..911d733 100644
--- a/drivers/block/xen-blkfront.c
+++ b/drivers/block/xen-blkfront.c
@@ -44,6 +44,7 @@
#include <linux/mutex.h>
#include <linux/scatterlist.h>
#include <linux/bitmap.h>
+#include <linux/llist.h>
#include <xen/xen.h>
#include <xen/xenbus.h>
@@ -64,10 +65,17 @@ enum blkif_state {
BLKIF_STATE_SUSPENDED,
};
+struct grant {
+ grant_ref_t gref;
+ unsigned long pfn;
+ struct llist_node node;
+};
+
struct blk_shadow {
struct blkif_request req;
struct request *request;
unsigned long frame[BLKIF_MAX_SEGMENTS_PER_REQUEST];
+ struct grant *grants_used[BLKIF_MAX_SEGMENTS_PER_REQUEST];
};
static DEFINE_MUTEX(blkfront_mutex);
@@ -97,6 +105,8 @@ struct blkfront_info
struct work_struct work;
struct gnttab_free_callback callback;
struct blk_shadow shadow[BLK_RING_SIZE];
+ struct llist_head persistent_gnts;
+ unsigned int persistent_gnts_c;
unsigned long shadow_free;
unsigned int feature_flush;
unsigned int flush_op;
@@ -104,6 +114,7 @@ struct blkfront_info
unsigned int feature_secdiscard:1;
unsigned int discard_granularity;
unsigned int discard_alignment;
+ unsigned int feature_persistent:1;
int is_ready;
};
@@ -287,21 +298,36 @@ static int blkif_queue_request(struct request *req)
unsigned long id;
unsigned int fsect, lsect;
int i, ref;
+
+ /*
+ * Used to store if we are able to queue the request by just using
+ * existing persistent grants, or if we have to get new grants,
+ * as there are not sufficiently many free.
+ */
+ bool new_persistent_gnts;
grant_ref_t gref_head;
+ struct page *granted_page;
+ struct grant *gnt_list_entry = NULL;
struct scatterlist *sg;
if (unlikely(info->connected != BLKIF_STATE_CONNECTED))
return 1;
- if (gnttab_alloc_grant_references(
- BLKIF_MAX_SEGMENTS_PER_REQUEST, &gref_head) < 0) {
- gnttab_request_free_callback(
- &info->callback,
- blkif_restart_queue_callback,
- info,
- BLKIF_MAX_SEGMENTS_PER_REQUEST);
- return 1;
- }
+ /* Check if we have enought grants to allocate a requests */
+ if (info->persistent_gnts_c < BLKIF_MAX_SEGMENTS_PER_REQUEST) {
+ new_persistent_gnts = 1;
+ if (gnttab_alloc_grant_references(
+ BLKIF_MAX_SEGMENTS_PER_REQUEST - info->persistent_gnts_c,
+ &gref_head) < 0) {
+ gnttab_request_free_callback(
+ &info->callback,
+ blkif_restart_queue_callback,
+ info,
+ BLKIF_MAX_SEGMENTS_PER_REQUEST);
+ return 1;
+ }
+ } else
+ new_persistent_gnts = 0;
/* Fill out a communications ring structure. */
ring_req = RING_GET_REQUEST(&info->ring, info->ring.req_prod_pvt);
@@ -341,18 +367,73 @@ static int blkif_queue_request(struct request *req)
BLKIF_MAX_SEGMENTS_PER_REQUEST);
for_each_sg(info->sg, sg, ring_req->u.rw.nr_segments, i) {
- buffer_mfn = pfn_to_mfn(page_to_pfn(sg_page(sg)));
fsect = sg->offset >> 9;
lsect = fsect + (sg->length >> 9) - 1;
- /* install a grant reference. */
- ref = gnttab_claim_grant_reference(&gref_head);
- BUG_ON(ref == -ENOSPC);
- gnttab_grant_foreign_access_ref(
- ref,
+ if (info->persistent_gnts_c) {
+ BUG_ON(llist_empty(&info->persistent_gnts));
+ gnt_list_entry = llist_entry(
+ llist_del_first(&info->persistent_gnts),
+ struct grant, node);
+
+ ref = gnt_list_entry->gref;
+ buffer_mfn = pfn_to_mfn(gnt_list_entry->pfn);
+ info->persistent_gnts_c--;
+ } else {
+ ref = gnttab_claim_grant_reference(&gref_head);
+ BUG_ON(ref == -ENOSPC);
+
+ gnt_list_entry =
+ kmalloc(sizeof(struct grant),
+ GFP_ATOMIC);
+ if (!gnt_list_entry)
+ return -ENOMEM;
+
+ granted_page = alloc_page(GFP_ATOMIC);
+ if (!granted_page) {
+ kfree(gnt_list_entry);
+ return -ENOMEM;
+ }
+
+ gnt_list_entry->pfn =
+ page_to_pfn(granted_page);
+ gnt_list_entry->gref = ref;
+
+ buffer_mfn = pfn_to_mfn(page_to_pfn(
+ granted_page));
+ gnttab_grant_foreign_access_ref(ref,
info->xbdev->otherend_id,
- buffer_mfn,
- rq_data_dir(req));
+ buffer_mfn, 0);
+ }
+
+ info->shadow[id].grants_used[i] = gnt_list_entry;
+
+ if (rq_data_dir(req)) {
+ char *bvec_data;
+ void *shared_data;
+
+ BUG_ON(sg->offset + sg->length > PAGE_SIZE);
+
+ shared_data = kmap_atomic(
+ pfn_to_page(gnt_list_entry->pfn));
+ bvec_data = kmap_atomic(sg_page(sg));
+
+ /*
+ * this does not wipe data stored outside the
+ * range sg->offset..sg->offset+sg->length.
+ * Therefore, blkback *could* see data from
+ * previous requests. This is OK as long as
+ * persistent grants are shared with just one
+ * domain. It may need refactoring if this
+ * changes
+ */
+ memcpy(shared_data + sg->offset,
+ bvec_data + sg->offset,
+ sg->length);
+
+ kunmap_atomic(bvec_data);
+ kunmap_atomic(shared_data);
+ }
info->shadow[id].frame[i] = mfn_to_pfn(buffer_mfn);
ring_req->u.rw.seg[i] =
@@ -368,7 +449,8 @@ static int blkif_queue_request(struct request *req)
/* Keep a private copy so we can reissue requests when recovering. */
info->shadow[id].req = *ring_req;
- gnttab_free_grant_references(gref_head);
+ if (new_persistent_gnts)
+ gnttab_free_grant_references(gref_head);
return 0;
}
@@ -480,12 +562,13 @@ static int xlvbd_init_blk_queue(struct gendisk *gd, u16 sector_size)
static void xlvbd_flush(struct blkfront_info *info)
{
blk_queue_flush(info->rq, info->feature_flush);
- printk(KERN_INFO "blkfront: %s: %s: %s\n",
+ printk(KERN_INFO "blkfront: %s: %s: %s %s\n",
info->gd->disk_name,
info->flush_op == BLKIF_OP_WRITE_BARRIER ?
"barrier" : (info->flush_op == BLKIF_OP_FLUSH_DISKCACHE ?
"flush diskcache" : "barrier or flush"),
- info->feature_flush ? "enabled" : "disabled");
+ info->feature_flush ? "enabled" : "disabled",
+ info->feature_persistent ? "using persistent grants" : "");
}
static int xen_translate_vdev(int vdevice, int *minor, unsigned int *offset)
@@ -707,6 +790,9 @@ static void blkif_restart_queue(struct work_struct *work)
static void blkif_free(struct blkfront_info *info, int suspend)
{
+ struct llist_node *all_gnts;
+ struct grant *persistent_gnt;
+
/* Prevent new requests being issued until we fix things up. */
spin_lock_irq(&info->io_lock);
info->connected = suspend ?
@@ -714,6 +800,17 @@ static void blkif_free(struct blkfront_info *info, int suspend)
/* No more blkif_request(). */
if (info->rq)
blk_stop_queue(info->rq);
+
+ /* Remove all persistent grants */
+ if (info->persistent_gnts_c) {
+ all_gnts = llist_del_all(&info->persistent_gnts);
+ llist_for_each_entry(persistent_gnt, all_gnts, node) {
+ gnttab_end_foreign_access(persistent_gnt->gref, 0, 0UL);
+ kfree(persistent_gnt);
+ }
+ info->persistent_gnts_c = 0;
+ }
+
/* No more gnttab callback work. */
gnttab_cancel_free_callback(&info->callback);
spin_unlock_irq(&info->io_lock);
@@ -734,13 +831,42 @@ static void blkif_free(struct blkfront_info *info, int suspend)
}
-static void blkif_completion(struct blk_shadow *s)
+static void blkif_completion(struct blk_shadow *s, struct blkfront_info *info,
+ struct blkif_response *bret)
{
int i;
- /* Do not let BLKIF_OP_DISCARD as nr_segment is in the same place
- * flag. */
- for (i = 0; i < s->req.u.rw.nr_segments; i++)
- gnttab_end_foreign_access(s->req.u.rw.seg[i].gref, 0, 0UL);
+ struct bio_vec *bvec;
+ struct req_iterator iter;
+ unsigned long flags;
+ char *bvec_data;
+ void *shared_data;
+ unsigned int offset = 0;
+
+ if (bret->operation == BLKIF_OP_READ) {
+ /*
+ * Copy the data received from the backend into the bvec.
+ * Since bv_offset can be different than 0, and bv_len different
+ * than PAGE_SIZE, we have to keep track of the current offset,
+ * to be sure we are copying the data from the right shared page.
+ */
+ rq_for_each_segment(bvec, s->request, iter) {
+ BUG_ON((bvec->bv_offset + bvec->bv_len) > PAGE_SIZE);
+ i = offset >> PAGE_SHIFT;
+ shared_data = kmap_atomic(
+ pfn_to_page(s->grants_used[i]->pfn));
+ bvec_data = bvec_kmap_irq(bvec, &flags);
+ memcpy(bvec_data, shared_data + bvec->bv_offset,
+ bvec->bv_len);
+ bvec_kunmap_irq(bvec_data, &flags);
+ kunmap_atomic(shared_data);
+ offset += bvec->bv_len;
+ }
+ }
+ /* Add the persistent grant into the list of free grants */
+ for (i = 0; i < s->req.u.rw.nr_segments; i++) {
+ llist_add(&s->grants_used[i]->node, &info->persistent_gnts);
+ info->persistent_gnts_c++;
+ }
}
static irqreturn_t blkif_interrupt(int irq, void *dev_id)
@@ -783,7 +909,7 @@ static irqreturn_t blkif_interrupt(int irq, void *dev_id)
req = info->shadow[id].request;
if (bret->operation != BLKIF_OP_DISCARD)
- blkif_completion(&info->shadow[id]);
+ blkif_completion(&info->shadow[id], info, bret);
if (add_id_to_freelist(info, id)) {
WARN(1, "%s: response to %s (id %ld) couldn't be recycled!\n",
@@ -942,6 +1068,11 @@ again:
message = "writing protocol";
goto abort_transaction;
}
+ err = xenbus_printf(xbt, dev->nodename,
+ "feature-persistent-grants", "%u", 1);
+ if (err)
+ dev_warn(&dev->dev,
+ "writing persistent grants feature to xenbus");
err = xenbus_transaction_end(xbt, 0);
if (err) {
@@ -1029,6 +1160,8 @@ static int blkfront_probe(struct xenbus_device *dev,
spin_lock_init(&info->io_lock);
info->xbdev = dev;
info->vdevice = vdevice;
+ init_llist_head(&info->persistent_gnts);
+ info->persistent_gnts_c = 0;
info->connected = BLKIF_STATE_DISCONNECTED;
INIT_WORK(&info->work, blkif_restart_queue);
@@ -1093,7 +1226,7 @@ static int blkif_recover(struct blkfront_info *info)
req->u.rw.seg[j].gref,
info->xbdev->otherend_id,
pfn_to_mfn(info->shadow[req->u.rw.id].frame[j]),
- rq_data_dir(info->shadow[req->u.rw.id].request));
+ 0);
}
info->shadow[req->u.rw.id].req = *req;
@@ -1225,7 +1358,7 @@ static void blkfront_connect(struct blkfront_info *info)
unsigned long sector_size;
unsigned int binfo;
int err;
- int barrier, flush, discard;
+ int barrier, flush, discard, persistent;
switch (info->connected) {
case BLKIF_STATE_CONNECTED:
@@ -1303,6 +1436,14 @@ static void blkfront_connect(struct blkfront_info *info)
if (!err && discard)
blkfront_setup_discard(info);
+ err = xenbus_gather(XBT_NIL, info->xbdev->otherend,
+ "feature-persistent", "%u", &persistent,
+ NULL);
+ if (err)
+ info->feature_persistent = 0;
+ else
+ info->feature_persistent = persistent;
+
err = xlvbd_alloc_gendisk(sectors, info, binfo, sector_size);
if (err) {
xenbus_dev_fatal(info->xbdev, err, "xlvbd_add at %s",
--
1.7.7.5 (Apple Git-26)
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH v2] Persistent grant maps for xen blk drivers
2012-10-24 16:58 [PATCH v2] Persistent grant maps for xen blk drivers Roger Pau Monne
@ 2012-10-29 13:57 ` Konrad Rzeszutek Wilk
2012-10-30 17:01 ` Konrad Rzeszutek Wilk
1 sibling, 0 replies; 13+ messages in thread
From: Konrad Rzeszutek Wilk @ 2012-10-29 13:57 UTC (permalink / raw)
To: Roger Pau Monne; +Cc: xen-devel, Oliver Chick, linux-kernel
On Wed, Oct 24, 2012 at 06:58:45PM +0200, Roger Pau Monne wrote:
> This patch implements persistent grants for the xen-blk{front,back}
> mechanism. The effect of this change is to reduce the number of unmap
> operations performed, since they cause a (costly) TLB shootdown. This
> allows the I/O performance to scale better when a large number of VMs
> are performing I/O.
>
> Previously, the blkfront driver was supplied a bvec[] from the request
> queue. This was granted to dom0; dom0 performed the I/O and wrote
> directly into the grant-mapped memory and unmapped it; blkfront then
> removed foreign access for that grant. The cost of unmapping scales
> badly with the number of CPUs in Dom0. An experiment showed that when
> Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
> ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
> (at which point 650,000 IOPS are being performed in total). If more
> than 5 guests are used, the performance declines. By 10 guests, only
> 400,000 IOPS are being performed.
>
> This patch improves performance by only unmapping when the connection
> between blkfront and back is broken.
>
> On startup blkfront notifies blkback that it is using persistent
> grants, and blkback will do the same. If blkback is not capable of
> persistent mapping, blkfront will still use the same grants, since it
> is compatible with the previous protocol, and simplifies the code
> complexity in blkfront.
>
> To perform a read, in persistent mode, blkfront uses a separate pool
> of pages that it maps to dom0. When a request comes in, blkfront
> transmutes the request so that blkback will write into one of these
> free pages. Blkback keeps note of which grefs it has already
> mapped. When a new ring request comes to blkback, it looks to see if
> it has already mapped that page. If so, it will not map it again. If
> the page hasn't been previously mapped, it is mapped now, and a record
> is kept of this mapping. Blkback proceeds as usual. When blkfront is
> notified that blkback has completed a request, it memcpy's from the
> shared memory, into the bvec supplied. A record that the {gref, page}
> tuple is mapped, and not inflight is kept.
>
> Writes are similar, except that the memcpy is peformed from the
> supplied bvecs, into the shared pages, before the request is put onto
> the ring.
>
> Blkback stores a mapping of grefs=>{page mapped to by gref} in
> a red-black tree. As the grefs are not known apriori, and provide no
> guarantees on their ordering, we have to perform a search
> through this tree to find the page, for every gref we receive. This
> operation takes O(log n) time in the worst case. In blkfront grants
> are stored using a single linked list.
>
> The maximum number of grants that blkback will persistenly map is
> currently set to RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, to
> prevent a malicios guest from attempting a DoS, by supplying fresh
> grefs, causing the Dom0 kernel to map excessively. If a guest
> is using persistent grants and exceeds the maximum number of grants to
> map persistenly the newly passed grefs will be mapped and unmaped.
> Using this approach, we can have requests that mix persistent and
> non-persistent grants, and we need to handle them correctly.
> This allows us to set the maximum number of persistent grants to a
> lower value than RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, although
> setting it will lead to unpredictable performance.
>
> In writing this patch, the question arrises as to if the additional
> cost of performing memcpys in the guest (to/from the pool of granted
> pages) outweigh the gains of not performing TLB shootdowns. The answer
> to that question is `no'. There appears to be very little, if any
> additional cost to the guest of using persistent grants. There is
> perhaps a small saving, from the reduced number of hypercalls
> performed in granting, and ending foreign access.
>
> Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
> Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
> Cc: <konrad.wilk@oracle.com>
> Cc: <linux-kernel@vger.kernel.org>
> ---
> Changes since v1:
> * Changed the unmap_seg array to a bitmap.
> * Only report using persistent grants in blkfront if blkback supports
> it.
> * Reword some comments.
> * Fix a bug when setting the handler, index j was not incremented
> correctly.
> * Check that the tree of grants in blkback is not empty before
> iterating over it when doing the cleanup.
> * Rebase on top of linux-net.
It looks good to me. Going to stick it on my for-jens-3.8 branch.
> ---
> Benchmarks showing the impact of this patch in blk performance can be
> found at:
>
> http://xenbits.xensource.com/people/royger/persistent_grants/
> ---
> drivers/block/xen-blkback/blkback.c | 292 ++++++++++++++++++++++++++++++++---
> drivers/block/xen-blkback/common.h | 17 ++
> drivers/block/xen-blkback/xenbus.c | 23 +++-
> drivers/block/xen-blkfront.c | 197 ++++++++++++++++++++----
> 4 files changed, 474 insertions(+), 55 deletions(-)
>
> diff --git a/drivers/block/xen-blkback/blkback.c b/drivers/block/xen-blkback/blkback.c
> index 280a138..663d42d 100644
> --- a/drivers/block/xen-blkback/blkback.c
> +++ b/drivers/block/xen-blkback/blkback.c
> @@ -39,6 +39,7 @@
> #include <linux/list.h>
> #include <linux/delay.h>
> #include <linux/freezer.h>
> +#include <linux/bitmap.h>
>
> #include <xen/events.h>
> #include <xen/page.h>
> @@ -79,6 +80,7 @@ struct pending_req {
> unsigned short operation;
> int status;
> struct list_head free_list;
> + DECLARE_BITMAP(unmap_seg, BLKIF_MAX_SEGMENTS_PER_REQUEST);
> };
>
> #define BLKBACK_INVALID_HANDLE (~0)
> @@ -99,6 +101,36 @@ struct xen_blkbk {
> static struct xen_blkbk *blkbk;
>
> /*
> + * Maximum number of grant pages that can be mapped in blkback.
> + * BLKIF_MAX_SEGMENTS_PER_REQUEST * RING_SIZE is the maximum number of
> + * pages that blkback will persistently map.
> + * Currently, this is:
> + * RING_SIZE = 32 (for all known ring types)
> + * BLKIF_MAX_SEGMENTS_PER_REQUEST = 11
> + * sizeof(struct persistent_gnt) = 48
> + * So the maximum memory used to store the grants is:
> + * 32 * 11 * 48 = 16896 bytes
> + */
> +static inline unsigned int max_mapped_grant_pages(enum blkif_protocol protocol)
> +{
> + switch (protocol) {
> + case BLKIF_PROTOCOL_NATIVE:
> + return __CONST_RING_SIZE(blkif, PAGE_SIZE) *
> + BLKIF_MAX_SEGMENTS_PER_REQUEST;
> + case BLKIF_PROTOCOL_X86_32:
> + return __CONST_RING_SIZE(blkif_x86_32, PAGE_SIZE) *
> + BLKIF_MAX_SEGMENTS_PER_REQUEST;
> + case BLKIF_PROTOCOL_X86_64:
> + return __CONST_RING_SIZE(blkif_x86_64, PAGE_SIZE) *
> + BLKIF_MAX_SEGMENTS_PER_REQUEST;
> + default:
> + BUG();
> + }
> + return 0;
> +}
> +
> +
> +/*
> * Little helpful macro to figure out the index and virtual address of the
> * pending_pages[..]. For each 'pending_req' we have have up to
> * BLKIF_MAX_SEGMENTS_PER_REQUEST (11) pages. The seg would be from 0 through
> @@ -129,6 +161,57 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif,
> static void make_response(struct xen_blkif *blkif, u64 id,
> unsigned short op, int st);
>
> +#define foreach_grant(pos, rbtree, node) \
> + for ((pos) = container_of(rb_first((rbtree)), typeof(*(pos)), node); \
> + &(pos)->node != NULL; \
> + (pos) = container_of(rb_next(&(pos)->node), typeof(*(pos)), node))
> +
> +
> +static void add_persistent_gnt(struct rb_root *root,
> + struct persistent_gnt *persistent_gnt)
> +{
> + struct rb_node **new = &(root->rb_node), *parent = NULL;
> + struct persistent_gnt *this;
> +
> + /* Figure out where to put new node */
> + while (*new) {
> + this = container_of(*new, struct persistent_gnt, node);
> +
> + parent = *new;
> + if (persistent_gnt->gnt < this->gnt)
> + new = &((*new)->rb_left);
> + else if (persistent_gnt->gnt > this->gnt)
> + new = &((*new)->rb_right);
> + else {
> + pr_alert(DRV_PFX " trying to add a gref that's already in the tree\n");
> + BUG();
> + }
> + }
> +
> + /* Add new node and rebalance tree. */
> + rb_link_node(&(persistent_gnt->node), parent, new);
> + rb_insert_color(&(persistent_gnt->node), root);
> +}
> +
> +static struct persistent_gnt *get_persistent_gnt(struct rb_root *root,
> + grant_ref_t gref)
> +{
> + struct persistent_gnt *data;
> + struct rb_node *node = root->rb_node;
> +
> + while (node) {
> + data = container_of(node, struct persistent_gnt, node);
> +
> + if (gref < data->gnt)
> + node = node->rb_left;
> + else if (gref > data->gnt)
> + node = node->rb_right;
> + else
> + return data;
> + }
> + return NULL;
> +}
> +
> /*
> * Retrieve from the 'pending_reqs' a free pending_req structure to be used.
> */
> @@ -275,6 +358,11 @@ int xen_blkif_schedule(void *arg)
> {
> struct xen_blkif *blkif = arg;
> struct xen_vbd *vbd = &blkif->vbd;
> + struct gnttab_unmap_grant_ref unmap[BLKIF_MAX_SEGMENTS_PER_REQUEST];
> + struct page *pages[BLKIF_MAX_SEGMENTS_PER_REQUEST];
> + struct persistent_gnt *persistent_gnt;
> + int ret = 0;
> + int segs_to_unmap = 0;
>
> xen_blkif_get(blkif);
>
> @@ -302,6 +390,36 @@ int xen_blkif_schedule(void *arg)
> print_stats(blkif);
> }
>
> + /* Free all persistent grant pages */
> + if (!RB_EMPTY_ROOT(&blkif->persistent_gnts)) {
> + foreach_grant(persistent_gnt, &blkif->persistent_gnts, node) {
> + BUG_ON(persistent_gnt->handle ==
> + BLKBACK_INVALID_HANDLE);
> + gnttab_set_unmap_op(&unmap[segs_to_unmap],
> + (unsigned long) pfn_to_kaddr(page_to_pfn(
> + persistent_gnt->page)),
> + GNTMAP_host_map,
> + persistent_gnt->handle);
> +
> + pages[segs_to_unmap] = persistent_gnt->page;
> + rb_erase(&persistent_gnt->node,
> + &blkif->persistent_gnts);
> + kfree(persistent_gnt);
> + blkif->persistent_gnt_c--;
> +
> + if (++segs_to_unmap == BLKIF_MAX_SEGMENTS_PER_REQUEST ||
> + !rb_next(&persistent_gnt->node)) {
> + ret = gnttab_unmap_refs(unmap, NULL, pages,
> + segs_to_unmap);
> + BUG_ON(ret);
> + segs_to_unmap = 0;
> + }
> + }
> + }
> +
> + BUG_ON(blkif->persistent_gnt_c != 0);
> + BUG_ON(!RB_EMPTY_ROOT(&blkif->persistent_gnts));
> +
> if (log_stats)
> print_stats(blkif);
>
> @@ -328,6 +446,8 @@ static void xen_blkbk_unmap(struct pending_req *req)
> int ret;
>
> for (i = 0; i < req->nr_pages; i++) {
> + if (!test_bit(i, req->unmap_seg))
> + continue;
> handle = pending_handle(req, i);
> if (handle == BLKBACK_INVALID_HANDLE)
> continue;
> @@ -344,12 +464,26 @@ static void xen_blkbk_unmap(struct pending_req *req)
>
> static int xen_blkbk_map(struct blkif_request *req,
> struct pending_req *pending_req,
> - struct seg_buf seg[])
> + struct seg_buf seg[],
> + struct page *pages[])
> {
> struct gnttab_map_grant_ref map[BLKIF_MAX_SEGMENTS_PER_REQUEST];
> - int i;
> + struct persistent_gnt *persistent_gnts[BLKIF_MAX_SEGMENTS_PER_REQUEST];
> + struct page *pages_to_gnt[BLKIF_MAX_SEGMENTS_PER_REQUEST];
> + struct persistent_gnt *persistent_gnt = NULL;
> + struct xen_blkif *blkif = pending_req->blkif;
> + phys_addr_t addr = 0;
> + int i, j;
> + bool new_map;
> int nseg = req->u.rw.nr_segments;
> + int segs_to_map = 0;
> int ret = 0;
> + int use_persistent_gnts;
> +
> + use_persistent_gnts = (blkif->vbd.feature_gnt_persistent);
> +
> + BUG_ON(blkif->persistent_gnt_c >
> + max_mapped_grant_pages(pending_req->blkif->blk_protocol));
>
> /*
> * Fill out preq.nr_sects with proper amount of sectors, and setup
> @@ -359,36 +493,143 @@ static int xen_blkbk_map(struct blkif_request *req,
> for (i = 0; i < nseg; i++) {
> uint32_t flags;
>
> - flags = GNTMAP_host_map;
> - if (pending_req->operation != BLKIF_OP_READ)
> - flags |= GNTMAP_readonly;
> - gnttab_set_map_op(&map[i], vaddr(pending_req, i), flags,
> - req->u.rw.seg[i].gref,
> - pending_req->blkif->domid);
> + if (use_persistent_gnts)
> + persistent_gnt = get_persistent_gnt(
> + &blkif->persistent_gnts,
> + req->u.rw.seg[i].gref);
> +
> + if (persistent_gnt) {
> + /*
> + * We are using persistent grants and
> + * the grant is already mapped
> + */
> + new_map = 0;
> + } else if (use_persistent_gnts &&
> + blkif->persistent_gnt_c <
> + max_mapped_grant_pages(blkif->blk_protocol)) {
> + /*
> + * We are using persistent grants, the grant is
> + * not mapped but we have room for it
> + */
> + new_map = 1;
> + persistent_gnt = kzalloc(
> + sizeof(struct persistent_gnt),
> + GFP_KERNEL);
> + if (!persistent_gnt)
> + return -ENOMEM;
> + persistent_gnt->page = alloc_page(GFP_KERNEL);
> + if (!persistent_gnt->page) {
> + kfree(persistent_gnt);
> + return -ENOMEM;
> + }
> + persistent_gnt->gnt = req->u.rw.seg[i].gref;
> +
> + pages_to_gnt[segs_to_map] =
> + persistent_gnt->page;
> + addr = (unsigned long) pfn_to_kaddr(
> + page_to_pfn(persistent_gnt->page));
> +
> + add_persistent_gnt(&blkif->persistent_gnts,
> + persistent_gnt);
> + blkif->persistent_gnt_c++;
> + pr_debug(DRV_PFX " grant %u added to the tree of persistent grants, using %u/%u\n",
> + persistent_gnt->gnt, blkif->persistent_gnt_c,
> + max_mapped_grant_pages(blkif->blk_protocol));
> + } else {
> + /*
> + * We are either using persistent grants and
> + * hit the maximum limit of grants mapped,
> + * or we are not using persistent grants.
> + */
> + if (use_persistent_gnts &&
> + !blkif->vbd.overflow_max_grants) {
> + blkif->vbd.overflow_max_grants = 1;
> + pr_alert(DRV_PFX " domain %u, device %#x is using maximum number of persistent grants\n",
> + blkif->domid, blkif->vbd.handle);
> + }
> + new_map = 1;
> + pages[i] = blkbk->pending_page(pending_req, i);
> + addr = vaddr(pending_req, i);
> + pages_to_gnt[segs_to_map] =
> + blkbk->pending_page(pending_req, i);
> + }
> +
> + if (persistent_gnt) {
> + pages[i] = persistent_gnt->page;
> + persistent_gnts[i] = persistent_gnt;
> + } else {
> + persistent_gnts[i] = NULL;
> + }
> +
> + if (new_map) {
> + flags = GNTMAP_host_map;
> + if (!persistent_gnt &&
> + (pending_req->operation != BLKIF_OP_READ))
> + flags |= GNTMAP_readonly;
> + gnttab_set_map_op(&map[segs_to_map++], addr,
> + flags, req->u.rw.seg[i].gref,
> + blkif->domid);
> + }
> }
>
> - ret = gnttab_map_refs(map, NULL, &blkbk->pending_page(pending_req, 0), nseg);
> - BUG_ON(ret);
> + if (segs_to_map) {
> + ret = gnttab_map_refs(map, NULL, pages_to_gnt, segs_to_map);
> + BUG_ON(ret);
> + }
>
> /*
> * Now swizzle the MFN in our domain with the MFN from the other domain
> * so that when we access vaddr(pending_req,i) it has the contents of
> * the page from the other domain.
> */
> - for (i = 0; i < nseg; i++) {
> - if (unlikely(map[i].status != 0)) {
> - pr_debug(DRV_PFX "invalid buffer -- could not remap it\n");
> - map[i].handle = BLKBACK_INVALID_HANDLE;
> - ret |= 1;
> + bitmap_zero(pending_req->unmap_seg, BLKIF_MAX_SEGMENTS_PER_REQUEST);
> + for (i = 0, j = 0; i < nseg; i++) {
> + if (!persistent_gnts[i] || !persistent_gnts[i]->handle) {
> + /* This is a newly mapped grant */
> + BUG_ON(j >= segs_to_map);
> + if (unlikely(map[j].status != 0)) {
> + pr_debug(DRV_PFX "invalid buffer -- could not remap it\n");
> + map[j].handle = BLKBACK_INVALID_HANDLE;
> + ret |= 1;
> + if (persistent_gnts[i]) {
> + rb_erase(&persistent_gnts[i]->node,
> + &blkif->persistent_gnts);
> + blkif->persistent_gnt_c--;
> + kfree(persistent_gnts[i]);
> + persistent_gnts[i] = NULL;
> + }
> + }
> + }
> + if (persistent_gnts[i]) {
> + if (!persistent_gnts[i]->handle) {
> + /*
> + * If this is a new persistent grant
> + * save the handler
> + */
> + persistent_gnts[i]->handle = map[j].handle;
> + persistent_gnts[i]->dev_bus_addr =
> + map[j++].dev_bus_addr;
> + }
> + pending_handle(pending_req, i) =
> + persistent_gnts[i]->handle;
> +
> + if (ret)
> + continue;
> +
> + seg[i].buf = persistent_gnts[i]->dev_bus_addr |
> + (req->u.rw.seg[i].first_sect << 9);
> + } else {
> + pending_handle(pending_req, i) = map[j].handle;
> + bitmap_set(pending_req->unmap_seg, i, 1);
> +
> + if (ret) {
> + j++;
> + continue;
> + }
> +
> + seg[i].buf = map[j++].dev_bus_addr |
> + (req->u.rw.seg[i].first_sect << 9);
> }
> -
> - pending_handle(pending_req, i) = map[i].handle;
> -
> - if (ret)
> - continue;
> -
> - seg[i].buf = map[i].dev_bus_addr |
> - (req->u.rw.seg[i].first_sect << 9);
> }
> return ret;
> }
> @@ -591,6 +832,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif,
> int operation;
> struct blk_plug plug;
> bool drain = false;
> + struct page *pages[BLKIF_MAX_SEGMENTS_PER_REQUEST];
>
> switch (req->operation) {
> case BLKIF_OP_READ:
> @@ -677,7 +919,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif,
> * the hypercall to unmap the grants - that is all done in
> * xen_blkbk_unmap.
> */
> - if (xen_blkbk_map(req, pending_req, seg))
> + if (xen_blkbk_map(req, pending_req, seg, pages))
> goto fail_flush;
>
> /*
> @@ -689,7 +931,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif,
> for (i = 0; i < nseg; i++) {
> while ((bio == NULL) ||
> (bio_add_page(bio,
> - blkbk->pending_page(pending_req, i),
> + pages[i],
> seg[i].nsec << 9,
> seg[i].buf & ~PAGE_MASK) == 0)) {
>
> diff --git a/drivers/block/xen-blkback/common.h b/drivers/block/xen-blkback/common.h
> index 9ad3b5e..ae7951f 100644
> --- a/drivers/block/xen-blkback/common.h
> +++ b/drivers/block/xen-blkback/common.h
> @@ -34,6 +34,7 @@
> #include <linux/vmalloc.h>
> #include <linux/wait.h>
> #include <linux/io.h>
> +#include <linux/rbtree.h>
> #include <asm/setup.h>
> #include <asm/pgalloc.h>
> #include <asm/hypervisor.h>
> @@ -160,10 +161,22 @@ struct xen_vbd {
> sector_t size;
> bool flush_support;
> bool discard_secure;
> +
> + unsigned int feature_gnt_persistent:1;
> + unsigned int overflow_max_grants:1;
> };
>
> struct backend_info;
>
> +
> +struct persistent_gnt {
> + struct page *page;
> + grant_ref_t gnt;
> + grant_handle_t handle;
> + uint64_t dev_bus_addr;
> + struct rb_node node;
> +};
> +
> struct xen_blkif {
> /* Unique identifier for this interface. */
> domid_t domid;
> @@ -190,6 +203,10 @@ struct xen_blkif {
> struct task_struct *xenblkd;
> unsigned int waiting_reqs;
>
> + /* tree to store persistent grants */
> + struct rb_root persistent_gnts;
> + unsigned int persistent_gnt_c;
> +
> /* statistics */
> unsigned long st_print;
> int st_rd_req;
> diff --git a/drivers/block/xen-blkback/xenbus.c b/drivers/block/xen-blkback/xenbus.c
> index 4f66171..b225026 100644
> --- a/drivers/block/xen-blkback/xenbus.c
> +++ b/drivers/block/xen-blkback/xenbus.c
> @@ -118,6 +118,7 @@ static struct xen_blkif *xen_blkif_alloc(domid_t domid)
> atomic_set(&blkif->drain, 0);
> blkif->st_print = jiffies;
> init_waitqueue_head(&blkif->waiting_to_free);
> + blkif->persistent_gnts.rb_node = NULL;
>
> return blkif;
> }
> @@ -673,6 +674,13 @@ again:
>
> xen_blkbk_barrier(xbt, be, be->blkif->vbd.flush_support);
>
> + err = xenbus_printf(xbt, dev->nodename, "feature-persistent", "%u", 1);
> + if (err) {
> + xenbus_dev_fatal(dev, err, "writing %s/feature-persistent",
> + dev->nodename);
> + goto abort;
> + }
> +
> err = xenbus_printf(xbt, dev->nodename, "sectors", "%llu",
> (unsigned long long)vbd_sz(&be->blkif->vbd));
> if (err) {
> @@ -721,6 +729,7 @@ static int connect_ring(struct backend_info *be)
> struct xenbus_device *dev = be->dev;
> unsigned long ring_ref;
> unsigned int evtchn;
> + unsigned int pers_grants;
> char protocol[64] = "";
> int err;
>
> @@ -750,8 +759,18 @@ static int connect_ring(struct backend_info *be)
> xenbus_dev_fatal(dev, err, "unknown fe protocol %s", protocol);
> return -1;
> }
> - pr_info(DRV_PFX "ring-ref %ld, event-channel %d, protocol %d (%s)\n",
> - ring_ref, evtchn, be->blkif->blk_protocol, protocol);
> + err = xenbus_gather(XBT_NIL, dev->otherend,
> + "feature-persistent-grants", "%u",
> + &pers_grants, NULL);
> + if (err)
> + pers_grants = 0;
> +
> + be->blkif->vbd.feature_gnt_persistent = pers_grants;
> + be->blkif->vbd.overflow_max_grants = 0;
> +
> + pr_info(DRV_PFX "ring-ref %ld, event-channel %d, protocol %d (%s) %s\n",
> + ring_ref, evtchn, be->blkif->blk_protocol, protocol,
> + pers_grants ? "persistent grants" : "");
>
> /* Map the shared frame, irq etc. */
> err = xen_blkif_map(be->blkif, ring_ref, evtchn);
> diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c
> index 007db89..911d733 100644
> --- a/drivers/block/xen-blkfront.c
> +++ b/drivers/block/xen-blkfront.c
> @@ -44,6 +44,7 @@
> #include <linux/mutex.h>
> #include <linux/scatterlist.h>
> #include <linux/bitmap.h>
> +#include <linux/llist.h>
>
> #include <xen/xen.h>
> #include <xen/xenbus.h>
> @@ -64,10 +65,17 @@ enum blkif_state {
> BLKIF_STATE_SUSPENDED,
> };
>
> +struct grant {
> + grant_ref_t gref;
> + unsigned long pfn;
> + struct llist_node node;
> +};
> +
> struct blk_shadow {
> struct blkif_request req;
> struct request *request;
> unsigned long frame[BLKIF_MAX_SEGMENTS_PER_REQUEST];
> + struct grant *grants_used[BLKIF_MAX_SEGMENTS_PER_REQUEST];
> };
>
> static DEFINE_MUTEX(blkfront_mutex);
> @@ -97,6 +105,8 @@ struct blkfront_info
> struct work_struct work;
> struct gnttab_free_callback callback;
> struct blk_shadow shadow[BLK_RING_SIZE];
> + struct llist_head persistent_gnts;
> + unsigned int persistent_gnts_c;
> unsigned long shadow_free;
> unsigned int feature_flush;
> unsigned int flush_op;
> @@ -104,6 +114,7 @@ struct blkfront_info
> unsigned int feature_secdiscard:1;
> unsigned int discard_granularity;
> unsigned int discard_alignment;
> + unsigned int feature_persistent:1;
> int is_ready;
> };
>
> @@ -287,21 +298,36 @@ static int blkif_queue_request(struct request *req)
> unsigned long id;
> unsigned int fsect, lsect;
> int i, ref;
> +
> + /*
> + * Used to store if we are able to queue the request by just using
> + * existing persistent grants, or if we have to get new grants,
> + * as there are not sufficiently many free.
> + */
> + bool new_persistent_gnts;
> grant_ref_t gref_head;
> + struct page *granted_page;
> + struct grant *gnt_list_entry = NULL;
> struct scatterlist *sg;
>
> if (unlikely(info->connected != BLKIF_STATE_CONNECTED))
> return 1;
>
> - if (gnttab_alloc_grant_references(
> - BLKIF_MAX_SEGMENTS_PER_REQUEST, &gref_head) < 0) {
> - gnttab_request_free_callback(
> - &info->callback,
> - blkif_restart_queue_callback,
> - info,
> - BLKIF_MAX_SEGMENTS_PER_REQUEST);
> - return 1;
> - }
> + /* Check if we have enought grants to allocate a requests */
> + if (info->persistent_gnts_c < BLKIF_MAX_SEGMENTS_PER_REQUEST) {
> + new_persistent_gnts = 1;
> + if (gnttab_alloc_grant_references(
> + BLKIF_MAX_SEGMENTS_PER_REQUEST - info->persistent_gnts_c,
> + &gref_head) < 0) {
> + gnttab_request_free_callback(
> + &info->callback,
> + blkif_restart_queue_callback,
> + info,
> + BLKIF_MAX_SEGMENTS_PER_REQUEST);
> + return 1;
> + }
> + } else
> + new_persistent_gnts = 0;
>
> /* Fill out a communications ring structure. */
> ring_req = RING_GET_REQUEST(&info->ring, info->ring.req_prod_pvt);
> @@ -341,18 +367,73 @@ static int blkif_queue_request(struct request *req)
> BLKIF_MAX_SEGMENTS_PER_REQUEST);
>
> for_each_sg(info->sg, sg, ring_req->u.rw.nr_segments, i) {
> - buffer_mfn = pfn_to_mfn(page_to_pfn(sg_page(sg)));
> fsect = sg->offset >> 9;
> lsect = fsect + (sg->length >> 9) - 1;
> - /* install a grant reference. */
> - ref = gnttab_claim_grant_reference(&gref_head);
> - BUG_ON(ref == -ENOSPC);
>
> - gnttab_grant_foreign_access_ref(
> - ref,
> + if (info->persistent_gnts_c) {
> + BUG_ON(llist_empty(&info->persistent_gnts));
> + gnt_list_entry = llist_entry(
> + llist_del_first(&info->persistent_gnts),
> + struct grant, node);
> +
> + ref = gnt_list_entry->gref;
> + buffer_mfn = pfn_to_mfn(gnt_list_entry->pfn);
> + info->persistent_gnts_c--;
> + } else {
> + ref = gnttab_claim_grant_reference(&gref_head);
> + BUG_ON(ref == -ENOSPC);
> +
> + gnt_list_entry =
> + kmalloc(sizeof(struct grant),
> + GFP_ATOMIC);
> + if (!gnt_list_entry)
> + return -ENOMEM;
> +
> + granted_page = alloc_page(GFP_ATOMIC);
> + if (!granted_page) {
> + kfree(gnt_list_entry);
> + return -ENOMEM;
> + }
> +
> + gnt_list_entry->pfn =
> + page_to_pfn(granted_page);
> + gnt_list_entry->gref = ref;
> +
> + buffer_mfn = pfn_to_mfn(page_to_pfn(
> + granted_page));
> + gnttab_grant_foreign_access_ref(ref,
> info->xbdev->otherend_id,
> - buffer_mfn,
> - rq_data_dir(req));
> + buffer_mfn, 0);
> + }
> +
> + info->shadow[id].grants_used[i] = gnt_list_entry;
> +
> + if (rq_data_dir(req)) {
> + char *bvec_data;
> + void *shared_data;
> +
> + BUG_ON(sg->offset + sg->length > PAGE_SIZE);
> +
> + shared_data = kmap_atomic(
> + pfn_to_page(gnt_list_entry->pfn));
> + bvec_data = kmap_atomic(sg_page(sg));
> +
> + /*
> + * this does not wipe data stored outside the
> + * range sg->offset..sg->offset+sg->length.
> + * Therefore, blkback *could* see data from
> + * previous requests. This is OK as long as
> + * persistent grants are shared with just one
> + * domain. It may need refactoring if this
> + * changes
> + */
> + memcpy(shared_data + sg->offset,
> + bvec_data + sg->offset,
> + sg->length);
> +
> + kunmap_atomic(bvec_data);
> + kunmap_atomic(shared_data);
> + }
>
> info->shadow[id].frame[i] = mfn_to_pfn(buffer_mfn);
> ring_req->u.rw.seg[i] =
> @@ -368,7 +449,8 @@ static int blkif_queue_request(struct request *req)
> /* Keep a private copy so we can reissue requests when recovering. */
> info->shadow[id].req = *ring_req;
>
> - gnttab_free_grant_references(gref_head);
> + if (new_persistent_gnts)
> + gnttab_free_grant_references(gref_head);
>
> return 0;
> }
> @@ -480,12 +562,13 @@ static int xlvbd_init_blk_queue(struct gendisk *gd, u16 sector_size)
> static void xlvbd_flush(struct blkfront_info *info)
> {
> blk_queue_flush(info->rq, info->feature_flush);
> - printk(KERN_INFO "blkfront: %s: %s: %s\n",
> + printk(KERN_INFO "blkfront: %s: %s: %s %s\n",
> info->gd->disk_name,
> info->flush_op == BLKIF_OP_WRITE_BARRIER ?
> "barrier" : (info->flush_op == BLKIF_OP_FLUSH_DISKCACHE ?
> "flush diskcache" : "barrier or flush"),
> - info->feature_flush ? "enabled" : "disabled");
> + info->feature_flush ? "enabled" : "disabled",
> + info->feature_persistent ? "using persistent grants" : "");
> }
>
> static int xen_translate_vdev(int vdevice, int *minor, unsigned int *offset)
> @@ -707,6 +790,9 @@ static void blkif_restart_queue(struct work_struct *work)
>
> static void blkif_free(struct blkfront_info *info, int suspend)
> {
> + struct llist_node *all_gnts;
> + struct grant *persistent_gnt;
> +
> /* Prevent new requests being issued until we fix things up. */
> spin_lock_irq(&info->io_lock);
> info->connected = suspend ?
> @@ -714,6 +800,17 @@ static void blkif_free(struct blkfront_info *info, int suspend)
> /* No more blkif_request(). */
> if (info->rq)
> blk_stop_queue(info->rq);
> +
> + /* Remove all persistent grants */
> + if (info->persistent_gnts_c) {
> + all_gnts = llist_del_all(&info->persistent_gnts);
> + llist_for_each_entry(persistent_gnt, all_gnts, node) {
> + gnttab_end_foreign_access(persistent_gnt->gref, 0, 0UL);
> + kfree(persistent_gnt);
> + }
> + info->persistent_gnts_c = 0;
> + }
> +
> /* No more gnttab callback work. */
> gnttab_cancel_free_callback(&info->callback);
> spin_unlock_irq(&info->io_lock);
> @@ -734,13 +831,42 @@ static void blkif_free(struct blkfront_info *info, int suspend)
>
> }
>
> -static void blkif_completion(struct blk_shadow *s)
> +static void blkif_completion(struct blk_shadow *s, struct blkfront_info *info,
> + struct blkif_response *bret)
> {
> int i;
> - /* Do not let BLKIF_OP_DISCARD as nr_segment is in the same place
> - * flag. */
> - for (i = 0; i < s->req.u.rw.nr_segments; i++)
> - gnttab_end_foreign_access(s->req.u.rw.seg[i].gref, 0, 0UL);
> + struct bio_vec *bvec;
> + struct req_iterator iter;
> + unsigned long flags;
> + char *bvec_data;
> + void *shared_data;
> + unsigned int offset = 0;
> +
> + if (bret->operation == BLKIF_OP_READ) {
> + /*
> + * Copy the data received from the backend into the bvec.
> + * Since bv_offset can be different than 0, and bv_len different
> + * than PAGE_SIZE, we have to keep track of the current offset,
> + * to be sure we are copying the data from the right shared page.
> + */
> + rq_for_each_segment(bvec, s->request, iter) {
> + BUG_ON((bvec->bv_offset + bvec->bv_len) > PAGE_SIZE);
> + i = offset >> PAGE_SHIFT;
> + shared_data = kmap_atomic(
> + pfn_to_page(s->grants_used[i]->pfn));
> + bvec_data = bvec_kmap_irq(bvec, &flags);
> + memcpy(bvec_data, shared_data + bvec->bv_offset,
> + bvec->bv_len);
> + bvec_kunmap_irq(bvec_data, &flags);
> + kunmap_atomic(shared_data);
> + offset += bvec->bv_len;
> + }
> + }
> + /* Add the persistent grant into the list of free grants */
> + for (i = 0; i < s->req.u.rw.nr_segments; i++) {
> + llist_add(&s->grants_used[i]->node, &info->persistent_gnts);
> + info->persistent_gnts_c++;
> + }
> }
>
> static irqreturn_t blkif_interrupt(int irq, void *dev_id)
> @@ -783,7 +909,7 @@ static irqreturn_t blkif_interrupt(int irq, void *dev_id)
> req = info->shadow[id].request;
>
> if (bret->operation != BLKIF_OP_DISCARD)
> - blkif_completion(&info->shadow[id]);
> + blkif_completion(&info->shadow[id], info, bret);
>
> if (add_id_to_freelist(info, id)) {
> WARN(1, "%s: response to %s (id %ld) couldn't be recycled!\n",
> @@ -942,6 +1068,11 @@ again:
> message = "writing protocol";
> goto abort_transaction;
> }
> + err = xenbus_printf(xbt, dev->nodename,
> + "feature-persistent-grants", "%u", 1);
> + if (err)
> + dev_warn(&dev->dev,
> + "writing persistent grants feature to xenbus");
>
> err = xenbus_transaction_end(xbt, 0);
> if (err) {
> @@ -1029,6 +1160,8 @@ static int blkfront_probe(struct xenbus_device *dev,
> spin_lock_init(&info->io_lock);
> info->xbdev = dev;
> info->vdevice = vdevice;
> + init_llist_head(&info->persistent_gnts);
> + info->persistent_gnts_c = 0;
> info->connected = BLKIF_STATE_DISCONNECTED;
> INIT_WORK(&info->work, blkif_restart_queue);
>
> @@ -1093,7 +1226,7 @@ static int blkif_recover(struct blkfront_info *info)
> req->u.rw.seg[j].gref,
> info->xbdev->otherend_id,
> pfn_to_mfn(info->shadow[req->u.rw.id].frame[j]),
> - rq_data_dir(info->shadow[req->u.rw.id].request));
> + 0);
> }
> info->shadow[req->u.rw.id].req = *req;
>
> @@ -1225,7 +1358,7 @@ static void blkfront_connect(struct blkfront_info *info)
> unsigned long sector_size;
> unsigned int binfo;
> int err;
> - int barrier, flush, discard;
> + int barrier, flush, discard, persistent;
>
> switch (info->connected) {
> case BLKIF_STATE_CONNECTED:
> @@ -1303,6 +1436,14 @@ static void blkfront_connect(struct blkfront_info *info)
> if (!err && discard)
> blkfront_setup_discard(info);
>
> + err = xenbus_gather(XBT_NIL, info->xbdev->otherend,
> + "feature-persistent", "%u", &persistent,
> + NULL);
> + if (err)
> + info->feature_persistent = 0;
> + else
> + info->feature_persistent = persistent;
> +
> err = xlvbd_alloc_gendisk(sectors, info, binfo, sector_size);
> if (err) {
> xenbus_dev_fatal(info->xbdev, err, "xlvbd_add at %s",
> --
> 1.7.7.5 (Apple Git-26)
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v2] Persistent grant maps for xen blk drivers
2012-10-24 16:58 [PATCH v2] Persistent grant maps for xen blk drivers Roger Pau Monne
2012-10-29 13:57 ` Konrad Rzeszutek Wilk
@ 2012-10-30 17:01 ` Konrad Rzeszutek Wilk
2012-10-30 18:33 ` Roger Pau Monné
1 sibling, 1 reply; 13+ messages in thread
From: Konrad Rzeszutek Wilk @ 2012-10-30 17:01 UTC (permalink / raw)
To: Roger Pau Monne; +Cc: xen-devel, Oliver Chick, linux-kernel
On Wed, Oct 24, 2012 at 06:58:45PM +0200, Roger Pau Monne wrote:
> This patch implements persistent grants for the xen-blk{front,back}
> mechanism. The effect of this change is to reduce the number of unmap
> operations performed, since they cause a (costly) TLB shootdown. This
> allows the I/O performance to scale better when a large number of VMs
> are performing I/O.
>
> Previously, the blkfront driver was supplied a bvec[] from the request
> queue. This was granted to dom0; dom0 performed the I/O and wrote
> directly into the grant-mapped memory and unmapped it; blkfront then
> removed foreign access for that grant. The cost of unmapping scales
> badly with the number of CPUs in Dom0. An experiment showed that when
> Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
> ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
> (at which point 650,000 IOPS are being performed in total). If more
> than 5 guests are used, the performance declines. By 10 guests, only
> 400,000 IOPS are being performed.
>
> This patch improves performance by only unmapping when the connection
> between blkfront and back is broken.
>
> On startup blkfront notifies blkback that it is using persistent
> grants, and blkback will do the same. If blkback is not capable of
> persistent mapping, blkfront will still use the same grants, since it
> is compatible with the previous protocol, and simplifies the code
> complexity in blkfront.
>
> To perform a read, in persistent mode, blkfront uses a separate pool
> of pages that it maps to dom0. When a request comes in, blkfront
> transmutes the request so that blkback will write into one of these
> free pages. Blkback keeps note of which grefs it has already
> mapped. When a new ring request comes to blkback, it looks to see if
> it has already mapped that page. If so, it will not map it again. If
> the page hasn't been previously mapped, it is mapped now, and a record
> is kept of this mapping. Blkback proceeds as usual. When blkfront is
> notified that blkback has completed a request, it memcpy's from the
> shared memory, into the bvec supplied. A record that the {gref, page}
> tuple is mapped, and not inflight is kept.
>
> Writes are similar, except that the memcpy is peformed from the
> supplied bvecs, into the shared pages, before the request is put onto
> the ring.
>
> Blkback stores a mapping of grefs=>{page mapped to by gref} in
> a red-black tree. As the grefs are not known apriori, and provide no
> guarantees on their ordering, we have to perform a search
> through this tree to find the page, for every gref we receive. This
> operation takes O(log n) time in the worst case. In blkfront grants
> are stored using a single linked list.
>
> The maximum number of grants that blkback will persistenly map is
> currently set to RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, to
> prevent a malicios guest from attempting a DoS, by supplying fresh
> grefs, causing the Dom0 kernel to map excessively. If a guest
> is using persistent grants and exceeds the maximum number of grants to
> map persistenly the newly passed grefs will be mapped and unmaped.
> Using this approach, we can have requests that mix persistent and
> non-persistent grants, and we need to handle them correctly.
> This allows us to set the maximum number of persistent grants to a
> lower value than RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, although
> setting it will lead to unpredictable performance.
>
> In writing this patch, the question arrises as to if the additional
> cost of performing memcpys in the guest (to/from the pool of granted
> pages) outweigh the gains of not performing TLB shootdowns. The answer
> to that question is `no'. There appears to be very little, if any
> additional cost to the guest of using persistent grants. There is
> perhaps a small saving, from the reduced number of hypercalls
> performed in granting, and ending foreign access.
>
> Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
> Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
> Cc: <konrad.wilk@oracle.com>
> Cc: <linux-kernel@vger.kernel.org>
> ---
> Changes since v1:
> * Changed the unmap_seg array to a bitmap.
> * Only report using persistent grants in blkfront if blkback supports
> it.
> * Reword some comments.
> * Fix a bug when setting the handler, index j was not incremented
> correctly.
> * Check that the tree of grants in blkback is not empty before
> iterating over it when doing the cleanup.
> * Rebase on top of linux-net.
I fixed the 'new_map = [1|0]' you had in and altered it to use 'true'
or 'false', but when running some tests (with a 64-bit PV guest) I got it
to bug.
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 3.7.0-rc3upstream-00220-g37b7153 (konrad@build.dumpdata.com) (gcc version 4.4.4 20100503 (Red Hat 4.4.4-2) (GCC) ) #1 SMP Tue Oct 30 12:15:12 EDT 2012
[ 0.000000] Command line: earlyprintk=xen debug nofb console=tty console=ttyS1,115200n8 xen-pciback.hide=(00:02:00) loglevel=10
[ 0.000000] Freeing 9d-100 pfn range: 99 pages freed
[ 0.000000] 1-1 mapping on 9d->100
[ 0.000000] 1-1 mapping on cf7fb->cfb63
[ 0.000000] 1-1 mapping on cfd15->cfd70
[ 0.000000] 1-1 mapping on cfd71->cfef7
[ 0.000000] 1-1 mapping on cff00->100001
[ 0.000000] Released 99 pages of unused memory
[ 0.000000] Set 198317 page(s) to 1-1 mapping
[ 0.000000] Populating 3e700-3e763 pfn range: 99 pages added
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] Xen: [mem 0x0000000000000000-0x000000000009cfff] usable
[ 0.000000] Xen: [mem 0x000000000009d800-0x00000000000fffff] reserved
[ 0.000000] Xen: [mem 0x0000000000100000-0x000000004d062fff] usable
[ 0.000000] Xen: [mem 0x000000004d063000-0x00000000cf7fafff] unusable
[ 0.000000] Xen: [mem 0x00000000cf7fb000-0x00000000cf95ffff] reserved
[ 0.000000] Xen: [mem 0x00000000cf960000-0x00000000cfb62fff] ACPI NVS
[ 0.000000] Xen: [mem 0x00000000cfb63000-0x00000000cfd14fff] unusable
[ 0.000000] Xen: [mem 0x00000000cfd15000-0x00000000cfd61fff] ACPI NVS
[ 0.000000] Xen: [mem 0x00000000cfd62000-0x00000000cfd6cfff] ACPI data
[ 0.000000] Xen: [mem 0x00000000cfd6d000-0x00000000cfd6ffff] ACPI NVS
[ 0.000000] Xen: [mem 0x00000000cfd70000-0x00000000cfd70fff] unusable
[ 0.000000] Xen: [mem 0x00000000cfd71000-0x00000000cfea8fff] reserved
[ 0.000000] Xen: [mem 0x00000000cfea9000-0x00000000cfeb9fff] ACPI NVS
[ 0.000000] Xen: [mem 0x00000000cfeba000-0x00000000cfecafff] reserved
[ 0.000000] Xen: [mem 0x00000000cfecb000-0x00000000cfecbfff] ACPI NVS
[ 0.000000] Xen: [mem 0x00000000cfecc000-0x00000000cfedbfff] reserved
[ 0.000000] Xen: [mem 0x00000000cfedc000-0x00000000cfedcfff] ACPI NVS
[ 0.000000] Xen: [mem 0x00000000cfedd000-0x00000000cfeddfff] reserved
[ 0.000000] Xen: [mem 0x00000000cfede000-0x00000000cfee3fff] ACPI NVS
[ 0.000000] Xen: [mem 0x00000000cfee4000-0x00000000cfef6fff] reserved
[ 0.000000] Xen: [mem 0x00000000cfef7000-0x00000000cfefffff] unusable
[ 0.000000] Xen: [mem 0x00000000e0000000-0x00000000efffffff] reserved
[ 0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
[ 0.000000] Xen: [mem 0x00000000fec10000-0x00000000fec10fff] reserved
[ 0.000000] Xen: [mem 0x00000000fed00000-0x00000000fed00fff] reserved
[ 0.000000] Xen: [mem 0x00000000fed40000-0x00000000fed44fff] reserved
[ 0.000000] Xen: [mem 0x00000000fed61000-0x00000000fed70fff] reserved
[ 0.000000] Xen: [mem 0x00000000fed80000-0x00000000fed8ffff] reserved
[ 0.000000] Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
[ 0.000000] Xen: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[ 0.000000] Xen: [mem 0x0000000100001000-0x000000020effffff] unusable
[ 0.000000] bootconsole [xenboot0] enabled
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] DMI 2.6 present.
[ 0.000000] DMI: System manufacturer System Product Name/F1A75-M, BIOS 0406 06/11/2011
[ 0.000000] e820: update [mem 0x00000000-0x0000ffff] usable ==> reserved
[ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[ 0.000000] No AGP bridge found
[ 0.000000] e820: last_pfn = 0x4d063 max_arch_pfn = 0x400000000
[ 0.000000] initial memory mapped: [mem 0x00000000-0x16bcdfff]
[ 0.000000] Base memory trampoline at [ffff880000097000] 97000 size 24576
[ 0.000000] init_memory_mapping: [mem 0x00000000-0x4d062fff]
[ 0.000000] [mem 0x00000000-0x4d062fff] page 4k
[ 0.000000] kernel direct mapping tables up to 0x4d062fff @ [mem 0x01e21000-0x0208cfff]
[ 0.000000] xen: setting RW the range 1fd3000 - 208d000
[ 0.000000] RAMDISK: [mem 0x0208d000-0x16bcdfff]
[ 0.000000] ACPI: RSDP 00000000000f0450 00024 (v02 ALASKA)
[ 0.000000] ACPI: XSDT 00000000cfd62068 00054 (v01 ALASKA A M I 01072009 AMI 00010013)
[ 0.000000] ACPI: FACP 00000000cfd69a68 000F4 (v04 ALASKA A M I 01072009 AMI 00010013)
[ 0.000000] ACPI BIOS Bug: Warning: Optional FADT field Pm2ControlBlock has zero address or length: 0x0000000000000000/0x1 (20120913/tbfadt-598)
[ 0.000000] ACPI: DSDT 00000000cfd62150 07917 (v02 ALASKA A M I 00000000 INTL 20051117)
[ 0.000000] ACPI: FACS 00000000cfedef80 00040
[ 0.000000] ACPI: APIC 00000000cfd69b60 00072 (v03 ALASKA A M I 01072009 AMI 00010013)
[ 0.000000] ACPI: MCFG 00000000cfd69bd8 0003C (v01 A M I GMCH945. 01072009 MSFT 00000097)
[ 0.000000] ACPI: HPET 00000000cfd69c18 00038 (v01 ALASKA A M I 01072009 AMI 00000004)
[ 0.000000] ACPI: SSDT 00000000cfd69c50 00FD8 (v01 AMD POWERNOW 00000001 AMD 00000001)
[ 0.000000] ACPI: SSDT 00000000cfd6ac28 01923 (v02 AMD ALIB 00000001 MSFT 04000000)
[ 0.000000] ACPI: Local APIC address 0xfee00000
[ 0.000000] NUMA turned off
[ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000004d062fff]
[ 0.000000] Initmem setup node 0 [mem 0x00000000-0x4d062fff]
[ 0.000000] NODE_DATA [mem 0x3e75f000-0x3e762fff]
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x00010000-0x00ffffff]
[ 0.000000] DMA32 [mem 0x01000000-0xffffffff]
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x00010000-0x0009cfff]
[ 0.000000] node 0: [mem 0x00100000-0x4d062fff]
[ 0.000000] On node 0 totalpages: 315376
[ 0.000000] DMA zone: 56 pages used for memmap
[ 0.000000] DMA zone: 6 pages reserved
[ 0.000000] DMA zone: 3919 pages, LIFO batch:0
[ 0.000000] DMA32 zone: 4258 pages used for memmap
[ 0.000000] DMA32 zone: 307137 pages, LIFO batch:31
[ 0.000000] ACPI: PM-Timer IO Port: 0x808
[ 0.000000] ACPI: Local APIC address 0xfee00000
[ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
[ 0.000000] ACPI: IOAPIC (id[0x05] address[0xfec00000] gsi_base[0])
[ 0.000000] IOAPIC[0]: apic_id 5, version 33, address 0xfec00000, GSI 0-23
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
[ 0.000000] ACPI: IRQ0 used by override.
[ 0.000000] ACPI: IRQ2 used by override.
[ 0.000000] ACPI: IRQ9 used by override.
[ 0.000000] Using ACPI (MADT) for SMP configuration information
[ 0.000000] ACPI: HPET id: 0xffffffff base: 0xfed00000
[ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs
[ 0.000000] nr_irqs_gsi: 40
[ 0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000
[ 0.000000] PM: Registered nosave memory: 000000000009e000 - 0000000000100000
[ 0.000000] e820: [mem 0xcff00000-0xdfffffff] available for PCI devices
[ 0.000000] Booting paravirtualized kernel on Xen
[ 0.000000] Xen version: 4.1.4-pre (preserve-AD)
[ 0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:4 nr_node_ids:1
[ 0.000000] PERCPU: Embedded 28 pages/cpu @ffff88003e000000 s84288 r8192 d22208 u524288
[ 0.000000] pcpu-alloc: s84288 r8192 d22208 u524288 alloc=1*2097152
[ 0.000000] pcpu-alloc: [0] 0 1 2 3
[ 2.763208] Built 1 zonelists in Node order, mobility grouping on. Total pages: 311056
[ 2.763213] Policy zone: DMA32
[ 2.763218] Kernel command line: earlyprintk=xen debug nofb console=tty console=ttyS1,115200n8 xen-pciback.hide=(00:02:00) loglevel=10
[ 2.763656] PID hash table entries: 4096 (order: 3, 32768 bytes)
[ 2.763663] __ex_table already sorted, skipping sort
[ 2.808064] software IO TLB [mem 0x38a00000-0x3ca00000] (64MB) mapped at [ffff880038a00000-ffff88003c9fffff]
[ 2.811300] Memory: 581728k/1261964k available (6413k kernel code, 460k absent, 679776k reserved, 4478k data, 752k init)
[ 2.811414] Hierarchical RCU implementation.
[ 2.811419] RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=3.
[ 2.811432] NR_IRQS:33024 nr_irqs:704 16
[ 2.811514] xen: sci override: global_irq=9 trigger=0 polarity=1
[ 2.811518] xen: registering gsi 9 triggering 0 polarity 1
[ 2.811531] xen: --> pirq=9 -> irq=9 (gsi=9)
[ 2.811539] xen: acpi sci 9
[ 2.811546] xen: --> pirq=1 -> irq=1 (gsi=1)
[ 2.811551] xen: --> pirq=2 -> irq=2 (gsi=2)
[ 2.811557] xen: --> pirq=3 -> irq=3 (gsi=3)
[ 2.811562] xen: --> pirq=4 -> irq=4 (gsi=4)
[ 2.811568] xen: --> pirq=5 -> irq=5 (gsi=5)
[ 2.811574] xen: --> pirq=6 -> irq=6 (gsi=6)
[ 2.811579] xen: --> pirq=7 -> irq=7 (gsi=7)
[ 2.811585] xen: --> pirq=8 -> irq=8 (gsi=8)
[ 2.811590] xen: --> pirq=10 -> irq=10 (gsi=10)
[ 2.811596] xen: --> pirq=11 -> irq=11 (gsi=11)
[ 2.811602] xen: --> pirq=12 -> irq=12 (gsi=12)
[ 2.811607] xen: --> pirq=13 -> irq=13 (gsi=13)
[ 2.811613] xen: --> pirq=14 -> irq=14 (gsi=14)
[ 2.811618] xen: --> pirq=15 -> irq=15 (gsi=15)
[ 2.813454] Console: colour VGA+ 80x25
[ 2.818363] console [tty0] enabled
[ 2.818422] console [ttyS1] enabled, bootconsole disabled
[ 2.818757] Xen: using vcpuop timer interface
[ 2.819017] installing Xen timer for CPU 0
[ 2.819285] tsc: Detected 2899.980 MHz processor
[ 2.819555] Calibrating delay loop (skipped), value calculated using timer frequency.. 5799.96 BogoMIPS (lpj=2899980)
[ 2.820151] pid_max: default: 32768 minimum: 301
[ 2.820479] Security Framework initialized
[ 2.820728] SELinux: Initializing.
[ 2.820948] SELinux: Starting in permissive mode
[ 2.821532] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
[ 2.822565] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
[ 2.823519] Mount-cache hash table entries: 256
[ 2.824126] Initializing cgroup subsys cpuacct
[ 2.824400] Initializing cgroup subsys freezer
[ 2.824730] tseg: 00cff00000
[ 2.824934] CPU: Physical Processor ID: 0
[ 2.825186] CPU: Processor Core ID: 0
[ 2.825417] mce: CPU supports 6 MCE banks
[ 2.825696] Last level iTLB entries: 4KB 512, 2MB 16, 4MB 8
[ 2.825696] Last level dTLB entries: 4KB 1024, 2MB 128, 4MB 64
[ 2.825696] tlb_flushall_shift: 5
[ 2.826625] Freeing SMP alternatives: 24k freed
[ 2.829704] ACPI: Core revision 20120913
[ 2.856787] cpu 0 spinlock event irq 41
[ 2.857062] Performance Events: Broken PMU hardware detected, using software events only.
[ 2.857602] Failed to access perfctr msr (MSR c0010004 is 0)
[ 2.858234] MCE: In-kernel MCE decoding enabled.
[ 2.858603] NMI watchdog: disabled (cpu0): hardware events not enabled
[ 2.859223] installing Xen timer for CPU 1
[ 2.859499] cpu 1 spinlock event irq 48
[ 2.860218] installing Xen timer for CPU 2
[ 2.860493] cpu 2 spinlock event irq 55
[ 2.860951] Brought up 3 CPUs
[ 2.864412] PM: Registering ACPI NVS region [mem 0xcf960000-0xcfb62fff] (2109440 bytes)
[ 2.865989] PM: Registering ACPI NVS region [mem 0xcfd15000-0xcfd61fff] (315392 bytes)
[ 2.866464] PM: Registering ACPI NVS region [mem 0xcfd6d000-0xcfd6ffff] (12288 bytes)
[ 2.866931] PM: Registering ACPI NVS region [mem 0xcfea9000-0xcfeb9fff] (69632 bytes)
[ 2.867404] PM: Registering ACPI NVS region [mem 0xcfecb000-0xcfecbfff] (4096 bytes)
[ 2.867861] PM: Registering ACPI NVS region [mem 0xcfedc000-0xcfedcfff] (4096 bytes)
[ 2.868321] PM: Registering ACPI NVS region [mem 0xcfede000-0xcfee3fff] (24576 bytes)
[ 2.869081] kworker/u:0 (26) used greatest stack depth: 6120 bytes left
[ 2.869105] Grant tables using version 2 layout.
[ 2.869122] Grant table initialized
[ 2.869164] RTC time: 16:43:55, date: 10/30/12
[ 2.870366] NET: Registered protocol family 16
[ 2.871181] kworker/u:0 (30) used greatest stack depth: 5504 bytes left
[ 2.872440] ACPI: bus type pci registered
[ 2.873480] dca service started, version 1.12.1
[ 2.873891] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000)
[ 2.874438] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in E820
[ 2.914335] PCI: Using configuration type 1 for base access
[ 2.932999] bio: create slab <bio-0> at 0
[ 2.933584] ACPI: Added _OSI(Module Device)
[ 2.933866] ACPI: Added _OSI(Processor Device)
[ 2.934145] ACPI: Added _OSI(3.0 _SCP Extensions)
[ 2.934432] ACPI: Added _OSI(Processor Aggregator Device)
[ 2.942011] ACPI: EC: Look up EC in DSDT
[ 2.950054] ACPI: Executed 1 blocks of module-level executable AML code
[ 2.956530] ACPI: Interpreter enabled
[ 2.956772] ACPI: (supports S0 S3 S4 S5)
[ 2.957218] ACPI: Using IOAPIC for interrupt routing
[ 2.982389] ACPI: No dock devices found.
[ 2.982650] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[ 2.983552] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[ 2.984298] PCI host bridge to bus 0000:00
[ 2.984571] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 2.984906] pci_bus 0000:00: root bus resource [io 0x0000-0x03af]
[ 2.985277] pci_bus 0000:00: root bus resource [io 0x03e0-0x0cf7]
[ 2.985657] pci_bus 0000:00: root bus resource [io 0x03b0-0x03df]
[ 2.986029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff]
[ 2.986399] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff]
[ 2.986810] pci_bus 0000:00: root bus resource [mem 0x000c0000-0x000dffff]
[ 2.987214] pci_bus 0000:00: root bus resource [mem 0xd0000000-0xffffffff]
[ 2.987631] pci 0000:00:00.0: [1022:1705] type 00 class 0x060000
[ 2.988106] pci 0000:00:01.0: [1002:9640] type 00 class 0x030000
[ 2.988487] pci 0000:00:01.0: reg 10: [mem 0xd0000000-0xdfffffff pref]
[ 2.988893] pci 0000:00:01.0: reg 14: [io 0xf000-0xf0ff]
[ 2.989246] pci 0000:00:01.0: reg 18: [mem 0xfeb00000-0xfeb3ffff]
[ 2.989743] pci 0000:00:01.0: supports D1 D2
[ 2.990053] pci 0000:00:01.1: [1002:1714] type 00 class 0x040300
[ 2.990442] pci 0000:00:01.1: reg 10: [mem 0xfeb44000-0xfeb47fff]
[ 2.990965] pci 0000:00:01.1: supports D1 D2
[ 2.991343] pci 0000:00:10.0: [1022:7812] type 00 class 0x0c0330
[ 2.991752] pci 0000:00:10.0: reg 10: [mem 0xfeb4a000-0xfeb4bfff 64bit]
[ 2.992355] pci 0000:00:10.0: PME# supported from D0 D3hot D3cold
[ 2.992803] pci 0000:00:10.1: [1022:7812] type 00 class 0x0c0330
[ 2.993201] pci 0000:00:10.1: reg 10: [mem 0xfeb48000-0xfeb49fff 64bit]
[ 2.993793] pci 0000:00:10.1: PME# supported from D0 D3hot D3cold
[ 2.994239] pci 0000:00:11.0: [1022:7801] type 00 class 0x010601
[ 2.994637] pci 0000:00:11.0: reg 10: [io 0xf140-0xf147]
[ 2.994982] pci 0000:00:11.0: reg 14: [io 0xf130-0xf133]
[ 2.995333] pci 0000:00:11.0: reg 18: [io 0xf120-0xf127]
[ 2.995678] pci 0000:00:11.0: reg 1c: [io 0xf110-0xf113]
[ 2.996024] pci 0000:00:11.0: reg 20: [io 0xf100-0xf10f]
[ 2.996374] pci 0000:00:11.0: reg 24: [mem 0xfeb51000-0xfeb517ff]
[ 2.996850] pci 0000:00:12.0: [1022:7807] type 00 class 0x0c0310
[ 2.997235] pci 0000:00:12.0: reg 10: [mem 0xfeb50000-0xfeb50fff]
[ 2.997765] pci 0000:00:12.2: [1022:7808] type 00 class 0x0c0320
[ 2.998161] pci 0000:00:12.2: reg 10: [mem 0xfeb4f000-0xfeb4f0ff]
[ 2.998712] pci 0000:00:12.2: supports D1 D2
[ 2.998977] pci 0000:00:12.2: PME# supported from D0 D1 D2 D3hot
[ 2.999375] pci 0000:00:13.0: [1022:7807] type 00 class 0x0c0310
[ 2.999768] pci 0000:00:13.0: reg 10: [mem 0xfeb4e000-0xfeb4efff]
[ 3.000283] pci 0000:00:13.2: [1022:7808] type 00 class 0x0c0320
[ 3.000681] pci 0000:00:13.2: reg 10: [mem 0xfeb4d000-0xfeb4d0ff]
[ 3.001227] pci 0000:00:13.2: supports D1 D2
[ 3.001495] pci 0000:00:13.2: PME# supported from D0 D1 D2 D3hot
[ 3.001902] pci 0000:00:14.0: [1022:780b] type 00 class 0x0c0500
[ 3.002433] pci 0000:00:14.2: [1022:780d] type 00 class 0x040300
[ 3.002840] pci 0000:00:14.2: reg 10: [mem 0xfeb40000-0xfeb43fff 64bit]
[ 3.003376] pci 0000:00:14.2: PME# supported from D0 D3hot D3cold
[ 3.003761] pci 0000:00:14.3: [1022:780e] type 00 class 0x060100
[ 3.004279] pci 0000:00:14.4: [1022:780f] type 01 class 0x060401
[ 3.004730] pci 0000:00:14.5: [1022:7809] type 00 class 0x0c0310
[ 3.005119] pci 0000:00:14.5: reg 10: [mem 0xfeb4c000-0xfeb4cfff]
[ 3.005641] pci 0000:00:15.0: [1022:43a0] type 01 class 0x060400
[ 3.006187] pci 0000:00:15.0: supports D1 D2
[ 3.006515] pci 0000:00:15.1: [1022:43a1] type 01 class 0x060400
[ 3.007059] pci 0000:00:15.1: supports D1 D2
[ 3.007403] pci 0000:00:18.0: [1022:1700] type 00 class 0x060000
[ 3.007868] pci 0000:00:18.1: [1022:1701] type 00 class 0x060000
[ 3.008320] pci 0000:00:18.2: [1022:1702] type 00 class 0x060000
[ 3.008778] pci 0000:00:18.3: [1022:1703] type 00 class 0x060000
[ 3.009271] pci 0000:00:18.4: [1022:1704] type 00 class 0x060000
[ 3.009716] pci 0000:00:18.5: [1022:1718] type 00 class 0x060000
[ 3.010167] pci 0000:00:18.6: [1022:1716] type 00 class 0x060000
[ 3.010615] pci 0000:00:18.7: [1022:1719] type 00 class 0x060000
[ 3.011117] pci 0000:01:05.0: [9710:9835] type 00 class 0x070002
[ 3.011513] pci 0000:01:05.0: reg 10: [io 0xe050-0xe057]
[ 3.011861] pci 0000:01:05.0: reg 14: [io 0xe040-0xe047]
[ 3.012210] pci 0000:01:05.0: reg 18: [io 0xe030-0xe037]
[ 3.012563] pci 0000:01:05.0: reg 1c: [io 0xe020-0xe027]
[ 3.012912] pci 0000:01:05.0: reg 20: [io 0xe010-0xe017]
[ 3.013259] pci 0000:01:05.0: reg 24: [io 0xe000-0xe00f]
[ 3.013710] pci 0000:00:14.4: PCI bridge to [bus 01] (subtractive decode)
[ 3.014118] pci 0000:00:14.4: bridge window [io 0xe000-0xefff]
[ 3.014496] pci 0000:00:14.4: bridge window [io 0x0000-0x03af] (subtractive decode)
[ 3.014967] pci 0000:00:14.4: bridge window [io 0x03e0-0x0cf7] (subtractive decode)
[ 3.015422] pci 0000:00:14.4: bridge window [io 0x03b0-0x03df] (subtractive decode)
[ 3.015878] pci 0000:00:14.4: bridge window [io 0x0d00-0xffff] (subtractive decode)
[ 3.016343] pci 0000:00:14.4: bridge window [mem 0x000a0000-0x000bffff] (subtractive decode)
[ 3.016836] pci 0000:00:14.4: bridge window [mem 0x000c0000-0x000dffff] (subtractive decode)
[ 3.017326] pci 0000:00:14.4: bridge window [mem 0xd0000000-0xffffffff] (subtractive decode)
[ 3.017990] pci 0000:02:00.0: [8086:10d3] type 00 class 0x020000
[ 3.018379] pci 0000:02:00.0: reg 10: [mem 0xfeac0000-0xfeadffff]
[ 3.018769] pci 0000:02:00.0: reg 14: [mem 0xfea00000-0xfea7ffff]
[ 3.019167] pci 0000:02:00.0: reg 18: [io 0xd000-0xd01f]
[ 3.019522] pci 0000:02:00.0: reg 1c: [mem 0xfeae0000-0xfeae3fff]
[ 3.019964] pci 0000:02:00.0: reg 30: [mem 0xfea80000-0xfeabffff pref]
[ 3.020508] pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
[ 3.023962] pci 0000:00:15.0: PCI bridge to [bus 02]
[ 3.024307] pci 0000:00:15.0: bridge window [io 0xd000-0xdfff]
[ 3.024689] pci 0000:00:15.0: bridge window [mem 0xfea00000-0xfeafffff]
[ 3.025293] pci 0000:03:00.0: [1969:1083] type 00 class 0x020000
[ 3.025712] pci 0000:03:00.0: reg 10: [mem 0xfe900000-0xfe93ffff 64bit]
[ 3.026139] pci 0000:03:00.0: reg 18: [io 0xc000-0xc07f]
[ 3.026691] pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 3.028949] pci 0000:00:15.1: PCI bridge to [bus 03]
[ 3.029290] pci 0000:00:15.1: bridge window [io 0xc000-0xcfff]
[ 3.029671] pci 0000:00:15.1: bridge window [mem 0xfe900000-0xfe9fffff]
[ 3.030148] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
[ 3.030844] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PE20._PRT]
[ 3.031280] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PE21._PRT]
[ 3.031782] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0PC._PRT]
[ 3.032377] pci0000:00: Requesting ACPI _OSC control (0x1d)
[ 3.032947] pci0000:00: ACPI _OSC control (0x1d) granted
[ 3.051344] ACPI: PCI Interrupt Link [LN24] (IRQs *24)
[ 3.051852] ACPI: PCI Interrupt Link [LN25] (IRQs *25)
[ 3.052329] ACPI: PCI Interrupt Link [LN26] (IRQs *26)
[ 3.052821] ACPI: PCI Interrupt Link [LN27] (IRQs *27)
[ 3.053298] ACPI: PCI Interrupt Link [LN28] (IRQs *28)
[ 3.053787] ACPI: PCI Interrupt Link [LN29] (IRQs *29)
[ 3.054271] ACPI: PCI Interrupt Link [LN30] (IRQs *30)
[ 3.054747] ACPI: PCI Interrupt Link [LN31] (IRQs *31)
[ 3.055243] ACPI: PCI Interrupt Link [LN32] (IRQs *32)
[ 3.055724] ACPI: PCI Interrupt Link [LN33] (IRQs *33)
[ 3.056211] ACPI: PCI Interrupt Link [LN34] (IRQs *34)
[ 3.056692] ACPI: PCI Interrupt Link [LN35] (IRQs *35)
[ 3.057187] ACPI: PCI Interrupt Link [LN36] (IRQs *36)
[ 3.057673] ACPI: PCI Interrupt Link [LN37] (IRQs *37)
[ 3.058165] ACPI: PCI Interrupt Link [LN38] (IRQs *38)
[ 3.058660] ACPI: PCI Interrupt Link [LN39] (IRQs *39)
[ 3.059154] ACPI: PCI Interrupt Link [LN40] (IRQs *40)
[ 3.059643] ACPI: PCI Interrupt Link [LN41] (IRQs *41)
[ 3.060128] ACPI: PCI Interrupt Link [LN42] (IRQs *42)
[ 3.060618] ACPI: PCI Interrupt Link [LN43] (IRQs *43)
[ 3.061095] ACPI: PCI Interrupt Link [LN44] (IRQs *44)
[ 3.061575] ACPI: PCI Interrupt Link [LN45] (IRQs *45)
[ 3.062059] ACPI: PCI Interrupt Link [LN46] (IRQs *46)
[ 3.062543] ACPI: PCI Interrupt Link [LN47] (IRQs *47)
[ 3.063029] ACPI: PCI Interrupt Link [LN48] (IRQs *48)
[ 3.063506] ACPI: PCI Interrupt Link [LN49] (IRQs *49)
[ 3.063989] ACPI: PCI Interrupt Link [LN50] (IRQs *50)
[ 3.064473] ACPI: PCI Interrupt Link [LN51] (IRQs *51)
[ 3.064950] ACPI: PCI Interrupt Link [LN52] (IRQs *52)
[ 3.065440] ACPI: PCI Interrupt Link [LN53] (IRQs *53)
[ 3.065918] ACPI: PCI Interrupt Link [LN54] (IRQs *54)
[ 3.066402] ACPI: PCI Interrupt Link [LN55] (IRQs *55)
[ 3.066889] ACPI: PCI Interrupt Link [LNKA] (IRQs 4 5 7 10 11 14 15) *0
[ 3.068826] ACPI: PCI Interrupt Link [LNKB] (IRQs 4 5 7 10 11 14 15) *0
[ 3.069717] ACPI: PCI Interrupt Link [LNKC] (IRQs 4 5 7 10 11 14 15) *0
[ 3.070606] ACPI: PCI Interrupt Link [LNKD] (IRQs 4 5 7 10 11 14 15) *0
[ 3.071479] ACPI: PCI Interrupt Link [LNKE] (IRQs 4 5 7 10 11 14 15) *0
[ 3.072339] ACPI: PCI Interrupt Link [LNKF] (IRQs 4 5 7 10 11 14 15) *0
[ 3.073195] ACPI: PCI Interrupt Link [LNKG] (IRQs 4 5 7 10 11 14 15) *0
[ 3.074064] ACPI: PCI Interrupt Link [LNKH] (IRQs 4 5 7 10 11 14 15) *0
[ 3.075123] xen/balloon: Initialising balloon driver.
[ 3.076280] xen-balloon: Initialising balloon driver.
[ 3.076885] xen/balloon: Xen selfballooning driver disabled for domain0.
[ 3.077545] vgaarb: device added: PCI:0000:00:01.0,decodes=io+mem,owns=io+mem,locks=none
[ 3.078080] vgaarb: loaded
[ 3.078268] vgaarb: bridge control possible 0000:00:01.0
[ 3.078970] ACPI: bus type usb registered
[ 3.079409] usbcore: registered new interface driver usbfs
[ 3.079813] usbcore: registered new interface driver hub
[ 3.080257] usbcore: registered new device driver usb
[ 3.081098] PCI: Using ACPI for IRQ routing
[ 3.098106] PCI: pci_cache_line_size set to 64 bytes
[ 3.098606] e820: reserve RAM buffer [mem 0x0009d000-0x0009ffff]
[ 3.098972] e820: reserve RAM buffer [mem 0x4d063000-0x4fffffff]
[ 3.099810] NetLabel: Initializing
[ 3.100040] NetLabel: domain hash size = 128
[ 3.100313] NetLabel: protocols = UNLABELED CIPSOv4
[ 3.100639] NetLabel: unlabeled traffic allowed by default
[ 3.101361] Switching to clocksource xen
[ 3.109135] pnp: PnP ACPI init
[ 3.109362] ACPI: bus type pnp registered
[ 3.109832] pnp 00:00: [bus 00-ff]
[ 3.110058] pnp 00:00: [io 0x0cf8-0x0cff]
[ 3.110319] pnp 00:00: [io 0x0000-0x03af window]
[ 3.110611] pnp 00:00: [io 0x03e0-0x0cf7 window]
[ 3.110912] pnp 00:00: [io 0x03b0-0x03df window]
[ 3.111200] pnp 00:00: [io 0x0d00-0xffff window]
[ 3.111492] pnp 00:00: [mem 0x000a0000-0x000bffff window]
[ 3.111826] pnp 00:00: [mem 0x000c0000-0x000dffff window]
[ 3.112154] pnp 00:00: [mem 0xd0000000-0xffffffff window]
[ 3.112482] pnp 00:00: [mem 0x00000000 window]
[ 3.113097] pnp 00:00: Plug and Play ACPI device, IDs PNP0a03 (active)
[ 3.113537] pnp 00:01: [mem 0xe0000000-0xefffffff]
[ 3.114213] system 00:01: [mem 0xe0000000-0xefffffff] has been reserved
[ 3.114626] system 00:01: Plug and Play ACPI device, IDs PNP0c01 (active)
[ 3.116690] pnp 00:02: [io 0x0010-0x001f]
[ 3.116955] pnp 00:02: [io 0x0022-0x003f]
[ 3.117210] pnp 00:02: [io 0x0063]
[ 3.117439] pnp 00:02: [io 0x0065]
[ 3.117667] pnp 00:02: [io 0x0067-0x006f]
[ 3.117932] pnp 00:02: [io 0x0072-0x007f]
[ 3.118191] pnp 00:02: [io 0x0080]
[ 3.118415] pnp 00:02: [io 0x0084-0x0086]
[ 3.118668] pnp 00:02: [io 0x0088]
[ 3.118910] pnp 00:02: [io 0x008c-0x008e]
[ 3.119174] pnp 00:02: [io 0x0090-0x009f]
[ 3.119436] pnp 00:02: [io 0x00a2-0x00bf]
[ 3.119706] pnp 00:02: [io 0x00b1]
[ 3.119936] pnp 00:02: [io 0x00e0-0x00ef]
[ 3.120194] pnp 00:02: [io 0x04d0-0x04d1]
[ 3.120455] pnp 00:02: [io 0x040b]
[ 3.120683] pnp 00:02: [io 0x04d6]
[ 3.120916] pnp 00:02: [io 0x0c00-0x0c01]
[ 3.121172] pnp 00:02: [io 0x0c14]
[ 3.121397] pnp 00:02: [io 0x0c50-0x0c51]
[ 3.121653] pnp 00:02: [io 0x0c52]
[ 3.121885] pnp 00:02: [io 0x0c6c]
[ 3.122109] pnp 00:02: [io 0x0c6f]
[ 3.122335] pnp 00:02: [io 0x0cd0-0x0cd1]
[ 3.122591] pnp 00:02: [io 0x0cd2-0x0cd3]
[ 3.122860] pnp 00:02: [io 0x0cd4-0x0cd5]
[ 3.123120] pnp 00:02: [io 0x0cd6-0x0cd7]
[ 3.123377] pnp 00:02: [io 0x0cd8-0x0cdf]
[ 3.123633] pnp 00:02: [io 0x0800-0x089f]
[ 3.123898] pnp 00:02: [io 0x0000-0xffffffffffffffff disabled]
[ 3.124249] pnp 00:02: [io 0x0000-0x000f]
[ 3.124506] pnp 00:02: [io 0x0b20-0x0b3f]
[ 3.124768] pnp 00:02: [io 0x0900-0x090f]
[ 3.125026] pnp 00:02: [io 0x0910-0x091f]
[ 3.125282] pnp 00:02: [io 0xfe00-0xfefe]
[ 3.125537] pnp 00:02: [io 0x0060-0x005f disabled]
[ 3.125841] pnp 00:02: [io 0x0064-0x0063 disabled]
[ 3.126136] pnp 00:02: [mem 0xfec00000-0xfec00fff]
[ 3.126426] pnp 00:02: [mem 0xfee00000-0xfee00fff]
[ 3.126724] pnp 00:02: [mem 0xfed80000-0xfed8ffff]
[ 3.127024] pnp 00:02: [mem 0xfed61000-0xfed70fff]
[ 3.127321] pnp 00:02: [mem 0xfec10000-0xfec10fff]
[ 3.127619] pnp 00:02: [mem 0xfed00000-0xfed00fff]
[ 3.127920] pnp 00:02: [mem 0xff000000-0xffffffff]
[ 3.128629] system 00:02: [io 0x04d0-0x04d1] has been reserved
[ 3.129033] system 00:02: [io 0x040b] has been reserved
[ 3.129358] system 00:02: [io 0x04d6] has been reserved
[ 3.129680] system 00:02: [io 0x0c00-0x0c01] has been reserved
[ 3.130042] system 00:02: [io 0x0c14] has been reserved
[ 3.130365] system 00:02: [io 0x0c50-0x0c51] has been reserved
[ 3.130727] system 00:02: [io 0x0c52] has been reserved
[ 3.131046] system 00:02: [io 0x0c6c] has been reserved
[ 3.131369] system 00:02: [io 0x0c6f] has been reserved
[ 3.131692] system 00:02: [io 0x0cd0-0x0cd1] has been reserved
[ 3.132047] system 00:02: [io 0x0cd2-0x0cd3] has been reserved
[ 3.132401] system 00:02: [io 0x0cd4-0x0cd5] has been reserved
[ 3.132756] system 00:02: [io 0x0cd6-0x0cd7] has been reserved
[ 3.133111] system 00:02: [io 0x0cd8-0x0cdf] has been reserved
[ 3.133460] system 00:02: [io 0x0800-0x089f] has been reserved
[ 3.133814] system 00:02: [io 0x0b20-0x0b3f] has been reserved
[ 3.134168] system 00:02: [io 0x0900-0x090f] has been reserved
[ 3.134524] system 00:02: [io 0x0910-0x091f] has been reserved
[ 3.134888] system 00:02: [io 0xfe00-0xfefe] has been reserved
[ 3.135250] system 00:02: [mem 0xfec00000-0xfec00fff] could not be reserved
[ 3.135651] system 00:02: [mem 0xfee00000-0xfee00fff] has been reserved
[ 3.136053] system 00:02: [mem 0xfed80000-0xfed8ffff] has been reserved
[ 3.136437] system 00:02: [mem 0xfed61000-0xfed70fff] has been reserved
[ 3.136829] system 00:02: [mem 0xfec10000-0xfec10fff] has been reserved
[ 3.137219] system 00:02: [mem 0xfed00000-0xfed00fff] has been reserved
[ 3.137608] system 00:02: [mem 0xff000000-0xffffffff] has been reserved
[ 3.138015] system 00:02: Plug and Play ACPI device, IDs PNP0c02 (active)
[ 3.138611] pnp 00:03: [io 0x0000-0xffffffffffffffff disabled]
[ 3.138981] pnp 00:03: [io 0x0300-0x031f]
[ 3.139249] pnp 00:03: [io 0x0290-0x029f]
[ 3.139512] pnp 00:03: [io 0x0230-0x023f]
[ 3.140070] system 00:03: [io 0x0300-0x031f] has been reserved
[ 3.140431] system 00:03: [io 0x0290-0x029f] has been reserved
[ 3.140800] system 00:03: [io 0x0230-0x023f] has been reserved
[ 3.141165] system 00:03: Plug and Play ACPI device, IDs PNP0c02 (active)
[ 3.141595] pnp 00:04: [dma 4]
[ 3.141809] pnp 00:04: [io 0x0000-0x000f]
[ 3.142070] pnp 00:04: [io 0x0081-0x0083]
[ 3.142330] pnp 00:04: [io 0x0087]
[ 3.142557] pnp 00:04: [io 0x0089-0x008b]
[ 3.142825] pnp 00:04: [io 0x008f]
[ 3.143052] pnp 00:04: [io 0x00c0-0x00df]
[ 3.143532] pnp 00:04: Plug and Play ACPI device, IDs PNP0200 (active)
[ 3.143961] pnp 00:05: [io 0x0070-0x0071]
[ 3.144222] xen: registering gsi 8 triggering 1 polarity 0
[ 3.144562] pnp 00:05: [irq 8]
[ 3.145001] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active)
[ 3.145408] pnp 00:06: [io 0x0061]
[ 3.145932] pnp 00:06: Plug and Play ACPI device, IDs PNP0800 (active)
[ 3.146422] pnp 00:07: [io 0x0010-0x001f]
[ 3.146682] pnp 00:07: [io 0x0022-0x003f]
[ 3.146977] pnp 00:07: [io 0x0044-0x005f]
[ 3.147238] pnp 00:07: [io 0x0072-0x007f]
[ 3.147491] pnp 00:07: [io 0x0080]
[ 3.147722] pnp 00:07: [io 0x0084-0x0086]
[ 3.147980] pnp 00:07: [io 0x0088]
[ 3.148211] pnp 00:07: [io 0x008c-0x008e]
[ 3.148469] pnp 00:07: [io 0x0090-0x009f]
[ 3.148734] pnp 00:07: [io 0x00a2-0x00bf]
[ 3.148988] pnp 00:07: [io 0x00e0-0x00ef]
[ 3.149250] pnp 00:07: [io 0x04d0-0x04d1]
[ 3.149697] system 00:07: [io 0x04d0-0x04d1] has been reserved
[ 3.150061] system 00:07: Plug and Play ACPI device, IDs PNP0c02 (active)
[ 3.150478] pnp 00:08: [io 0x00f0-0x00ff]
[ 3.150755] xen: registering gsi 13 triggering 1 polarity 0
[ 3.151101] pnp 00:08: [irq 13]
[ 3.151448] pnp 00:08: Plug and Play ACPI device, IDs PNP0c04 (active)
[ 3.152100] system 00:09: Plug and Play ACPI device, IDs PNP0c02 (active)
[ 3.152778] pnp 00:0a: [io 0x03f8-0x03ff]
[ 3.153037] xen: registering gsi 4 triggering 1 polarity 0
[ 3.153376] pnp 00:0a: [irq 4]
[ 3.153581] pnp 00:0a: [dma 0 disabled]
[ 3.153998] pnp 00:0a: Plug and Play ACPI device, IDs PNP0501 (active)
[ 3.155394] pnp 00:0b: [mem 0xfed00000-0xfed003ff]
[ 3.155978] pnp 00:0b: Plug and Play ACPI device, IDs PNP0103 (active)
[ 3.156388] pnp: PnP ACPI: found 12 devices
[ 3.156653] ACPI: ACPI bus type pnp unregistered
[ 3.156953] xen-pciback: Error parsing pci_devs_to_hide at "(00:02:00)"
[ 3.170459] PM-Timer failed consistency check (0x0xffffff) - aborting.
[ 3.170947] pci 0000:00:14.4: PCI bridge to [bus 01]
[ 3.171261] pci 0000:00:14.4: bridge window [io 0xe000-0xefff]
[ 3.171655] pci 0000:00:15.0: PCI bridge to [bus 02]
[ 3.171974] pci 0000:00:15.0: bridge window [io 0xd000-0xdfff]
[ 3.172345] pci 0000:00:15.0: bridge window [mem 0xfea00000-0xfeafffff]
[ 3.172762] pci 0000:00:15.1: PCI bridge to [bus 03]
[ 3.173067] pci 0000:00:15.1: bridge window [io 0xc000-0xcfff]
[ 3.173436] pci 0000:00:15.1: bridge window [mem 0xfe900000-0xfe9fffff]
[ 3.173889] xen: registering gsi 16 triggering 0 polarity 1
[ 3.174247] xen: --> pirq=16 -> irq=16 (gsi=16)
[ 3.174547] xen: registering gsi 16 triggering 0 polarity 1
[ 3.174894] Already setup the GSI :16
[ 3.175131] pci_bus 0000:00: resource 4 [io 0x0000-0x03af]
[ 3.175470] pci_bus 0000:00: resource 5 [io 0x03e0-0x0cf7]
[ 3.176854] pci_bus 0000:00: resource 6 [io 0x03b0-0x03df]
[ 3.177184] pci_bus 0000:00: resource 7 [io 0x0d00-0xffff]
[ 3.177514] pci_bus 0000:00: resource 8 [mem 0x000a0000-0x000bffff]
[ 3.177891] pci_bus 0000:00: resource 9 [mem 0x000c0000-0x000dffff]
[ 3.178267] pci_bus 0000:00: resource 10 [mem 0xd0000000-0xffffffff]
[ 3.178649] pci_bus 0000:01: resource 0 [io 0xe000-0xefff]
[ 3.178994] pci_bus 0000:01: resource 4 [io 0x0000-0x03af]
[ 3.179326] pci_bus 0000:01: resource 5 [io 0x03e0-0x0cf7]
[ 3.179672] pci_bus 0000:01: resource 6 [io 0x03b0-0x03df]
[ 3.180015] pci_bus 0000:01: resource 7 [io 0x0d00-0xffff]
[ 3.180344] pci_bus 0000:01: resource 8 [mem 0x000a0000-0x000bffff]
[ 3.180716] pci_bus 0000:01: resource 9 [mem 0x000c0000-0x000dffff]
[ 3.181081] pci_bus 0000:01: resource 10 [mem 0xd0000000-0xffffffff]
[ 3.181452] pci_bus 0000:02: resource 0 [io 0xd000-0xdfff]
[ 3.181792] pci_bus 0000:02: resource 1 [mem 0xfea00000-0xfeafffff]
[ 3.182168] pci_bus 0000:03: resource 0 [io 0xc000-0xcfff]
[ 3.182505] pci_bus 0000:03: resource 1 [mem 0xfe900000-0xfe9fffff]
[ 3.183045] NET: Registered protocol family 2
[ 3.184322] TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
[ 3.186023] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[ 3.186732] TCP: Hash tables configured (established 262144 bind 65536)
[ 3.187181] TCP: reno registered
[ 3.187411] UDP hash table entries: 1024 (order: 3, 32768 bytes)
[ 3.187803] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)
[ 3.188405] NET: Registered protocol family 1
[ 3.188924] RPC: Registered named UNIX socket transport module.
[ 3.189285] RPC: Registered udp transport module.
[ 3.189588] RPC: Registered tcp transport module.
[ 3.189901] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 3.190306] pci 0000:00:01.0: Boot video device
[ 3.190606] xen: registering gsi 18 triggering 0 polarity 1
[ 3.190967] xen: --> pirq=18 -> irq=18 (gsi=18)
[ 3.191310] xen: registering gsi 17 triggering 0 polarity 1
[ 3.191651] xen: --> pirq=17 -> irq=17 (gsi=17)
[ 3.191987] xen: registering gsi 18 triggering 0 polarity 1
[ 3.192322] Already setup the GSI :18
[ 3.779210] xen: registering gsi 17 triggering 0 polarity 1
[ 3.779560] Already setup the GSI :17
[ 3.779854] xen: registering gsi 18 triggering 0 polarity 1
[ 3.780189] Already setup the GSI :18
[ 3.852950] xen: registering gsi 17 triggering 0 polarity 1
[ 3.853302] Already setup the GSI :17
[ 3.853601] xen: registering gsi 18 triggering 0 polarity 1
[ 3.853944] Already setup the GSI :18
[ 3.927246] PCI: CLS 64 bytes, default 64
[ 3.927731] Unpacking initramfs...
[ 4.409499] Freeing initrd memory: 339204k freed
[ 4.502233] Machine check injector initialized
[ 4.504138] microcode: CPU0: patch_level=0x0300000f
[ 4.504463] microcode: CPU1: patch_level=0x0300000f
[ 4.504809] microcode: CPU2: patch_level=0x0300000f
[ 4.505347] microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
[ 4.506684] audit: initializing netlink socket (disabled)
[ 4.507060] type=2000 audit(1351615437.471:1): initialized
[ 4.520849] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[ 4.521549] VFS: Disk quotas dquot_6.5.2
[ 4.521865] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 4.522596] NFS: Registering the id_resolver key type
[ 4.522935] Key type id_resolver registered
[ 4.523199] Key type id_legacy registered
[ 4.523463] NTFS driver 2.1.30 [Flags: R/W].
[ 4.523927] msgmni has been set to 1798
[ 4.524230] SELinux: Registering netfilter hooks
[ 4.526131] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[ 4.526572] io scheduler noop registered
[ 4.526834] io scheduler deadline registered
[ 4.527129] io scheduler cfq registered (default)
[ 4.528468] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[ 4.529395] input: Power Button as /devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input0
[ 4.529907] ACPI: Power Button [PWRB]
[ 4.530299] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1
[ 4.530782] ACPI: Power Button [PWRF]
[ 4.592637] GHES: HEST is not enabled!
[ 4.592910] ioatdma: Intel(R) QuickData Technology Driver 4.00
[ 4.594402] xen-pciback: backend is vpci
[ 4.666851] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 4.688746] 00:0a: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[ 4.692137] xen: registering gsi 20 triggering 0 polarity 1
[ 4.692685] xen: --> pirq=20 -> irq=20 (gsi=20)
[ 4.715090] 0000:01:05.0: ttyS1 at I/O 0xe050 (irq = 20) is a 16550A
[ 4.723123] hpet_acpi_add: no address or irqs in _CRS
[ 4.729073] Non-volatile memory driver v1.3
[ 4.733807] Linux agpgart interface v0.103
[ 4.739678] [drm] Initialized drm 1.1.0 20060810
[ 4.747593] loop: module loaded
[ 4.751865] libphy: Fixed MDIO Bus: probed
[ 4.756157] tun: Universal TUN/TAP device driver, 1.6
[ 4.761408] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[ 4.768474] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.6.0-k
[ 4.778143] ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
[ 4.786308] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 4.793112] ehci_hcd: block sizes: qh 104 qtd 96 itd 192 sitd 96
[ 4.799394] xen: registering gsi 17 triggering 0 polarity 1
[ 4.805186] Already setup the GSI :17
[ 4.809041] ehci_hcd 0000:00:12.2: EHCI Host Controller
[ 4.814827] ehci_hcd 0000:00:12.2: new USB bus registered, assigned bus number 1
[ 4.822582] QUIRK: Enable AMD PLL fix
[ 4.826402] ehci_hcd 0000:00:12.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
[ 4.835422] ehci_hcd 0000:00:12.2: reset hcs_params 0x101505 dbg=1 cc=1 pcc=5 ordered !ppc ports=5
[ 4.844716] ehci_hcd 0000:00:12.2: reset hcc_params a076 thresh 7 uframes 256/512/1024 park
[ 4.853444] ehci_hcd 0000:00:12.2: park 0
[ 4.857628] ehci_hcd 0000:00:12.2: reset command 0080b02 park=3 ithresh=8 period=1024 Reset HALT
[ 4.866843] ehci_hcd 0000:00:12.2: debug port 1
[ 4.871568] ehci_hcd 0000:00:12.2: MWI active
[ 4.876100] ehci_hcd 0000:00:12.2: supports USB remote wakeup
[ 4.882116] ehci_hcd 0000:00:12.2: irq 17, io mem 0xfeb4f000
[ 4.887997] ehci_hcd 0000:00:12.2: init command 0010005 (park)=0 ithresh=1 period=512 RUN
[ 4.901882] ehci_hcd 0000:00:12.2: USB 2.0 started, EHCI 1.00
[ 4.907965] usb usb1: default language 0x0409
[ 4.912513] usb usb1: udev 1, busnum 1, minor = 0
[ 4.917409] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[ 4.924454] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 4.931952] usb usb1: Product: EHCI Host Controller
[ 4.937024] usb usb1: Manufacturer: Linux 3.7.0-rc3upstream-00220-g37b7153 ehci_hcd
[ 4.944967] usb usb1: SerialNumber: 0000:00:12.2
[ 4.950259] usb usb1: usb_probe_device
[ 4.954193] usb usb1: configuration #1 chosen from 1 choice
[ 4.959997] usb usb1: adding 1-0:1.0 (config #1, interface 0)
[ 4.966339] hub 1-0:1.0: usb_probe_interface
[ 4.970812] hub 1-0:1.0: usb_probe_interface - got id
[ 4.976063] hub 1-0:1.0: USB hub found
[ 4.979980] hub 1-0:1.0: 5 ports detected
[ 4.984155] hub 1-0:1.0: standalone hub
[ 4.988148] hub 1-0:1.0: no power switching (usb 1.0)
[ 4.993398] hub 1-0:1.0: individual port over-current protection
[ 4.999631] hub 1-0:1.0: power on to power good time: 20ms
[ 5.005333] hub 1-0:1.0: local power source is good
[ 5.010998] hub 1-0:1.0: trying to enable port power on non-switchable hub
[ 5.018235] xen: registering gsi 17 triggering 0 polarity 1
[ 5.024028] Already setup the GSI :17
[ 5.027880] ehci_hcd 0000:00:13.2: EHCI Host Controller
[ 5.033663] ehci_hcd 0000:00:13.2: new USB bus registered, assigned bus number 2
[ 5.041377] ehci_hcd 0000:00:13.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
[ 5.050398] ehci_hcd 0000:00:13.2: reset hcs_params 0x101505 dbg=1 cc=1 pcc=5 ordered !ppc ports=5
[ 5.059685] ehci_hcd 0000:00:13.2: reset hcc_params a076 thresh 7 uframes 256/512/1024 park
[ 5.068427] ehci_hcd 0000:00:13.2: park 0
[ 5.072610] ehci_hcd 0000:00:13.2: reset command 0080b02 park=3 ithresh=8 period=1024 Reset HALT
[ 5.081823] ehci_hcd 0000:00:13.2: debug port 1
[ 5.086547] ehci_hcd 0000:00:13.2: MWI active
[ 5.091081] ehci_hcd 0000:00:13.2: supports USB remote wakeup
[ 5.097059] ehci_hcd 0000:00:13.2: irq 17, io mem 0xfeb4d000
[ 5.102937] ehci_hcd 0000:00:13.2: init command 0010005 (park)=0 ithresh=1 period=512 RUN
[ 5.116889] ehci_hcd 0000:00:13.2: USB 2.0 started, EHCI 1.00
[ 5.123004] usb usb2: default language 0x0409
[ 5.127553] usb usb2: udev 1, busnum 2, minor = 128
[ 5.132627] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
[ 5.139668] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 5.147164] usb usb2: Product: EHCI Host Controller
[ 5.152236] usb usb2: Manufacturer: Linux 3.7.0-rc3upstream-00220-g37b7153 ehci_hcd
[ 5.160180] usb usb2: SerialNumber: 0000:00:13.2
[ 5.165185] hub 1-0:1.0: state 7 ports 5 chg 0000 evt 0000
[ 5.165547] usb usb2: usb_probe_device
[ 5.165551] usb usb2: configuration #1 chosen from 1 choice
[ 5.165570] usb usb2: adding 2-0:1.0 (config #1, interface 0)
[ 5.165905] hub 2-0:1.0: usb_probe_interface
[ 5.165908] hub 2-0:1.0: usb_probe_interface - got id
[ 5.165911] hub 2-0:1.0: USB hub found
[ 5.165929] hub 2-0:1.0: 5 ports detected
[ 5.165930] hub 2-0:1.0: standalone hub
[ 5.165931] hub 2-0:1.0: no power switching (usb 1.0)
[ 5.165933] hub 2-0:1.0: individual port over-current protection
[ 5.165935] hub 2-0:1.0: power on to power good time: 20ms
[ 5.165942] hub 2-0:1.0: local power source is good
[ 5.166518] hub 2-0:1.0: trying to enable port power on non-switchable hub
[ 5.166966] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[ 5.166969] ohci_hcd: block sizes: ed 80 td 96
[ 5.167028] xen: registering gsi 18 triggering 0 polarity 1
[ 5.167033] Already setup the GSI :18
[ 5.167075] ohci_hcd 0000:00:12.0: OHCI Host Controller
[ 5.167260] ohci_hcd 0000:00:12.0: new USB bus registered, assigned bus number 3
[ 5.167338] ohci_hcd 0000:00:12.0: created debug files
[ 5.167340] ohci_hcd 0000:00:12.0: supports USB remote wakeup
[ 5.167385] ohci_hcd 0000:00:12.0: irq 18, io mem 0xfeb50000
[ 5.289966] hub 2-0:1.0: state 7 ports 5 chg 0000 evt 0000
[ 5.293989] ohci_hcd 0000:00:12.0: OHCI controller state
[ 5.293995] ohci_hcd 0000:00:12.0: OHCI 1.0, NO legacy support registers, rh state running
[ 5.293999] ohci_hcd 0000:00:12.0: control 0x283 RWC HCFS=operational CBSR=3
[ 5.294002] ohci_hcd 0000:00:12.0: cmdstatus 0x00000 SOC=0
[ 5.294006] ohci_hcd 0000:00:12.0: intrstatus 0x00000004 SF
[ 5.294009] ohci_hcd 0000:00:12.0: intrenable 0x8000005a MIE RHSC UE RD WDH
[ 5.294019] ohci_hcd 0000:00:12.0: hcca frame #0005
[ 5.294022] ohci_hcd 0000:00:12.0: roothub.a 02001205 POTPGT=2 NOCP NPS NDP=5(5)
[ 5.294025] ohci_hcd 0000:00:12.0: roothub.b 00000000 PPCM=0000 DR=0000
[ 5.294027] ohci_hcd 0000:00:12.0: roothub.status 00008000 DRWE
[ 5.294031] ohci_hcd 0000:00:12.0: roothub.portstatus [0] 0x00000100 PPS
[ 5.294034] ohci_hcd 0000:00:12.0: roothub.portstatus [1] 0x00000100 PPS
[ 5.294037] ohci_hcd 0000:00:12.0: roothub.portstatus [2] 0x00000100 PPS
[ 5.294041] ohci_hcd 0000:00:12.0: roothub.portstatus [3] 0x00000100 PPS
[ 5.294044] ohci_hcd 0000:00:12.0: roothub.portstatus [4] 0x00000100 PPS
[ 5.294080] usb usb3: default language 0x0409
[ 5.294093] usb usb3: udev 1, busnum 3, minor = 256
[ 5.294095] usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
[ 5.294097] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 5.294098] usb usb3: Product: OHCI Host Controller
[ 5.294100] usb usb3: Manufacturer: Linux 3.7.0-rc3upstream-00220-g37b7153 ohci_hcd
[ 5.294101] usb usb3: SerialNumber: 0000:00:12.0
[ 5.294371] usb usb3: usb_probe_device
[ 5.294374] usb usb3: configuration #1 chosen from 1 choice
[ 5.294386] usb usb3: adding 3-0:1.0 (config #1, interface 0)
[ 5.294481] hub 3-0:1.0: usb_probe_interface
[ 5.294482] hub 3-0:1.0: usb_probe_interface - got id
[ 5.294484] hub 3-0:1.0: USB hub found
[ 5.294492] hub 3-0:1.0: 5 ports detected
[ 5.294493] hub 3-0:1.0: standalone hub
[ 5.294495] hub 3-0:1.0: no power switching (usb 1.0)
[ 5.294496] hub 3-0:1.0: no over-current protection
[ 5.294497] hub 3-0:1.0: power on to power good time: 4ms
[ 5.294505] hub 3-0:1.0: local power source is good
[ 5.295074] hub 3-0:1.0: trying to enable port power on non-switchable hub
[ 5.295131] ehci_hcd 0000:00:12.2: HS companion for 0000:00:12.0
[ 5.295174] xen: registering gsi 18 triggering 0 polarity 1
[ 5.295178] Already setup the GSI :18
[ 5.295219] ohci_hcd 0000:00:13.0: OHCI Host Controller
[ 5.295337] ohci_hcd 0000:00:13.0: new USB bus registered, assigned bus number 4
[ 5.295397] ohci_hcd 0000:00:13.0: created debug files
[ 5.295399] ohci_hcd 0000:00:13.0: supports USB remote wakeup
[ 5.295409] ohci_hcd 0000:00:13.0: irq 18, io mem 0xfeb4e000
[ 5.549971] hub 3-0:1.0: state 7 ports 5 chg 0000 evt 0000
[ 5.553975] ohci_hcd 0000:00:13.0: OHCI controller state
[ 5.553980] ohci_hcd 0000:00:13.0: OHCI 1.0, NO legacy support registers, rh state running
[ 5.553984] ohci_hcd 0000:00:13.0: control 0x283 RWC HCFS=operational CBSR=3
[ 5.553987] ohci_hcd 0000:00:13.0: cmdstatus 0x00000 SOC=0
[ 5.553990] ohci_hcd 0000:00:13.0: intrstatus 0x00000004 SF
[ 5.553993] ohci_hcd 0000:00:13.0: intrenable 0x8000005a MIE RHSC UE RD WDH
[ 5.554004] ohci_hcd 0000:00:13.0: hcca frame #0005
[ 5.554008] ohci_hcd 0000:00:13.0: roothub.a 02001205 POTPGT=2 NOCP NPS NDP=5(5)
[ 5.554011] ohci_hcd 0000:00:13.0: roothub.b 00000000 PPCM=0000 DR=0000
[ 5.554014] ohci_hcd 0000:00:13.0: roothub.status 00008000 DRWE
[ 5.554018] ohci_hcd 0000:00:13.0: roothub.portstatus [0] 0x00000100 PPS
[ 5.554021] ohci_hcd 0000:00:13.0: roothub.portstatus [1] 0x00000100 PPS
[ 5.554025] ohci_hcd 0000:00:13.0: roothub.portstatus [2] 0x00000100 PPS
[ 5.554028] ohci_hcd 0000:00:13.0: roothub.portstatus [3] 0x00000100 PPS
[ 5.554031] ohci_hcd 0000:00:13.0: roothub.portstatus [4] 0x00000100 PPS
[ 5.554051] usb usb4: default language 0x0409
[ 5.554064] usb usb4: udev 1, busnum 4, minor = 384
[ 5.554066] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
[ 5.554068] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 5.554070] usb usb4: Product: OHCI Host Controller
[ 5.554071] usb usb4: Manufacturer: Linux 3.7.0-rc3upstream-00220-g37b7153 ohci_hcd
[ 5.554072] usb usb4: SerialNumber: 0000:00:13.0
[ 5.554538] usb usb4: usb_probe_device
[ 5.554542] usb usb4: configuration #1 chosen from 1 choice
[ 5.554559] usb usb4: adding 4-0:1.0 (config #1, interface 0)
[ 5.554665] hub 4-0:1.0: usb_probe_interface
[ 5.554667] hub 4-0:1.0: usb_probe_interface - got id
[ 5.554669] hub 4-0:1.0: USB hub found
[ 5.554678] hub 4-0:1.0: 5 ports detected
[ 5.554679] hub 4-0:1.0: standalone hub
[ 5.554681] hub 4-0:1.0: no power switching (usb 1.0)
[ 5.554682] hub 4-0:1.0: no over-current protection
[ 5.554683] hub 4-0:1.0: power on to power good time: 4ms
[ 5.554691] hub 4-0:1.0: local power source is good
[ 5.555283] hub 4-0:1.0: trying to enable port power on non-switchable hub
[ 5.555346] ehci_hcd 0000:00:13.2: HS companion for 0000:00:13.0
[ 5.555389] xen: registering gsi 18 triggering 0 polarity 1
[ 5.555394] Already setup the GSI :18
[ 5.555433] ohci_hcd 0000:00:14.5: OHCI Host Controller
[ 5.555598] ohci_hcd 0000:00:14.5: new USB bus registered, assigned bus number 5
[ 5.555670] ohci_hcd 0000:00:14.5: created debug files
[ 5.555672] ohci_hcd 0000:00:14.5: supports USB remote wakeup
[ 5.555685] ohci_hcd 0000:00:14.5: irq 18, io mem 0xfeb4c000
[ 5.809859] hub 4-0:1.0: state 7 ports 5 chg 0000 evt 0000
[ 5.813841] ohci_hcd 0000:00:14.5: OHCI controller state
[ 5.813847] ohci_hcd 0000:00:14.5: OHCI 1.0, NO legacy support registers, rh state running
[ 5.813851] ohci_hcd 0000:00:14.5: control 0x283 RWC HCFS=operational CBSR=3
[ 5.813854] ohci_hcd 0000:00:14.5: cmdstatus 0x00000 SOC=0
[ 5.813857] ohci_hcd 0000:00:14.5: intrstatus 0x00000004 SF
[ 5.813860] ohci_hcd 0000:00:14.5: intrenable 0x8000005a MIE RHSC UE RD WDH
[ 5.813871] ohci_hcd 0000:00:14.5: hcca frame #0004
[ 5.813874] ohci_hcd 0000:00:14.5: roothub.a 02001202 POTPGT=2 NOCP NPS NDP=2(2)
[ 5.813876] ohci_hcd 0000:00:14.5: roothub.b 00000000 PPCM=0000 DR=0000
[ 5.813880] ohci_hcd 0000:00:14.5: roothub.status 00008000 DRWE
[ 5.813884] ohci_hcd 0000:00:14.5: roothub.portstatus [0] 0x00000100 PPS
[ 5.813886] ohci_hcd 0000:00:14.5: roothub.portstatus [1] 0x00000100 PPS
[ 5.813923] usb usb5: default language 0x0409
[ 5.813935] usb usb5: udev 1, busnum 5, minor = 512
[ 5.813938] usb usb5: New USB device found, idVendor=1d6b, idProduct=0001
[ 5.813939] usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 5.813941] usb usb5: Product: OHCI Host Controller
[ 5.813942] usb usb5: Manufacturer: Linux 3.7.0-rc3upstream-00220-g37b7153 ohci_hcd
[ 5.813944] usb usb5: SerialNumber: 0000:00:14.5
[ 5.814220] usb usb5: usb_probe_device
[ 5.814223] usb usb5: configuration #1 chosen from 1 choice
[ 5.814238] usb usb5: adding 5-0:1.0 (config #1, interface 0)
[ 5.814373] hub 5-0:1.0: usb_probe_interface
[ 5.814375] hub 5-0:1.0: usb_probe_interface - got id
[ 5.814377] hub 5-0:1.0: USB hub found
[ 5.814390] hub 5-0:1.0: 2 ports detected
[ 5.814392] hub 5-0:1.0: standalone hub
[ 5.814393] hub 5-0:1.0: no power switching (usb 1.0)
[ 5.814394] hub 5-0:1.0: no over-current protection
[ 5.814396] hub 5-0:1.0: power on to power good time: 4ms
[ 5.814405] hub 5-0:1.0: local power source is good
[ 5.814425] hub 5-0:1.0: trying to enable port power on non-switchable hub
[ 5.814584] uhci_hcd: USB Universal Host Controller Interface driver
[ 6.009259] hub 5-0:1.0: state 7 ports 2 chg 0000 evt 0000
[ 6.009372] usbcore: registered new interface driver usblp
[ 6.009676] i8042: PNP: No PS/2 controller found. Probing ports directly.
[ 6.010397] serio: i8042 KBD port at 0x60,0x64 irq 1
[ 6.010409] serio: i8042 AUX port at 0x60,0x64 irq 12
[ 6.010691] mousedev: PS/2 mouse device common for all mice
[ 6.011327] rtc_cmos 00:05: RTC can wake from S4
[ 6.011618] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0
[ 6.011662] rtc0: alarms up to one month, y3k, 114 bytes nvram
[ 6.011868] EFI Variables Facility v0.08 2004-May-17
[ 6.011957] zram: num_devices not specified. Using default: 1
[ 6.011958] zram: Creating 1 devices ...
[ 6.077064] Netfilter messages via NETLINK v0.30.
[ 6.082006] nf_conntrack version 0.5.0 (7194 buckets, 28776 max)
[ 6.088334] ctnetlink v0.93: registering with nfnetlink.
[ 6.094201] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 6.099817] TCP: cubic registered
[ 6.103279] Initializing XFRM netlink socket
[ 6.107873] NET: Registered protocol family 10
[ 6.112784] ip6_tables: (C) 2000-2006 Netfilter Core Team
[ 6.119125] sit: IPv6 over IPv4 tunneling driver
[ 6.124870] NET: Registered protocol family 17
[ 6.129593] Key type dns_resolver registered
[ 6.135108] PM: Hibernation image not present or could not be loaded.
[ 6.141853] registered taskstats version 1
[ 6.146837] Magic number: 0:661:743
[ 6.151899] Freeing unused kernel memory: 752k freed
[ 6.157248] Write protecting the kernel read-only data: 10240k
[ 6.169212] Freeing unused kernel memory: 1768k freed
[ 6.175129] Freeing unused kernel memory: 168k freed
[ 6.188257] consoletype (1257) used greatest stack depth: 5272 bytes left
[ 6.524961] modprobe (1286) used greatest stack depth: 5256 bytes left
[ 6.547403] core_filesystem (1258) used greatest stack depth: 4952 bytes left
[ 6.592358] Initialising Xen virtual ethernet driver.
[ 6.722568] wmi: Mapper loaded
[ 6.796831] xen: registering gsi 17 triggering 0 polarity 1
[ 6.802666] Already setup the GSI :17
[ 6.808260] e1000e: Intel(R) PRO/1000 Network Driver - 2.1.4-k
[ 6.814354] e1000e: Copyright(c) 1999 - 2012 Intel Corporation.
[ 6.816573] SCSI subsystem initialized
[ 6.824498] e1000e 0000:02:00.0: Disabling ASPM L0s L1
[ 6.824524] xen: registering gsi 16 triggering 0 polarity 1
[ 6.824531] Already setup the GSI :16
[ 6.824740] e1000e 0000:02:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[ 6.859455] [drm] radeon defaulting to kernel modesetting.
[ 6.866428] [drm] radeon kernel modesetting enabled.
[ 6.873121] xen: registering gsi 18 triggering 0 polarity 1
[ 6.873126] Already setup the GSI :18
[ 6.883269] atl1c 0000:03:00.0: version 1.0.1.0-NAPI
[ 6.883447] [drm] initializing kernel modesetting (SUMO 0x1002:0x9640 0x1043:0x84C8).
[ 6.883509] [drm] register mmio base: 0xFEB00000
[ 6.883511] [drm] register mmio size: 262144
[ 6.883646] ATOM BIOS: General
[ 6.883717] radeon 0000:00:01.0: VRAM: 512M 0x0000000000000000 - 0x000000001FFFFFFF (512M used)
[ 6.883720] radeon 0000:00:01.0: GTT: 512M 0x0000000020000000 - 0x000000003FFFFFFF
[ 6.883722] [drm] Detected VRAM RAM=512M, BAR=256M
[ 6.883725] [drm] RAM width 32bits DDR
[ 6.884463] [TTM] Zone kernel: Available graphics memory: 461822 kiB
[ 6.884466] [TTM] Initializing pool allocator
[ 6.884476] [TTM] Initializing DMA pool allocator
[ 6.884529] [drm] radeon: 512M of VRAM memory ready
[ 6.884531] [drm] radeon: 512M of GTT memory ready.
[ 6.884622] [drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
[ 6.884625] [drm] Driver supports precise vblank timestamp query.
[ 6.884802] radeon 0000:00:01.0: radeon: using MSI.
[ 6.884865] [drm] radeon: irq initialized.
[ 6.884871] [drm] GART: num cpu pages 131072, num gpu pages 131072
[ 6.895835] [drm] Loading SUMO Microcode
[ 6.933742] ACPI: bus type scsi registered
[ 6.941860] e1000e 0000:02:00.0 eth1: (PCI Express:2.5GT/s:Width x1) 00:1b:21:ab:c6:12
[ 6.941862] e1000e 0000:02:00.0 eth1: Intel(R) PRO/1000 Network Connection
[ 6.941882] e1000e 0000:02:00.0 eth1: MAC: 3, PHY: 8, PBA No: E46981-005
[ 6.997612] ip (1909) used greatest stack depth: 3896 bytes left
[ 7.048540] libata version 3.00 loaded.
[ 7.144568] [drm] PCIE GART of 512M enabled (table at 0x0000000000040000).
[ 7.151898] radeon 0000:00:01.0: WB enabled
[ 7.156253] radeon 0000:00:01.0: fence driver on ring 0 use gpu addr 0x0000000020000c00 and cpu addr 0xffff880023235c00
[ 7.185375] [drm] ring test on 0 succeeded in 1 usecs
[ 7.191817] [drm] ib test on ring 0 succeeded in 0 usecs
[ 7.209657] [drm] Radeon Display Connectors
[ 7.214052] [drm] Connector 0:
[ 7.217253] [drm] VGA-1
[ 7.220007] [drm] HPD2
[ 7.222674] [drm] DDC: 0x6440 0x6440 0x6444 0x6444 0x6448 0x6448 0x644c 0x644c
[ 7.230416] [drm] Encoders:
[ 7.233539] [drm] CRT1: INTERNAL_UNIPHY2
[ 7.238001] [drm] CRT1: NUTMEG
[ 7.238003] [drm] Connector 1:
[ 7.238005] [drm] HDMI-A-1
[ 7.238006] [drm] HPD1
[ 7.238009] [drm] DDC: 0x6430 0x6430 0x6434 0x6434 0x6438 0x6438 0x643c 0x643c
[ 7.238011] [drm] Encoders:
[ 7.238011] [drm] DFP1: INTERNAL_UNIPHY2
[ 7.255949] [drm] Internal thermal controller without fan control
[ 7.259955] [drm] radeon: power management initialized
[ 7.335529] No connectors reported connected with modes
[ 7.340968] [drm] Cannot find any crtc or sizes - going 1024x768
[ 7.350906] [drm] fb mappable at 0xD0142000
[ 7.355261] [drm] vram apper at 0xD0000000
[ 7.359523] [drm] size 3145728
[ 7.362716] [drm] fb depth is 24
[ 7.366146] [drm] pitch is 4096
[ 7.370065] fbcon: radeondrmfb (fb0) is primary device
[ 7.376982] ttyS1: 1 input overrun(s)
[ 7.411210] Console: switching to colour frame buffer device 128x48
[ 7.424071] fb0: radeondrmfb frame buffer device
[ 7.428949] drm: registered panic notifier
[ 7.433284] [drm] Initialized radeon 2.24.0 20080528 for 0000:00:01.0 on minor 0
[ 7.441164] ahci 0000:00:11.0: version 3.0
[ 7.445546] xen: registering gsi 19 triggering 0 polarity 1
[ 7.451464] xen: --> pirq=19 -> irq=19 (gsi=19)
[ 7.456473] ahci 0000:00:11.0: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
[ 7.465104] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part
[ 7.478140] scsi0 : ahci
[ 7.481629] scsi1 : ahci
[ 7.484929] scsi2 : ahci
[ 7.488287] scsi3 : ahci
[ 7.491496] scsi4 : ahci
[ 7.494907] scsi5 : ahci
[ 7.498649] ata1: SATA max UDMA/133 abar m2048@0xfeb51000 port 0xfeb51100 irq 71
[ 7.507585] ata2: SATA max UDMA/133 abar m2048@0xfeb51000 port 0xfeb51180 irq 71
[ 7.515392] ata3: SATA max UDMA/133 abar m2048@0xfeb51000 port 0xfeb51200 irq 71
[ 7.523177] ata4: SATA max UDMA/133 abar m2048@0xfeb51000 port 0xfeb51280 irq 71
[ 7.530982] ata5: SATA max UDMA/133 abar m2048@0xfeb51000 port 0xfeb51300 irq 71
[ 7.538786] ata6: SATA max UDMA/133 abar m2048@0xfeb51000 port 0xfeb51380 irq 71
[ 7.850775] ata1: SATA link down (SStatus 0 SControl 300)
[ 7.856613] ata2: SATA link down (SStatus 0 SControl 300)
[ 7.862497] ata6: SATA link down (SStatus 0 SControl 300)
[ 7.871228] ata3: SATA link down (SStatus 0 SControl 300)
[ 7.879887] ata5: SATA link down (SStatus 0 SControl 300)
[ 8.038131] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 8.062593] ata4.00: ATA-7: WDC WD800AAJS-18TDA0, 01.00A03, max UDMA/133
[ 8.072417] ata4.00: 156250000 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
[ 8.082336] ata4.00: failed to get Identify Device Data, Emask 0x1
[ 8.092297] ata4.00: failed to get Identify Device Data, Emask 0x1
[ 8.100682] ata4.00: configured for UDMA/133
[ 8.107497] scsi 3:0:0:0: Direct-Access ATA WDC WD800AAJS-18 01.0 PQ: 0 ANSI: 5
[ 8.129843] sd 3:0:0:0: [sda] 156250000 512-byte logical blocks: (80.0 GB/74.5 GiB)
[ 8.140007] sd 3:0:0:0: [sda] Write Protect is off
[ 8.146945] sd 3:0:0:0: [sda] Mode Sense: 00 3a 00 00
[ 8.154244] sd 3:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 8.199771] sda: sda1 sda2 sda3 sda4
[ 8.210772] sd 3:0:0:0: [sda] Attached SCSI disk
[ 8.224663] sd 3:0:0:0: Attached scsi generic sg0 type 0
[ 8.944409] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 8.963120] device eth0 entered promiscuous mode
[ 9.181754] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[ 10.053518] atl1c 0000:03:00.0: atl1c: eth0 NIC Link is Up<1000 Mbps Full Duplex>
[ 10.064785] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 11.555156] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[ 11.566749] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[ 13.964892] switch: port 1(eth0) entered forwarding state
[ 13.973524] switch: port 1(eth0) entered forwarding state
[ 16.232334] Loading iSCSI transport class v2.0-870.
[ 16.248793] iscsi: registered transport (tcp)
[ 16.306470] Event-channel device installed.
[ 19.630065] mount.nfs (3143) used greatest stack depth: 3224 bytes left
[ 21.157586] device-mapper: ioctl: 4.23.0-ioctl (2012-07-25) initialised: dm-devel@redhat.com
[ 21.170420] device-mapper: multipath: version 1.5.0 loaded
[ 21.450531] scsi6 : iSCSI Initiator over TCP/IP
[ 21.731710] scsi 6:0:0:0: Direct-Access LIO-ORG IBLOCK 4.0 PQ: 0 ANSI: 5
[ 21.747167] sd 6:0:0:0: [sdb] 503316480 512-byte logical blocks: (257 GB/240 GiB)
[ 21.747211] sd 6:0:0:0: Attached scsi generic sg1 type 0
[ 21.748648] scsi 6:0:0:1: Direct-Access LIO-ORG IBLOCK 4.0 PQ: 0 ANSI: 5
[ 21.752968] sd 6:0:0:1: [sdc] 167772160 512-byte logical blocks: (85.8 GB/80.0 GiB)
[ 21.753042] sd 6:0:0:1: Attached scsi generic sg2 type 0
[ 21.755647] sd 6:0:0:1: [sdc] Write Protect is off
[ 21.755655] sd 6:0:0:1: [sdc] Mode Sense: 2f 00 00 00
[ 21.756638] sd 6:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 21.761219] scsi 6:0:0:2: Direct-Access LIO-ORG IBLOCK 4.0 PQ: 0 ANSI: 5
[ 21.770871] sd 6:0:0:2: Attached scsi generic sg3 type 0
[ 21.771168] sd 6:0:0:2: [sdd] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
[ 21.775163] ttyS1: 1 input overrun(s)
[ 21.776400] sd 6:0:0:2: [sdd] Write Protect is off
[ 21.776405] sd 6:0:0:2: [sdd] Mode Sense: 2f 00 00 00
[ 21.776884] sd 6:0:0:2: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 21.780001] sdd: unknown partition table
[ 21.781272] sd 6:0:0:2: [sdd] Attached SCSI disk
[ 21.804226] sdc: sdc1 sdc2 < sdc5 >
[ 21.806353] sd 6:0:0:1: [sdc] Attached SCSI disk
[ 21.918632] sd 6:0:0:0: [sdb] Write Protect is off
[ 21.926413] sd 6:0:0:0: [sdb] Mode Sense: 2f 00 00 00
[ 21.935417] sd 6:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 21.963859] sdb: unknown partition table
[ 21.974284] sd 6:0:0:0: [sdb] Attached SCSI disk
[ 28.589544] bio: create slab <bio-1> at 1
[ 28.997728] switch: port 1(eth0) entered forwarding state
[ 57.703132] device vif1.0 entered promiscuous mode
[ 57.715328] IPv6: ADDRCONF(NETDEV_UP): vif1.0: link is not ready
[ 59.760909] IPv6: ADDRCONF(NETDEV_CHANGE): vif1.0: link becomes ready
[ 59.770384] switch: port 2(vif1.0) entered forwarding state
[ 59.778900] switch: port 2(vif1.0) entered forwarding state
[ 59.876529] xen-blkback:ring-ref 10, event-channel 18, protocol 2 (x86_32-abi) persistent grants
[ 65.135453] switch: port 2(vif1.0) entered disabled state
[ 65.145638] device vif1.0 left promiscuous mode
[ 65.154458] switch: port 2(vif1.0) entered disabled state
[ 70.836602] device vif2.0 entered promiscuous mode
[ 70.848191] IPv6: ADDRCONF(NETDEV_UP): vif2.0: link is not ready
[ 72.285945] IPv6: ADDRCONF(NETDEV_CHANGE): vif2.0: link becomes ready
[ 72.295310] switch: port 2(vif2.0) entered forwarding state
[ 72.303597] switch: port 2(vif2.0) entered forwarding state
[ 72.403140] xen-blkback:ring-ref 10, event-channel 11, protocol 1 (x86_64-abi) persistent grants
[ 72.544101] ------------[ cut here ]------------
[ 72.552932] kernel BUG at /home/konrad/linux/drivers/block/xen-blkback/blkback.c:589!
[ 72.563680] invalid opcode: 0000 [#1] SMP
[ 72.570865] Modules linked in: dm_multipath dm_mod xen_evtchn iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi libcrc32c crc32c sg sd_mod ahci libahci libata radeon e1000e scsi_mod atl1c fbcon ttm tileblit font bitblit softcursor drm_kms_helper wmi xen_blkfront xen_netfront fb_sys_fops sysimgblt sysfillrect syscopyarea xenfs xen_privcmd [last unloaded: dump_dma]
[ 72.617251] CPU 0
[ 72.619173] Pid: 3823, comm: blkback.2.xvda Tainted: G O 3.7.0-rc3upstream-00220-g37b7153 #1 System manufacturer System Product Name/F1A75-M
[ 72.641606] RIP: e030:[<ffffffff81409766>] [<ffffffff81409766>] xen_blkbk_map+0x696/0x6e0
[ 72.653181] RSP: e02b:ffff880027dd3728 EFLAGS: 00010246
[ 72.661651] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[ 72.672090] RDX: ffff8800232d7f40 RSI: 0000000000000000 RDI: ffff880027dd3d88
[ 72.682437] RBP: ffff880027dd39e8 R08: 0000000000000000 R09: 0000000000000000
[ 72.692835] R10: 0000000000000001 R11: dead000000200200 R12: 0000000000000000
[ 72.703261] R13: 0000000000000000 R14: ffff88002b5e7070 R15: 0000000000000000
[ 72.713560] FS: 00007f6cc62d5700(0000) GS:ffff88003e000000(0000) knlGS:0000000000000000
[ 72.724921] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 72.733811] CR2: 00000000006dd384 CR3: 0000000027b8f000 CR4: 0000000000000660
[ 72.744131] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 72.754517] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 72.764882] Process blkback.2.xvda (pid: 3823, threadinfo ffff880027dd2000, task ffff88002bb15040)
[ 72.777140] Stack:
[ 72.782184] ffff880027dd3738 ffff880026081af0 ffff880027dd3798 ffffffff810ac7df
[ 72.792960] ffff880027dd3798 ffffffff8104c506 0000000000000117 ffff88002160d030
[ 72.803712] ffff880027dd3a38 ffff88002b5e7120 ffff88002160d000 ffff880027dd3d88
[ 72.814469] Call Trace:
[ 72.820094] [<ffffffff810ac7df>] ? __queue_work+0xff/0x420
[ 72.829023] [<ffffffff8104c506>] ? xen_spin_lock_flags+0xb6/0x120
[ 72.838462] [<ffffffff810acb61>] ? queue_work_on+0x31/0x50
[ 72.847279] [<ffffffff81636eb9>] ? _raw_spin_unlock_irqrestore+0x19/0x30
[ 72.854408] ttyS1: 2 input overrun(s)
[ 72.864352] [<ffffffff8107e068>] ? pvclock_clocksource_read+0x58/0xd0
[ 72.874471] [<ffffffff812ea543>] ? cpumask_next_and+0x23/0x40
[ 72.883656] [<ffffffff812ea543>] ? cpumask_next_and+0x23/0x40
[ 72.892716] [<ffffffff810cc5e7>] ? update_sd_lb_stats+0x157/0x6c0
[ 72.902178] [<ffffffff81636e90>] ? _raw_spin_lock_irq+0x20/0x30
[ 72.911486] [<ffffffff810cd441>] ? find_busiest_group+0x31/0x4d0
[ 72.920849] [<ffffffff81409e87>] dispatch_rw_block_io+0x377/0x600
[ 72.930187] [<ffffffff8107e068>] ? pvclock_clocksource_read+0x58/0xd0
[ 72.939957] [<ffffffff8103e0c0>] ? xen_mc_flush+0xc0/0x1f0
[ 72.948743] [<ffffffff8103c9e9>] ? xen_end_context_switch+0x19/0x20
[ 72.958251] [<ffffffff8107e068>] ? pvclock_clocksource_read+0x58/0xd0
[ 72.967917] [<ffffffff8107e068>] ? pvclock_clocksource_read+0x58/0xd0
[ 72.977617] [<ffffffff8107e068>] ? pvclock_clocksource_read+0x58/0xd0
[ 72.987474] [<ffffffff81044359>] ? xen_clocksource_read+0x39/0x50
[ 72.997278] [<ffffffff8104c506>] ? xen_spin_lock_flags+0xb6/0x120
[ 73.006476] [<ffffffff8140a32e>] xen_blkif_schedule+0x21e/0xa00
[ 73.015493] [<ffffffff81111442>] ? irq_to_desc+0x12/0x20
[ 73.023833] [<ffffffff81114779>] ? irq_get_irq_data+0x9/0x10
[ 73.032418] [<ffffffff81382909>] ? info_for_irq+0x9/0x20
[ 73.040554] [<ffffffff81383cb9>] ? notify_remote_via_irq+0x29/0x50
[ 73.049523] [<ffffffff810c844d>] ? sched_clock_cpu+0xcd/0x110
[ 73.058024] [<ffffffff8107e068>] ? pvclock_clocksource_read+0x58/0xd0
[ 73.067191] [<ffffffff8103e0c0>] ? xen_mc_flush+0xc0/0x1f0
[ 73.075338] [<ffffffff81635e9e>] ? __schedule+0x3be/0x7c0
[ 73.083311] [<ffffffff810b52a0>] ? wake_up_bit+0x40/0x40
[ 73.091108] [<ffffffff8140a110>] ? dispatch_rw_block_io+0x600/0x600
[ 73.099902] [<ffffffff810b4b16>] kthread+0xc6/0xd0
[ 73.107124] [<ffffffff8103c9e9>] ? xen_end_context_switch+0x19/0x20
[ 73.115845] [<ffffffff810b4a50>] ? kthread_freezable_should_stop+0x80/0x80
[ 73.125266] [<ffffffff8163f1fc>] ret_from_fork+0x7c/0xb0
[ 73.133089] [<ffffffff810b4a50>] ? kthread_freezable_should_stop+0x80/0x80
[ 73.142576] Code: 48 89 d7 e8 ad 66 d8 ff 4a c7 84 3d 70 ff ff ff 00 00 00 00 4c 8b 85 60 fd ff ff 41 8b b0 e4 fd ff ff 41 83 cd 01 e9 ef fb ff ff <0f> 0b eb fe 48 8d 95 10 ff ff ff 48 8d bd b0 fd ff ff 31 f6 44
[ 73.167611] RIP [<ffffffff81409766>] xen_blkbk_map+0x696/0x6e0
[ 73.176081] RSP <ffff880027dd3728>
[ 73.182024] ---[ end trace 914a52d8b62134db ]---
[ 87.339441] switch: port 2(vif2.0) entered forwarding state
[ 315.067569] device tap3.0 entered promiscuous mode
[ 315.074965] switch: port 3(tap3.0) entered forwarding state
[ 315.083116] switch: port 3(tap3.0) entered forwarding state
[ 315.142543] switch: port 3(tap3.0) entered disabled state
[ 315.161150] switch: port 3(tap3.0) entered forwarding state
[ 315.169097] switch: port 3(tap3.0) entered forwarding state
[ 330.162411] switch: port 3(tap3.0) entered forwarding state
[ 415.483626] switch: port 3(tap3.0) entered disabled state
[ 415.491439] device tap3.0 left promiscuous mode
[ 415.498191] switch: port 3(tap3.0) entered disabled state
[ 658.839306] device tap4.0 entered promiscuous mode
[ 658.846451] switch: port 3(tap4.0) entered forwarding state
[ 658.854354] switch: port 3(tap4.0) entered forwarding state
[ 658.923751] switch: port 3(tap4.0) entered disabled state
[ 658.942642] switch: port 3(tap4.0) entered forwarding state
[ 658.950492] switch: port 3(tap4.0) entered forwarding state
[ 673.990219] switch: port 3(tap4.0) entered forwarding state
[ 759.263762] switch: port 3(tap4.0) entered disabled state
[ 759.271529] device tap4.0 left promiscuous mode
[ 759.278257] switch: port 3(tap4.0) entered disabled state
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v2] Persistent grant maps for xen blk drivers
2012-10-30 17:01 ` Konrad Rzeszutek Wilk
@ 2012-10-30 18:33 ` Roger Pau Monné
2012-10-30 20:38 ` Konrad Rzeszutek Wilk
0 siblings, 1 reply; 13+ messages in thread
From: Roger Pau Monné @ 2012-10-30 18:33 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk; +Cc: xen-devel, linux-kernel
On 30/10/12 18:01, Konrad Rzeszutek Wilk wrote:
> On Wed, Oct 24, 2012 at 06:58:45PM +0200, Roger Pau Monne wrote:
>> This patch implements persistent grants for the xen-blk{front,back}
>> mechanism. The effect of this change is to reduce the number of unmap
>> operations performed, since they cause a (costly) TLB shootdown. This
>> allows the I/O performance to scale better when a large number of VMs
>> are performing I/O.
>>
>> Previously, the blkfront driver was supplied a bvec[] from the request
>> queue. This was granted to dom0; dom0 performed the I/O and wrote
>> directly into the grant-mapped memory and unmapped it; blkfront then
>> removed foreign access for that grant. The cost of unmapping scales
>> badly with the number of CPUs in Dom0. An experiment showed that when
>> Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
>> ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
>> (at which point 650,000 IOPS are being performed in total). If more
>> than 5 guests are used, the performance declines. By 10 guests, only
>> 400,000 IOPS are being performed.
>>
>> This patch improves performance by only unmapping when the connection
>> between blkfront and back is broken.
>>
>> On startup blkfront notifies blkback that it is using persistent
>> grants, and blkback will do the same. If blkback is not capable of
>> persistent mapping, blkfront will still use the same grants, since it
>> is compatible with the previous protocol, and simplifies the code
>> complexity in blkfront.
>>
>> To perform a read, in persistent mode, blkfront uses a separate pool
>> of pages that it maps to dom0. When a request comes in, blkfront
>> transmutes the request so that blkback will write into one of these
>> free pages. Blkback keeps note of which grefs it has already
>> mapped. When a new ring request comes to blkback, it looks to see if
>> it has already mapped that page. If so, it will not map it again. If
>> the page hasn't been previously mapped, it is mapped now, and a record
>> is kept of this mapping. Blkback proceeds as usual. When blkfront is
>> notified that blkback has completed a request, it memcpy's from the
>> shared memory, into the bvec supplied. A record that the {gref, page}
>> tuple is mapped, and not inflight is kept.
>>
>> Writes are similar, except that the memcpy is peformed from the
>> supplied bvecs, into the shared pages, before the request is put onto
>> the ring.
>>
>> Blkback stores a mapping of grefs=>{page mapped to by gref} in
>> a red-black tree. As the grefs are not known apriori, and provide no
>> guarantees on their ordering, we have to perform a search
>> through this tree to find the page, for every gref we receive. This
>> operation takes O(log n) time in the worst case. In blkfront grants
>> are stored using a single linked list.
>>
>> The maximum number of grants that blkback will persistenly map is
>> currently set to RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, to
>> prevent a malicios guest from attempting a DoS, by supplying fresh
>> grefs, causing the Dom0 kernel to map excessively. If a guest
>> is using persistent grants and exceeds the maximum number of grants to
>> map persistenly the newly passed grefs will be mapped and unmaped.
>> Using this approach, we can have requests that mix persistent and
>> non-persistent grants, and we need to handle them correctly.
>> This allows us to set the maximum number of persistent grants to a
>> lower value than RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, although
>> setting it will lead to unpredictable performance.
>>
>> In writing this patch, the question arrises as to if the additional
>> cost of performing memcpys in the guest (to/from the pool of granted
>> pages) outweigh the gains of not performing TLB shootdowns. The answer
>> to that question is `no'. There appears to be very little, if any
>> additional cost to the guest of using persistent grants. There is
>> perhaps a small saving, from the reduced number of hypercalls
>> performed in granting, and ending foreign access.
>>
>> Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
>> Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
>> Cc: <konrad.wilk@oracle.com>
>> Cc: <linux-kernel@vger.kernel.org>
>> ---
>> Changes since v1:
>> * Changed the unmap_seg array to a bitmap.
>> * Only report using persistent grants in blkfront if blkback supports
>> it.
>> * Reword some comments.
>> * Fix a bug when setting the handler, index j was not incremented
>> correctly.
>> * Check that the tree of grants in blkback is not empty before
>> iterating over it when doing the cleanup.
>> * Rebase on top of linux-net.
>
> I fixed the 'new_map = [1|0]' you had in and altered it to use 'true'
> or 'false', but when running some tests (with a 64-bit PV guest) I got it
> to bug.
Thanks for the testing. I'm going to rebase on top of your linux-next
branch and see if I can reproduce it. Did you run any kind of specific
test/benchmark? I've been running with this patch for a long time (on
top of your previous linux-next branch), and I haven't been able to get
it to bug.
> [ 0.000000] Initializing cgroup subsys cpuset
> [ 0.000000] Initializing cgroup subsys cpu
> [ 0.000000] Linux version 3.7.0-rc3upstream-00220-g37b7153 (konrad@build.dumpdata.com) (gcc version 4.4.4 20100503 (Red Hat 4.4.4-2) (GCC) ) #1 SMP Tue Oct 30 12:15:12 EDT 2012
> [ 0.000000] Command line: earlyprintk=xen debug nofb console=tty console=ttyS1,115200n8 xen-pciback.hide=(00:02:00) loglevel=10
> [ 0.000000] Freeing 9d-100 pfn range: 99 pages freed
> [ 0.000000] 1-1 mapping on 9d->100
> [ 0.000000] 1-1 mapping on cf7fb->cfb63
> [ 0.000000] 1-1 mapping on cfd15->cfd70
> [ 0.000000] 1-1 mapping on cfd71->cfef7
> [ 0.000000] 1-1 mapping on cff00->100001
> [ 0.000000] Released 99 pages of unused memory
> [ 0.000000] Set 198317 page(s) to 1-1 mapping
> [ 0.000000] Populating 3e700-3e763 pfn range: 99 pages added
> [ 0.000000] e820: BIOS-provided physical RAM map:
> [ 0.000000] Xen: [mem 0x0000000000000000-0x000000000009cfff] usable
> [ 0.000000] Xen: [mem 0x000000000009d800-0x00000000000fffff] reserved
> [ 0.000000] Xen: [mem 0x0000000000100000-0x000000004d062fff] usable
> [ 0.000000] Xen: [mem 0x000000004d063000-0x00000000cf7fafff] unusable
> [ 0.000000] Xen: [mem 0x00000000cf7fb000-0x00000000cf95ffff] reserved
> [ 0.000000] Xen: [mem 0x00000000cf960000-0x00000000cfb62fff] ACPI NVS
> [ 0.000000] Xen: [mem 0x00000000cfb63000-0x00000000cfd14fff] unusable
> [ 0.000000] Xen: [mem 0x00000000cfd15000-0x00000000cfd61fff] ACPI NVS
> [ 0.000000] Xen: [mem 0x00000000cfd62000-0x00000000cfd6cfff] ACPI data
> [ 0.000000] Xen: [mem 0x00000000cfd6d000-0x00000000cfd6ffff] ACPI NVS
> [ 0.000000] Xen: [mem 0x00000000cfd70000-0x00000000cfd70fff] unusable
> [ 0.000000] Xen: [mem 0x00000000cfd71000-0x00000000cfea8fff] reserved
> [ 0.000000] Xen: [mem 0x00000000cfea9000-0x00000000cfeb9fff] ACPI NVS
> [ 0.000000] Xen: [mem 0x00000000cfeba000-0x00000000cfecafff] reserved
> [ 0.000000] Xen: [mem 0x00000000cfecb000-0x00000000cfecbfff] ACPI NVS
> [ 0.000000] Xen: [mem 0x00000000cfecc000-0x00000000cfedbfff] reserved
> [ 0.000000] Xen: [mem 0x00000000cfedc000-0x00000000cfedcfff] ACPI NVS
> [ 0.000000] Xen: [mem 0x00000000cfedd000-0x00000000cfeddfff] reserved
> [ 0.000000] Xen: [mem 0x00000000cfede000-0x00000000cfee3fff] ACPI NVS
> [ 0.000000] Xen: [mem 0x00000000cfee4000-0x00000000cfef6fff] reserved
> [ 0.000000] Xen: [mem 0x00000000cfef7000-0x00000000cfefffff] unusable
> [ 0.000000] Xen: [mem 0x00000000e0000000-0x00000000efffffff] reserved
> [ 0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
> [ 0.000000] Xen: [mem 0x00000000fec10000-0x00000000fec10fff] reserved
> [ 0.000000] Xen: [mem 0x00000000fed00000-0x00000000fed00fff] reserved
> [ 0.000000] Xen: [mem 0x00000000fed40000-0x00000000fed44fff] reserved
> [ 0.000000] Xen: [mem 0x00000000fed61000-0x00000000fed70fff] reserved
> [ 0.000000] Xen: [mem 0x00000000fed80000-0x00000000fed8ffff] reserved
> [ 0.000000] Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
> [ 0.000000] Xen: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
> [ 0.000000] Xen: [mem 0x0000000100001000-0x000000020effffff] unusable
> [ 0.000000] bootconsole [xenboot0] enabled
> [ 0.000000] NX (Execute Disable) protection: active
> [ 0.000000] DMI 2.6 present.
> [ 0.000000] DMI: System manufacturer System Product Name/F1A75-M, BIOS 0406 06/11/2011
> [ 0.000000] e820: update [mem 0x00000000-0x0000ffff] usable ==> reserved
> [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
> [ 0.000000] No AGP bridge found
> [ 0.000000] e820: last_pfn = 0x4d063 max_arch_pfn = 0x400000000
> [ 0.000000] initial memory mapped: [mem 0x00000000-0x16bcdfff]
> [ 0.000000] Base memory trampoline at [ffff880000097000] 97000 size 24576
> [ 0.000000] init_memory_mapping: [mem 0x00000000-0x4d062fff]
> [ 0.000000] [mem 0x00000000-0x4d062fff] page 4k
> [ 0.000000] kernel direct mapping tables up to 0x4d062fff @ [mem 0x01e21000-0x0208cfff]
> [ 0.000000] xen: setting RW the range 1fd3000 - 208d000
> [ 0.000000] RAMDISK: [mem 0x0208d000-0x16bcdfff]
> [ 0.000000] ACPI: RSDP 00000000000f0450 00024 (v02 ALASKA)
> [ 0.000000] ACPI: XSDT 00000000cfd62068 00054 (v01 ALASKA A M I 01072009 AMI 00010013)
> [ 0.000000] ACPI: FACP 00000000cfd69a68 000F4 (v04 ALASKA A M I 01072009 AMI 00010013)
> [ 0.000000] ACPI BIOS Bug: Warning: Optional FADT field Pm2ControlBlock has zero address or length: 0x0000000000000000/0x1 (20120913/tbfadt-598)
> [ 0.000000] ACPI: DSDT 00000000cfd62150 07917 (v02 ALASKA A M I 00000000 INTL 20051117)
> [ 0.000000] ACPI: FACS 00000000cfedef80 00040
> [ 0.000000] ACPI: APIC 00000000cfd69b60 00072 (v03 ALASKA A M I 01072009 AMI 00010013)
> [ 0.000000] ACPI: MCFG 00000000cfd69bd8 0003C (v01 A M I GMCH945. 01072009 MSFT 00000097)
> [ 0.000000] ACPI: HPET 00000000cfd69c18 00038 (v01 ALASKA A M I 01072009 AMI 00000004)
> [ 0.000000] ACPI: SSDT 00000000cfd69c50 00FD8 (v01 AMD POWERNOW 00000001 AMD 00000001)
> [ 0.000000] ACPI: SSDT 00000000cfd6ac28 01923 (v02 AMD ALIB 00000001 MSFT 04000000)
> [ 0.000000] ACPI: Local APIC address 0xfee00000
> [ 0.000000] NUMA turned off
> [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000004d062fff]
> [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x4d062fff]
> [ 0.000000] NODE_DATA [mem 0x3e75f000-0x3e762fff]
> [ 0.000000] Zone ranges:
> [ 0.000000] DMA [mem 0x00010000-0x00ffffff]
> [ 0.000000] DMA32 [mem 0x01000000-0xffffffff]
> [ 0.000000] Normal empty
> [ 0.000000] Movable zone start for each node
> [ 0.000000] Early memory node ranges
> [ 0.000000] node 0: [mem 0x00010000-0x0009cfff]
> [ 0.000000] node 0: [mem 0x00100000-0x4d062fff]
> [ 0.000000] On node 0 totalpages: 315376
> [ 0.000000] DMA zone: 56 pages used for memmap
> [ 0.000000] DMA zone: 6 pages reserved
> [ 0.000000] DMA zone: 3919 pages, LIFO batch:0
> [ 0.000000] DMA32 zone: 4258 pages used for memmap
> [ 0.000000] DMA32 zone: 307137 pages, LIFO batch:31
> [ 0.000000] ACPI: PM-Timer IO Port: 0x808
> [ 0.000000] ACPI: Local APIC address 0xfee00000
> [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
> [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
> [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
> [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
> [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
> [ 0.000000] ACPI: IOAPIC (id[0x05] address[0xfec00000] gsi_base[0])
> [ 0.000000] IOAPIC[0]: apic_id 5, version 33, address 0xfec00000, GSI 0-23
> [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
> [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
> [ 0.000000] ACPI: IRQ0 used by override.
> [ 0.000000] ACPI: IRQ2 used by override.
> [ 0.000000] ACPI: IRQ9 used by override.
> [ 0.000000] Using ACPI (MADT) for SMP configuration information
> [ 0.000000] ACPI: HPET id: 0xffffffff base: 0xfed00000
> [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs
> [ 0.000000] nr_irqs_gsi: 40
> [ 0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000
> [ 0.000000] PM: Registered nosave memory: 000000000009e000 - 0000000000100000
> [ 0.000000] e820: [mem 0xcff00000-0xdfffffff] available for PCI devices
> [ 0.000000] Booting paravirtualized kernel on Xen
> [ 0.000000] Xen version: 4.1.4-pre (preserve-AD)
> [ 0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:4 nr_node_ids:1
> [ 0.000000] PERCPU: Embedded 28 pages/cpu @ffff88003e000000 s84288 r8192 d22208 u524288
> [ 0.000000] pcpu-alloc: s84288 r8192 d22208 u524288 alloc=1*2097152
> [ 0.000000] pcpu-alloc: [0] 0 1 2 3
> [ 2.763208] Built 1 zonelists in Node order, mobility grouping on. Total pages: 311056
> [ 2.763213] Policy zone: DMA32
> [ 2.763218] Kernel command line: earlyprintk=xen debug nofb console=tty console=ttyS1,115200n8 xen-pciback.hide=(00:02:00) loglevel=10
> [ 2.763656] PID hash table entries: 4096 (order: 3, 32768 bytes)
> [ 2.763663] __ex_table already sorted, skipping sort
> [ 2.808064] software IO TLB [mem 0x38a00000-0x3ca00000] (64MB) mapped at [ffff880038a00000-ffff88003c9fffff]
> [ 2.811300] Memory: 581728k/1261964k available (6413k kernel code, 460k absent, 679776k reserved, 4478k data, 752k init)
> [ 2.811414] Hierarchical RCU implementation.
> [ 2.811419] RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=3.
> [ 2.811432] NR_IRQS:33024 nr_irqs:704 16
> [ 2.811514] xen: sci override: global_irq=9 trigger=0 polarity=1
> [ 2.811518] xen: registering gsi 9 triggering 0 polarity 1
> [ 2.811531] xen: --> pirq=9 -> irq=9 (gsi=9)
> [ 2.811539] xen: acpi sci 9
> [ 2.811546] xen: --> pirq=1 -> irq=1 (gsi=1)
> [ 2.811551] xen: --> pirq=2 -> irq=2 (gsi=2)
> [ 2.811557] xen: --> pirq=3 -> irq=3 (gsi=3)
> [ 2.811562] xen: --> pirq=4 -> irq=4 (gsi=4)
> [ 2.811568] xen: --> pirq=5 -> irq=5 (gsi=5)
> [ 2.811574] xen: --> pirq=6 -> irq=6 (gsi=6)
> [ 2.811579] xen: --> pirq=7 -> irq=7 (gsi=7)
> [ 2.811585] xen: --> pirq=8 -> irq=8 (gsi=8)
> [ 2.811590] xen: --> pirq=10 -> irq=10 (gsi=10)
> [ 2.811596] xen: --> pirq=11 -> irq=11 (gsi=11)
> [ 2.811602] xen: --> pirq=12 -> irq=12 (gsi=12)
> [ 2.811607] xen: --> pirq=13 -> irq=13 (gsi=13)
> [ 2.811613] xen: --> pirq=14 -> irq=14 (gsi=14)
> [ 2.811618] xen: --> pirq=15 -> irq=15 (gsi=15)
> [ 2.813454] Console: colour VGA+ 80x25
> [ 2.818363] console [tty0] enabled
> [ 2.818422] console [ttyS1] enabled, bootconsole disabled
> [ 2.818757] Xen: using vcpuop timer interface
> [ 2.819017] installing Xen timer for CPU 0
> [ 2.819285] tsc: Detected 2899.980 MHz processor
> [ 2.819555] Calibrating delay loop (skipped), value calculated using timer frequency.. 5799.96 BogoMIPS (lpj=2899980)
> [ 2.820151] pid_max: default: 32768 minimum: 301
> [ 2.820479] Security Framework initialized
> [ 2.820728] SELinux: Initializing.
> [ 2.820948] SELinux: Starting in permissive mode
> [ 2.821532] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
> [ 2.822565] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
> [ 2.823519] Mount-cache hash table entries: 256
> [ 2.824126] Initializing cgroup subsys cpuacct
> [ 2.824400] Initializing cgroup subsys freezer
> [ 2.824730] tseg: 00cff00000
> [ 2.824934] CPU: Physical Processor ID: 0
> [ 2.825186] CPU: Processor Core ID: 0
> [ 2.825417] mce: CPU supports 6 MCE banks
> [ 2.825696] Last level iTLB entries: 4KB 512, 2MB 16, 4MB 8
> [ 2.825696] Last level dTLB entries: 4KB 1024, 2MB 128, 4MB 64
> [ 2.825696] tlb_flushall_shift: 5
> [ 2.826625] Freeing SMP alternatives: 24k freed
> [ 2.829704] ACPI: Core revision 20120913
> [ 2.856787] cpu 0 spinlock event irq 41
> [ 2.857062] Performance Events: Broken PMU hardware detected, using software events only.
> [ 2.857602] Failed to access perfctr msr (MSR c0010004 is 0)
> [ 2.858234] MCE: In-kernel MCE decoding enabled.
> [ 2.858603] NMI watchdog: disabled (cpu0): hardware events not enabled
> [ 2.859223] installing Xen timer for CPU 1
> [ 2.859499] cpu 1 spinlock event irq 48
> [ 2.860218] installing Xen timer for CPU 2
> [ 2.860493] cpu 2 spinlock event irq 55
> [ 2.860951] Brought up 3 CPUs
> [ 2.864412] PM: Registering ACPI NVS region [mem 0xcf960000-0xcfb62fff] (2109440 bytes)
> [ 2.865989] PM: Registering ACPI NVS region [mem 0xcfd15000-0xcfd61fff] (315392 bytes)
> [ 2.866464] PM: Registering ACPI NVS region [mem 0xcfd6d000-0xcfd6ffff] (12288 bytes)
> [ 2.866931] PM: Registering ACPI NVS region [mem 0xcfea9000-0xcfeb9fff] (69632 bytes)
> [ 2.867404] PM: Registering ACPI NVS region [mem 0xcfecb000-0xcfecbfff] (4096 bytes)
> [ 2.867861] PM: Registering ACPI NVS region [mem 0xcfedc000-0xcfedcfff] (4096 bytes)
> [ 2.868321] PM: Registering ACPI NVS region [mem 0xcfede000-0xcfee3fff] (24576 bytes)
> [ 2.869081] kworker/u:0 (26) used greatest stack depth: 6120 bytes left
> [ 2.869105] Grant tables using version 2 layout.
> [ 2.869122] Grant table initialized
> [ 2.869164] RTC time: 16:43:55, date: 10/30/12
> [ 2.870366] NET: Registered protocol family 16
> [ 2.871181] kworker/u:0 (30) used greatest stack depth: 5504 bytes left
> [ 2.872440] ACPI: bus type pci registered
> [ 2.873480] dca service started, version 1.12.1
> [ 2.873891] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000)
> [ 2.874438] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in E820
> [ 2.914335] PCI: Using configuration type 1 for base access
> [ 2.932999] bio: create slab <bio-0> at 0
> [ 2.933584] ACPI: Added _OSI(Module Device)
> [ 2.933866] ACPI: Added _OSI(Processor Device)
> [ 2.934145] ACPI: Added _OSI(3.0 _SCP Extensions)
> [ 2.934432] ACPI: Added _OSI(Processor Aggregator Device)
> [ 2.942011] ACPI: EC: Look up EC in DSDT
> [ 2.950054] ACPI: Executed 1 blocks of module-level executable AML code
> [ 2.956530] ACPI: Interpreter enabled
> [ 2.956772] ACPI: (supports S0 S3 S4 S5)
> [ 2.957218] ACPI: Using IOAPIC for interrupt routing
> [ 2.982389] ACPI: No dock devices found.
> [ 2.982650] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
> [ 2.983552] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
> [ 2.984298] PCI host bridge to bus 0000:00
> [ 2.984571] pci_bus 0000:00: root bus resource [bus 00-ff]
> [ 2.984906] pci_bus 0000:00: root bus resource [io 0x0000-0x03af]
> [ 2.985277] pci_bus 0000:00: root bus resource [io 0x03e0-0x0cf7]
> [ 2.985657] pci_bus 0000:00: root bus resource [io 0x03b0-0x03df]
> [ 2.986029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff]
> [ 2.986399] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff]
> [ 2.986810] pci_bus 0000:00: root bus resource [mem 0x000c0000-0x000dffff]
> [ 2.987214] pci_bus 0000:00: root bus resource [mem 0xd0000000-0xffffffff]
> [ 2.987631] pci 0000:00:00.0: [1022:1705] type 00 class 0x060000
> [ 2.988106] pci 0000:00:01.0: [1002:9640] type 00 class 0x030000
> [ 2.988487] pci 0000:00:01.0: reg 10: [mem 0xd0000000-0xdfffffff pref]
> [ 2.988893] pci 0000:00:01.0: reg 14: [io 0xf000-0xf0ff]
> [ 2.989246] pci 0000:00:01.0: reg 18: [mem 0xfeb00000-0xfeb3ffff]
> [ 2.989743] pci 0000:00:01.0: supports D1 D2
> [ 2.990053] pci 0000:00:01.1: [1002:1714] type 00 class 0x040300
> [ 2.990442] pci 0000:00:01.1: reg 10: [mem 0xfeb44000-0xfeb47fff]
> [ 2.990965] pci 0000:00:01.1: supports D1 D2
> [ 2.991343] pci 0000:00:10.0: [1022:7812] type 00 class 0x0c0330
> [ 2.991752] pci 0000:00:10.0: reg 10: [mem 0xfeb4a000-0xfeb4bfff 64bit]
> [ 2.992355] pci 0000:00:10.0: PME# supported from D0 D3hot D3cold
> [ 2.992803] pci 0000:00:10.1: [1022:7812] type 00 class 0x0c0330
> [ 2.993201] pci 0000:00:10.1: reg 10: [mem 0xfeb48000-0xfeb49fff 64bit]
> [ 2.993793] pci 0000:00:10.1: PME# supported from D0 D3hot D3cold
> [ 2.994239] pci 0000:00:11.0: [1022:7801] type 00 class 0x010601
> [ 2.994637] pci 0000:00:11.0: reg 10: [io 0xf140-0xf147]
> [ 2.994982] pci 0000:00:11.0: reg 14: [io 0xf130-0xf133]
> [ 2.995333] pci 0000:00:11.0: reg 18: [io 0xf120-0xf127]
> [ 2.995678] pci 0000:00:11.0: reg 1c: [io 0xf110-0xf113]
> [ 2.996024] pci 0000:00:11.0: reg 20: [io 0xf100-0xf10f]
> [ 2.996374] pci 0000:00:11.0: reg 24: [mem 0xfeb51000-0xfeb517ff]
> [ 2.996850] pci 0000:00:12.0: [1022:7807] type 00 class 0x0c0310
> [ 2.997235] pci 0000:00:12.0: reg 10: [mem 0xfeb50000-0xfeb50fff]
> [ 2.997765] pci 0000:00:12.2: [1022:7808] type 00 class 0x0c0320
> [ 2.998161] pci 0000:00:12.2: reg 10: [mem 0xfeb4f000-0xfeb4f0ff]
> [ 2.998712] pci 0000:00:12.2: supports D1 D2
> [ 2.998977] pci 0000:00:12.2: PME# supported from D0 D1 D2 D3hot
> [ 2.999375] pci 0000:00:13.0: [1022:7807] type 00 class 0x0c0310
> [ 2.999768] pci 0000:00:13.0: reg 10: [mem 0xfeb4e000-0xfeb4efff]
> [ 3.000283] pci 0000:00:13.2: [1022:7808] type 00 class 0x0c0320
> [ 3.000681] pci 0000:00:13.2: reg 10: [mem 0xfeb4d000-0xfeb4d0ff]
> [ 3.001227] pci 0000:00:13.2: supports D1 D2
> [ 3.001495] pci 0000:00:13.2: PME# supported from D0 D1 D2 D3hot
> [ 3.001902] pci 0000:00:14.0: [1022:780b] type 00 class 0x0c0500
> [ 3.002433] pci 0000:00:14.2: [1022:780d] type 00 class 0x040300
> [ 3.002840] pci 0000:00:14.2: reg 10: [mem 0xfeb40000-0xfeb43fff 64bit]
> [ 3.003376] pci 0000:00:14.2: PME# supported from D0 D3hot D3cold
> [ 3.003761] pci 0000:00:14.3: [1022:780e] type 00 class 0x060100
> [ 3.004279] pci 0000:00:14.4: [1022:780f] type 01 class 0x060401
> [ 3.004730] pci 0000:00:14.5: [1022:7809] type 00 class 0x0c0310
> [ 3.005119] pci 0000:00:14.5: reg 10: [mem 0xfeb4c000-0xfeb4cfff]
> [ 3.005641] pci 0000:00:15.0: [1022:43a0] type 01 class 0x060400
> [ 3.006187] pci 0000:00:15.0: supports D1 D2
> [ 3.006515] pci 0000:00:15.1: [1022:43a1] type 01 class 0x060400
> [ 3.007059] pci 0000:00:15.1: supports D1 D2
> [ 3.007403] pci 0000:00:18.0: [1022:1700] type 00 class 0x060000
> [ 3.007868] pci 0000:00:18.1: [1022:1701] type 00 class 0x060000
> [ 3.008320] pci 0000:00:18.2: [1022:1702] type 00 class 0x060000
> [ 3.008778] pci 0000:00:18.3: [1022:1703] type 00 class 0x060000
> [ 3.009271] pci 0000:00:18.4: [1022:1704] type 00 class 0x060000
> [ 3.009716] pci 0000:00:18.5: [1022:1718] type 00 class 0x060000
> [ 3.010167] pci 0000:00:18.6: [1022:1716] type 00 class 0x060000
> [ 3.010615] pci 0000:00:18.7: [1022:1719] type 00 class 0x060000
> [ 3.011117] pci 0000:01:05.0: [9710:9835] type 00 class 0x070002
> [ 3.011513] pci 0000:01:05.0: reg 10: [io 0xe050-0xe057]
> [ 3.011861] pci 0000:01:05.0: reg 14: [io 0xe040-0xe047]
> [ 3.012210] pci 0000:01:05.0: reg 18: [io 0xe030-0xe037]
> [ 3.012563] pci 0000:01:05.0: reg 1c: [io 0xe020-0xe027]
> [ 3.012912] pci 0000:01:05.0: reg 20: [io 0xe010-0xe017]
> [ 3.013259] pci 0000:01:05.0: reg 24: [io 0xe000-0xe00f]
> [ 3.013710] pci 0000:00:14.4: PCI bridge to [bus 01] (subtractive decode)
> [ 3.014118] pci 0000:00:14.4: bridge window [io 0xe000-0xefff]
> [ 3.014496] pci 0000:00:14.4: bridge window [io 0x0000-0x03af] (subtractive decode)
> [ 3.014967] pci 0000:00:14.4: bridge window [io 0x03e0-0x0cf7] (subtractive decode)
> [ 3.015422] pci 0000:00:14.4: bridge window [io 0x03b0-0x03df] (subtractive decode)
> [ 3.015878] pci 0000:00:14.4: bridge window [io 0x0d00-0xffff] (subtractive decode)
> [ 3.016343] pci 0000:00:14.4: bridge window [mem 0x000a0000-0x000bffff] (subtractive decode)
> [ 3.016836] pci 0000:00:14.4: bridge window [mem 0x000c0000-0x000dffff] (subtractive decode)
> [ 3.017326] pci 0000:00:14.4: bridge window [mem 0xd0000000-0xffffffff] (subtractive decode)
> [ 3.017990] pci 0000:02:00.0: [8086:10d3] type 00 class 0x020000
> [ 3.018379] pci 0000:02:00.0: reg 10: [mem 0xfeac0000-0xfeadffff]
> [ 3.018769] pci 0000:02:00.0: reg 14: [mem 0xfea00000-0xfea7ffff]
> [ 3.019167] pci 0000:02:00.0: reg 18: [io 0xd000-0xd01f]
> [ 3.019522] pci 0000:02:00.0: reg 1c: [mem 0xfeae0000-0xfeae3fff]
> [ 3.019964] pci 0000:02:00.0: reg 30: [mem 0xfea80000-0xfeabffff pref]
> [ 3.020508] pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
> [ 3.023962] pci 0000:00:15.0: PCI bridge to [bus 02]
> [ 3.024307] pci 0000:00:15.0: bridge window [io 0xd000-0xdfff]
> [ 3.024689] pci 0000:00:15.0: bridge window [mem 0xfea00000-0xfeafffff]
> [ 3.025293] pci 0000:03:00.0: [1969:1083] type 00 class 0x020000
> [ 3.025712] pci 0000:03:00.0: reg 10: [mem 0xfe900000-0xfe93ffff 64bit]
> [ 3.026139] pci 0000:03:00.0: reg 18: [io 0xc000-0xc07f]
> [ 3.026691] pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> [ 3.028949] pci 0000:00:15.1: PCI bridge to [bus 03]
> [ 3.029290] pci 0000:00:15.1: bridge window [io 0xc000-0xcfff]
> [ 3.029671] pci 0000:00:15.1: bridge window [mem 0xfe900000-0xfe9fffff]
> [ 3.030148] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
> [ 3.030844] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PE20._PRT]
> [ 3.031280] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PE21._PRT]
> [ 3.031782] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0PC._PRT]
> [ 3.032377] pci0000:00: Requesting ACPI _OSC control (0x1d)
> [ 3.032947] pci0000:00: ACPI _OSC control (0x1d) granted
> [ 3.051344] ACPI: PCI Interrupt Link [LN24] (IRQs *24)
> [ 3.051852] ACPI: PCI Interrupt Link [LN25] (IRQs *25)
> [ 3.052329] ACPI: PCI Interrupt Link [LN26] (IRQs *26)
> [ 3.052821] ACPI: PCI Interrupt Link [LN27] (IRQs *27)
> [ 3.053298] ACPI: PCI Interrupt Link [LN28] (IRQs *28)
> [ 3.053787] ACPI: PCI Interrupt Link [LN29] (IRQs *29)
> [ 3.054271] ACPI: PCI Interrupt Link [LN30] (IRQs *30)
> [ 3.054747] ACPI: PCI Interrupt Link [LN31] (IRQs *31)
> [ 3.055243] ACPI: PCI Interrupt Link [LN32] (IRQs *32)
> [ 3.055724] ACPI: PCI Interrupt Link [LN33] (IRQs *33)
> [ 3.056211] ACPI: PCI Interrupt Link [LN34] (IRQs *34)
> [ 3.056692] ACPI: PCI Interrupt Link [LN35] (IRQs *35)
> [ 3.057187] ACPI: PCI Interrupt Link [LN36] (IRQs *36)
> [ 3.057673] ACPI: PCI Interrupt Link [LN37] (IRQs *37)
> [ 3.058165] ACPI: PCI Interrupt Link [LN38] (IRQs *38)
> [ 3.058660] ACPI: PCI Interrupt Link [LN39] (IRQs *39)
> [ 3.059154] ACPI: PCI Interrupt Link [LN40] (IRQs *40)
> [ 3.059643] ACPI: PCI Interrupt Link [LN41] (IRQs *41)
> [ 3.060128] ACPI: PCI Interrupt Link [LN42] (IRQs *42)
> [ 3.060618] ACPI: PCI Interrupt Link [LN43] (IRQs *43)
> [ 3.061095] ACPI: PCI Interrupt Link [LN44] (IRQs *44)
> [ 3.061575] ACPI: PCI Interrupt Link [LN45] (IRQs *45)
> [ 3.062059] ACPI: PCI Interrupt Link [LN46] (IRQs *46)
> [ 3.062543] ACPI: PCI Interrupt Link [LN47] (IRQs *47)
> [ 3.063029] ACPI: PCI Interrupt Link [LN48] (IRQs *48)
> [ 3.063506] ACPI: PCI Interrupt Link [LN49] (IRQs *49)
> [ 3.063989] ACPI: PCI Interrupt Link [LN50] (IRQs *50)
> [ 3.064473] ACPI: PCI Interrupt Link [LN51] (IRQs *51)
> [ 3.064950] ACPI: PCI Interrupt Link [LN52] (IRQs *52)
> [ 3.065440] ACPI: PCI Interrupt Link [LN53] (IRQs *53)
> [ 3.065918] ACPI: PCI Interrupt Link [LN54] (IRQs *54)
> [ 3.066402] ACPI: PCI Interrupt Link [LN55] (IRQs *55)
> [ 3.066889] ACPI: PCI Interrupt Link [LNKA] (IRQs 4 5 7 10 11 14 15) *0
> [ 3.068826] ACPI: PCI Interrupt Link [LNKB] (IRQs 4 5 7 10 11 14 15) *0
> [ 3.069717] ACPI: PCI Interrupt Link [LNKC] (IRQs 4 5 7 10 11 14 15) *0
> [ 3.070606] ACPI: PCI Interrupt Link [LNKD] (IRQs 4 5 7 10 11 14 15) *0
> [ 3.071479] ACPI: PCI Interrupt Link [LNKE] (IRQs 4 5 7 10 11 14 15) *0
> [ 3.072339] ACPI: PCI Interrupt Link [LNKF] (IRQs 4 5 7 10 11 14 15) *0
> [ 3.073195] ACPI: PCI Interrupt Link [LNKG] (IRQs 4 5 7 10 11 14 15) *0
> [ 3.074064] ACPI: PCI Interrupt Link [LNKH] (IRQs 4 5 7 10 11 14 15) *0
> [ 3.075123] xen/balloon: Initialising balloon driver.
> [ 3.076280] xen-balloon: Initialising balloon driver.
> [ 3.076885] xen/balloon: Xen selfballooning driver disabled for domain0.
> [ 3.077545] vgaarb: device added: PCI:0000:00:01.0,decodes=io+mem,owns=io+mem,locks=none
> [ 3.078080] vgaarb: loaded
> [ 3.078268] vgaarb: bridge control possible 0000:00:01.0
> [ 3.078970] ACPI: bus type usb registered
> [ 3.079409] usbcore: registered new interface driver usbfs
> [ 3.079813] usbcore: registered new interface driver hub
> [ 3.080257] usbcore: registered new device driver usb
> [ 3.081098] PCI: Using ACPI for IRQ routing
> [ 3.098106] PCI: pci_cache_line_size set to 64 bytes
> [ 3.098606] e820: reserve RAM buffer [mem 0x0009d000-0x0009ffff]
> [ 3.098972] e820: reserve RAM buffer [mem 0x4d063000-0x4fffffff]
> [ 3.099810] NetLabel: Initializing
> [ 3.100040] NetLabel: domain hash size = 128
> [ 3.100313] NetLabel: protocols = UNLABELED CIPSOv4
> [ 3.100639] NetLabel: unlabeled traffic allowed by default
> [ 3.101361] Switching to clocksource xen
> [ 3.109135] pnp: PnP ACPI init
> [ 3.109362] ACPI: bus type pnp registered
> [ 3.109832] pnp 00:00: [bus 00-ff]
> [ 3.110058] pnp 00:00: [io 0x0cf8-0x0cff]
> [ 3.110319] pnp 00:00: [io 0x0000-0x03af window]
> [ 3.110611] pnp 00:00: [io 0x03e0-0x0cf7 window]
> [ 3.110912] pnp 00:00: [io 0x03b0-0x03df window]
> [ 3.111200] pnp 00:00: [io 0x0d00-0xffff window]
> [ 3.111492] pnp 00:00: [mem 0x000a0000-0x000bffff window]
> [ 3.111826] pnp 00:00: [mem 0x000c0000-0x000dffff window]
> [ 3.112154] pnp 00:00: [mem 0xd0000000-0xffffffff window]
> [ 3.112482] pnp 00:00: [mem 0x00000000 window]
> [ 3.113097] pnp 00:00: Plug and Play ACPI device, IDs PNP0a03 (active)
> [ 3.113537] pnp 00:01: [mem 0xe0000000-0xefffffff]
> [ 3.114213] system 00:01: [mem 0xe0000000-0xefffffff] has been reserved
> [ 3.114626] system 00:01: Plug and Play ACPI device, IDs PNP0c01 (active)
> [ 3.116690] pnp 00:02: [io 0x0010-0x001f]
> [ 3.116955] pnp 00:02: [io 0x0022-0x003f]
> [ 3.117210] pnp 00:02: [io 0x0063]
> [ 3.117439] pnp 00:02: [io 0x0065]
> [ 3.117667] pnp 00:02: [io 0x0067-0x006f]
> [ 3.117932] pnp 00:02: [io 0x0072-0x007f]
> [ 3.118191] pnp 00:02: [io 0x0080]
> [ 3.118415] pnp 00:02: [io 0x0084-0x0086]
> [ 3.118668] pnp 00:02: [io 0x0088]
> [ 3.118910] pnp 00:02: [io 0x008c-0x008e]
> [ 3.119174] pnp 00:02: [io 0x0090-0x009f]
> [ 3.119436] pnp 00:02: [io 0x00a2-0x00bf]
> [ 3.119706] pnp 00:02: [io 0x00b1]
> [ 3.119936] pnp 00:02: [io 0x00e0-0x00ef]
> [ 3.120194] pnp 00:02: [io 0x04d0-0x04d1]
> [ 3.120455] pnp 00:02: [io 0x040b]
> [ 3.120683] pnp 00:02: [io 0x04d6]
> [ 3.120916] pnp 00:02: [io 0x0c00-0x0c01]
> [ 3.121172] pnp 00:02: [io 0x0c14]
> [ 3.121397] pnp 00:02: [io 0x0c50-0x0c51]
> [ 3.121653] pnp 00:02: [io 0x0c52]
> [ 3.121885] pnp 00:02: [io 0x0c6c]
> [ 3.122109] pnp 00:02: [io 0x0c6f]
> [ 3.122335] pnp 00:02: [io 0x0cd0-0x0cd1]
> [ 3.122591] pnp 00:02: [io 0x0cd2-0x0cd3]
> [ 3.122860] pnp 00:02: [io 0x0cd4-0x0cd5]
> [ 3.123120] pnp 00:02: [io 0x0cd6-0x0cd7]
> [ 3.123377] pnp 00:02: [io 0x0cd8-0x0cdf]
> [ 3.123633] pnp 00:02: [io 0x0800-0x089f]
> [ 3.123898] pnp 00:02: [io 0x0000-0xffffffffffffffff disabled]
> [ 3.124249] pnp 00:02: [io 0x0000-0x000f]
> [ 3.124506] pnp 00:02: [io 0x0b20-0x0b3f]
> [ 3.124768] pnp 00:02: [io 0x0900-0x090f]
> [ 3.125026] pnp 00:02: [io 0x0910-0x091f]
> [ 3.125282] pnp 00:02: [io 0xfe00-0xfefe]
> [ 3.125537] pnp 00:02: [io 0x0060-0x005f disabled]
> [ 3.125841] pnp 00:02: [io 0x0064-0x0063 disabled]
> [ 3.126136] pnp 00:02: [mem 0xfec00000-0xfec00fff]
> [ 3.126426] pnp 00:02: [mem 0xfee00000-0xfee00fff]
> [ 3.126724] pnp 00:02: [mem 0xfed80000-0xfed8ffff]
> [ 3.127024] pnp 00:02: [mem 0xfed61000-0xfed70fff]
> [ 3.127321] pnp 00:02: [mem 0xfec10000-0xfec10fff]
> [ 3.127619] pnp 00:02: [mem 0xfed00000-0xfed00fff]
> [ 3.127920] pnp 00:02: [mem 0xff000000-0xffffffff]
> [ 3.128629] system 00:02: [io 0x04d0-0x04d1] has been reserved
> [ 3.129033] system 00:02: [io 0x040b] has been reserved
> [ 3.129358] system 00:02: [io 0x04d6] has been reserved
> [ 3.129680] system 00:02: [io 0x0c00-0x0c01] has been reserved
> [ 3.130042] system 00:02: [io 0x0c14] has been reserved
> [ 3.130365] system 00:02: [io 0x0c50-0x0c51] has been reserved
> [ 3.130727] system 00:02: [io 0x0c52] has been reserved
> [ 3.131046] system 00:02: [io 0x0c6c] has been reserved
> [ 3.131369] system 00:02: [io 0x0c6f] has been reserved
> [ 3.131692] system 00:02: [io 0x0cd0-0x0cd1] has been reserved
> [ 3.132047] system 00:02: [io 0x0cd2-0x0cd3] has been reserved
> [ 3.132401] system 00:02: [io 0x0cd4-0x0cd5] has been reserved
> [ 3.132756] system 00:02: [io 0x0cd6-0x0cd7] has been reserved
> [ 3.133111] system 00:02: [io 0x0cd8-0x0cdf] has been reserved
> [ 3.133460] system 00:02: [io 0x0800-0x089f] has been reserved
> [ 3.133814] system 00:02: [io 0x0b20-0x0b3f] has been reserved
> [ 3.134168] system 00:02: [io 0x0900-0x090f] has been reserved
> [ 3.134524] system 00:02: [io 0x0910-0x091f] has been reserved
> [ 3.134888] system 00:02: [io 0xfe00-0xfefe] has been reserved
> [ 3.135250] system 00:02: [mem 0xfec00000-0xfec00fff] could not be reserved
> [ 3.135651] system 00:02: [mem 0xfee00000-0xfee00fff] has been reserved
> [ 3.136053] system 00:02: [mem 0xfed80000-0xfed8ffff] has been reserved
> [ 3.136437] system 00:02: [mem 0xfed61000-0xfed70fff] has been reserved
> [ 3.136829] system 00:02: [mem 0xfec10000-0xfec10fff] has been reserved
> [ 3.137219] system 00:02: [mem 0xfed00000-0xfed00fff] has been reserved
> [ 3.137608] system 00:02: [mem 0xff000000-0xffffffff] has been reserved
> [ 3.138015] system 00:02: Plug and Play ACPI device, IDs PNP0c02 (active)
> [ 3.138611] pnp 00:03: [io 0x0000-0xffffffffffffffff disabled]
> [ 3.138981] pnp 00:03: [io 0x0300-0x031f]
> [ 3.139249] pnp 00:03: [io 0x0290-0x029f]
> [ 3.139512] pnp 00:03: [io 0x0230-0x023f]
> [ 3.140070] system 00:03: [io 0x0300-0x031f] has been reserved
> [ 3.140431] system 00:03: [io 0x0290-0x029f] has been reserved
> [ 3.140800] system 00:03: [io 0x0230-0x023f] has been reserved
> [ 3.141165] system 00:03: Plug and Play ACPI device, IDs PNP0c02 (active)
> [ 3.141595] pnp 00:04: [dma 4]
> [ 3.141809] pnp 00:04: [io 0x0000-0x000f]
> [ 3.142070] pnp 00:04: [io 0x0081-0x0083]
> [ 3.142330] pnp 00:04: [io 0x0087]
> [ 3.142557] pnp 00:04: [io 0x0089-0x008b]
> [ 3.142825] pnp 00:04: [io 0x008f]
> [ 3.143052] pnp 00:04: [io 0x00c0-0x00df]
> [ 3.143532] pnp 00:04: Plug and Play ACPI device, IDs PNP0200 (active)
> [ 3.143961] pnp 00:05: [io 0x0070-0x0071]
> [ 3.144222] xen: registering gsi 8 triggering 1 polarity 0
> [ 3.144562] pnp 00:05: [irq 8]
> [ 3.145001] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active)
> [ 3.145408] pnp 00:06: [io 0x0061]
> [ 3.145932] pnp 00:06: Plug and Play ACPI device, IDs PNP0800 (active)
> [ 3.146422] pnp 00:07: [io 0x0010-0x001f]
> [ 3.146682] pnp 00:07: [io 0x0022-0x003f]
> [ 3.146977] pnp 00:07: [io 0x0044-0x005f]
> [ 3.147238] pnp 00:07: [io 0x0072-0x007f]
> [ 3.147491] pnp 00:07: [io 0x0080]
> [ 3.147722] pnp 00:07: [io 0x0084-0x0086]
> [ 3.147980] pnp 00:07: [io 0x0088]
> [ 3.148211] pnp 00:07: [io 0x008c-0x008e]
> [ 3.148469] pnp 00:07: [io 0x0090-0x009f]
> [ 3.148734] pnp 00:07: [io 0x00a2-0x00bf]
> [ 3.148988] pnp 00:07: [io 0x00e0-0x00ef]
> [ 3.149250] pnp 00:07: [io 0x04d0-0x04d1]
> [ 3.149697] system 00:07: [io 0x04d0-0x04d1] has been reserved
> [ 3.150061] system 00:07: Plug and Play ACPI device, IDs PNP0c02 (active)
> [ 3.150478] pnp 00:08: [io 0x00f0-0x00ff]
> [ 3.150755] xen: registering gsi 13 triggering 1 polarity 0
> [ 3.151101] pnp 00:08: [irq 13]
> [ 3.151448] pnp 00:08: Plug and Play ACPI device, IDs PNP0c04 (active)
> [ 3.152100] system 00:09: Plug and Play ACPI device, IDs PNP0c02 (active)
> [ 3.152778] pnp 00:0a: [io 0x03f8-0x03ff]
> [ 3.153037] xen: registering gsi 4 triggering 1 polarity 0
> [ 3.153376] pnp 00:0a: [irq 4]
> [ 3.153581] pnp 00:0a: [dma 0 disabled]
> [ 3.153998] pnp 00:0a: Plug and Play ACPI device, IDs PNP0501 (active)
> [ 3.155394] pnp 00:0b: [mem 0xfed00000-0xfed003ff]
> [ 3.155978] pnp 00:0b: Plug and Play ACPI device, IDs PNP0103 (active)
> [ 3.156388] pnp: PnP ACPI: found 12 devices
> [ 3.156653] ACPI: ACPI bus type pnp unregistered
> [ 3.156953] xen-pciback: Error parsing pci_devs_to_hide at "(00:02:00)"
> [ 3.170459] PM-Timer failed consistency check (0x0xffffff) - aborting.
> [ 3.170947] pci 0000:00:14.4: PCI bridge to [bus 01]
> [ 3.171261] pci 0000:00:14.4: bridge window [io 0xe000-0xefff]
> [ 3.171655] pci 0000:00:15.0: PCI bridge to [bus 02]
> [ 3.171974] pci 0000:00:15.0: bridge window [io 0xd000-0xdfff]
> [ 3.172345] pci 0000:00:15.0: bridge window [mem 0xfea00000-0xfeafffff]
> [ 3.172762] pci 0000:00:15.1: PCI bridge to [bus 03]
> [ 3.173067] pci 0000:00:15.1: bridge window [io 0xc000-0xcfff]
> [ 3.173436] pci 0000:00:15.1: bridge window [mem 0xfe900000-0xfe9fffff]
> [ 3.173889] xen: registering gsi 16 triggering 0 polarity 1
> [ 3.174247] xen: --> pirq=16 -> irq=16 (gsi=16)
> [ 3.174547] xen: registering gsi 16 triggering 0 polarity 1
> [ 3.174894] Already setup the GSI :16
> [ 3.175131] pci_bus 0000:00: resource 4 [io 0x0000-0x03af]
> [ 3.175470] pci_bus 0000:00: resource 5 [io 0x03e0-0x0cf7]
> [ 3.176854] pci_bus 0000:00: resource 6 [io 0x03b0-0x03df]
> [ 3.177184] pci_bus 0000:00: resource 7 [io 0x0d00-0xffff]
> [ 3.177514] pci_bus 0000:00: resource 8 [mem 0x000a0000-0x000bffff]
> [ 3.177891] pci_bus 0000:00: resource 9 [mem 0x000c0000-0x000dffff]
> [ 3.178267] pci_bus 0000:00: resource 10 [mem 0xd0000000-0xffffffff]
> [ 3.178649] pci_bus 0000:01: resource 0 [io 0xe000-0xefff]
> [ 3.178994] pci_bus 0000:01: resource 4 [io 0x0000-0x03af]
> [ 3.179326] pci_bus 0000:01: resource 5 [io 0x03e0-0x0cf7]
> [ 3.179672] pci_bus 0000:01: resource 6 [io 0x03b0-0x03df]
> [ 3.180015] pci_bus 0000:01: resource 7 [io 0x0d00-0xffff]
> [ 3.180344] pci_bus 0000:01: resource 8 [mem 0x000a0000-0x000bffff]
> [ 3.180716] pci_bus 0000:01: resource 9 [mem 0x000c0000-0x000dffff]
> [ 3.181081] pci_bus 0000:01: resource 10 [mem 0xd0000000-0xffffffff]
> [ 3.181452] pci_bus 0000:02: resource 0 [io 0xd000-0xdfff]
> [ 3.181792] pci_bus 0000:02: resource 1 [mem 0xfea00000-0xfeafffff]
> [ 3.182168] pci_bus 0000:03: resource 0 [io 0xc000-0xcfff]
> [ 3.182505] pci_bus 0000:03: resource 1 [mem 0xfe900000-0xfe9fffff]
> [ 3.183045] NET: Registered protocol family 2
> [ 3.184322] TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
> [ 3.186023] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
> [ 3.186732] TCP: Hash tables configured (established 262144 bind 65536)
> [ 3.187181] TCP: reno registered
> [ 3.187411] UDP hash table entries: 1024 (order: 3, 32768 bytes)
> [ 3.187803] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)
> [ 3.188405] NET: Registered protocol family 1
> [ 3.188924] RPC: Registered named UNIX socket transport module.
> [ 3.189285] RPC: Registered udp transport module.
> [ 3.189588] RPC: Registered tcp transport module.
> [ 3.189901] RPC: Registered tcp NFSv4.1 backchannel transport module.
> [ 3.190306] pci 0000:00:01.0: Boot video device
> [ 3.190606] xen: registering gsi 18 triggering 0 polarity 1
> [ 3.190967] xen: --> pirq=18 -> irq=18 (gsi=18)
> [ 3.191310] xen: registering gsi 17 triggering 0 polarity 1
> [ 3.191651] xen: --> pirq=17 -> irq=17 (gsi=17)
> [ 3.191987] xen: registering gsi 18 triggering 0 polarity 1
> [ 3.192322] Already setup the GSI :18
> [ 3.779210] xen: registering gsi 17 triggering 0 polarity 1
> [ 3.779560] Already setup the GSI :17
> [ 3.779854] xen: registering gsi 18 triggering 0 polarity 1
> [ 3.780189] Already setup the GSI :18
> [ 3.852950] xen: registering gsi 17 triggering 0 polarity 1
> [ 3.853302] Already setup the GSI :17
> [ 3.853601] xen: registering gsi 18 triggering 0 polarity 1
> [ 3.853944] Already setup the GSI :18
> [ 3.927246] PCI: CLS 64 bytes, default 64
> [ 3.927731] Unpacking initramfs...
> [ 4.409499] Freeing initrd memory: 339204k freed
> [ 4.502233] Machine check injector initialized
> [ 4.504138] microcode: CPU0: patch_level=0x0300000f
> [ 4.504463] microcode: CPU1: patch_level=0x0300000f
> [ 4.504809] microcode: CPU2: patch_level=0x0300000f
> [ 4.505347] microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
> [ 4.506684] audit: initializing netlink socket (disabled)
> [ 4.507060] type=2000 audit(1351615437.471:1): initialized
> [ 4.520849] HugeTLB registered 2 MB page size, pre-allocated 0 pages
> [ 4.521549] VFS: Disk quotas dquot_6.5.2
> [ 4.521865] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
> [ 4.522596] NFS: Registering the id_resolver key type
> [ 4.522935] Key type id_resolver registered
> [ 4.523199] Key type id_legacy registered
> [ 4.523463] NTFS driver 2.1.30 [Flags: R/W].
> [ 4.523927] msgmni has been set to 1798
> [ 4.524230] SELinux: Registering netfilter hooks
> [ 4.526131] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
> [ 4.526572] io scheduler noop registered
> [ 4.526834] io scheduler deadline registered
> [ 4.527129] io scheduler cfq registered (default)
> [ 4.528468] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
> [ 4.529395] input: Power Button as /devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input0
> [ 4.529907] ACPI: Power Button [PWRB]
> [ 4.530299] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1
> [ 4.530782] ACPI: Power Button [PWRF]
> [ 4.592637] GHES: HEST is not enabled!
> [ 4.592910] ioatdma: Intel(R) QuickData Technology Driver 4.00
> [ 4.594402] xen-pciback: backend is vpci
> [ 4.666851] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
> [ 4.688746] 00:0a: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
> [ 4.692137] xen: registering gsi 20 triggering 0 polarity 1
> [ 4.692685] xen: --> pirq=20 -> irq=20 (gsi=20)
> [ 4.715090] 0000:01:05.0: ttyS1 at I/O 0xe050 (irq = 20) is a 16550A
> [ 4.723123] hpet_acpi_add: no address or irqs in _CRS
> [ 4.729073] Non-volatile memory driver v1.3
> [ 4.733807] Linux agpgart interface v0.103
> [ 4.739678] [drm] Initialized drm 1.1.0 20060810
> [ 4.747593] loop: module loaded
> [ 4.751865] libphy: Fixed MDIO Bus: probed
> [ 4.756157] tun: Universal TUN/TAP device driver, 1.6
> [ 4.761408] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
> [ 4.768474] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.6.0-k
> [ 4.778143] ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
> [ 4.786308] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
> [ 4.793112] ehci_hcd: block sizes: qh 104 qtd 96 itd 192 sitd 96
> [ 4.799394] xen: registering gsi 17 triggering 0 polarity 1
> [ 4.805186] Already setup the GSI :17
> [ 4.809041] ehci_hcd 0000:00:12.2: EHCI Host Controller
> [ 4.814827] ehci_hcd 0000:00:12.2: new USB bus registered, assigned bus number 1
> [ 4.822582] QUIRK: Enable AMD PLL fix
> [ 4.826402] ehci_hcd 0000:00:12.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
> [ 4.835422] ehci_hcd 0000:00:12.2: reset hcs_params 0x101505 dbg=1 cc=1 pcc=5 ordered !ppc ports=5
> [ 4.844716] ehci_hcd 0000:00:12.2: reset hcc_params a076 thresh 7 uframes 256/512/1024 park
> [ 4.853444] ehci_hcd 0000:00:12.2: park 0
> [ 4.857628] ehci_hcd 0000:00:12.2: reset command 0080b02 park=3 ithresh=8 period=1024 Reset HALT
> [ 4.866843] ehci_hcd 0000:00:12.2: debug port 1
> [ 4.871568] ehci_hcd 0000:00:12.2: MWI active
> [ 4.876100] ehci_hcd 0000:00:12.2: supports USB remote wakeup
> [ 4.882116] ehci_hcd 0000:00:12.2: irq 17, io mem 0xfeb4f000
> [ 4.887997] ehci_hcd 0000:00:12.2: init command 0010005 (park)=0 ithresh=1 period=512 RUN
> [ 4.901882] ehci_hcd 0000:00:12.2: USB 2.0 started, EHCI 1.00
> [ 4.907965] usb usb1: default language 0x0409
> [ 4.912513] usb usb1: udev 1, busnum 1, minor = 0
> [ 4.917409] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
> [ 4.924454] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
> [ 4.931952] usb usb1: Product: EHCI Host Controller
> [ 4.937024] usb usb1: Manufacturer: Linux 3.7.0-rc3upstream-00220-g37b7153 ehci_hcd
> [ 4.944967] usb usb1: SerialNumber: 0000:00:12.2
> [ 4.950259] usb usb1: usb_probe_device
> [ 4.954193] usb usb1: configuration #1 chosen from 1 choice
> [ 4.959997] usb usb1: adding 1-0:1.0 (config #1, interface 0)
> [ 4.966339] hub 1-0:1.0: usb_probe_interface
> [ 4.970812] hub 1-0:1.0: usb_probe_interface - got id
> [ 4.976063] hub 1-0:1.0: USB hub found
> [ 4.979980] hub 1-0:1.0: 5 ports detected
> [ 4.984155] hub 1-0:1.0: standalone hub
> [ 4.988148] hub 1-0:1.0: no power switching (usb 1.0)
> [ 4.993398] hub 1-0:1.0: individual port over-current protection
> [ 4.999631] hub 1-0:1.0: power on to power good time: 20ms
> [ 5.005333] hub 1-0:1.0: local power source is good
> [ 5.010998] hub 1-0:1.0: trying to enable port power on non-switchable hub
> [ 5.018235] xen: registering gsi 17 triggering 0 polarity 1
> [ 5.024028] Already setup the GSI :17
> [ 5.027880] ehci_hcd 0000:00:13.2: EHCI Host Controller
> [ 5.033663] ehci_hcd 0000:00:13.2: new USB bus registered, assigned bus number 2
> [ 5.041377] ehci_hcd 0000:00:13.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
> [ 5.050398] ehci_hcd 0000:00:13.2: reset hcs_params 0x101505 dbg=1 cc=1 pcc=5 ordered !ppc ports=5
> [ 5.059685] ehci_hcd 0000:00:13.2: reset hcc_params a076 thresh 7 uframes 256/512/1024 park
> [ 5.068427] ehci_hcd 0000:00:13.2: park 0
> [ 5.072610] ehci_hcd 0000:00:13.2: reset command 0080b02 park=3 ithresh=8 period=1024 Reset HALT
> [ 5.081823] ehci_hcd 0000:00:13.2: debug port 1
> [ 5.086547] ehci_hcd 0000:00:13.2: MWI active
> [ 5.091081] ehci_hcd 0000:00:13.2: supports USB remote wakeup
> [ 5.097059] ehci_hcd 0000:00:13.2: irq 17, io mem 0xfeb4d000
> [ 5.102937] ehci_hcd 0000:00:13.2: init command 0010005 (park)=0 ithresh=1 period=512 RUN
> [ 5.116889] ehci_hcd 0000:00:13.2: USB 2.0 started, EHCI 1.00
> [ 5.123004] usb usb2: default language 0x0409
> [ 5.127553] usb usb2: udev 1, busnum 2, minor = 128
> [ 5.132627] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
> [ 5.139668] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
> [ 5.147164] usb usb2: Product: EHCI Host Controller
> [ 5.152236] usb usb2: Manufacturer: Linux 3.7.0-rc3upstream-00220-g37b7153 ehci_hcd
> [ 5.160180] usb usb2: SerialNumber: 0000:00:13.2
> [ 5.165185] hub 1-0:1.0: state 7 ports 5 chg 0000 evt 0000
> [ 5.165547] usb usb2: usb_probe_device
> [ 5.165551] usb usb2: configuration #1 chosen from 1 choice
> [ 5.165570] usb usb2: adding 2-0:1.0 (config #1, interface 0)
> [ 5.165905] hub 2-0:1.0: usb_probe_interface
> [ 5.165908] hub 2-0:1.0: usb_probe_interface - got id
> [ 5.165911] hub 2-0:1.0: USB hub found
> [ 5.165929] hub 2-0:1.0: 5 ports detected
> [ 5.165930] hub 2-0:1.0: standalone hub
> [ 5.165931] hub 2-0:1.0: no power switching (usb 1.0)
> [ 5.165933] hub 2-0:1.0: individual port over-current protection
> [ 5.165935] hub 2-0:1.0: power on to power good time: 20ms
> [ 5.165942] hub 2-0:1.0: local power source is good
> [ 5.166518] hub 2-0:1.0: trying to enable port power on non-switchable hub
> [ 5.166966] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
> [ 5.166969] ohci_hcd: block sizes: ed 80 td 96
> [ 5.167028] xen: registering gsi 18 triggering 0 polarity 1
> [ 5.167033] Already setup the GSI :18
> [ 5.167075] ohci_hcd 0000:00:12.0: OHCI Host Controller
> [ 5.167260] ohci_hcd 0000:00:12.0: new USB bus registered, assigned bus number 3
> [ 5.167338] ohci_hcd 0000:00:12.0: created debug files
> [ 5.167340] ohci_hcd 0000:00:12.0: supports USB remote wakeup
> [ 5.167385] ohci_hcd 0000:00:12.0: irq 18, io mem 0xfeb50000
> [ 5.289966] hub 2-0:1.0: state 7 ports 5 chg 0000 evt 0000
> [ 5.293989] ohci_hcd 0000:00:12.0: OHCI controller state
> [ 5.293995] ohci_hcd 0000:00:12.0: OHCI 1.0, NO legacy support registers, rh state running
> [ 5.293999] ohci_hcd 0000:00:12.0: control 0x283 RWC HCFS=operational CBSR=3
> [ 5.294002] ohci_hcd 0000:00:12.0: cmdstatus 0x00000 SOC=0
> [ 5.294006] ohci_hcd 0000:00:12.0: intrstatus 0x00000004 SF
> [ 5.294009] ohci_hcd 0000:00:12.0: intrenable 0x8000005a MIE RHSC UE RD WDH
> [ 5.294019] ohci_hcd 0000:00:12.0: hcca frame #0005
> [ 5.294022] ohci_hcd 0000:00:12.0: roothub.a 02001205 POTPGT=2 NOCP NPS NDP=5(5)
> [ 5.294025] ohci_hcd 0000:00:12.0: roothub.b 00000000 PPCM=0000 DR=0000
> [ 5.294027] ohci_hcd 0000:00:12.0: roothub.status 00008000 DRWE
> [ 5.294031] ohci_hcd 0000:00:12.0: roothub.portstatus [0] 0x00000100 PPS
> [ 5.294034] ohci_hcd 0000:00:12.0: roothub.portstatus [1] 0x00000100 PPS
> [ 5.294037] ohci_hcd 0000:00:12.0: roothub.portstatus [2] 0x00000100 PPS
> [ 5.294041] ohci_hcd 0000:00:12.0: roothub.portstatus [3] 0x00000100 PPS
> [ 5.294044] ohci_hcd 0000:00:12.0: roothub.portstatus [4] 0x00000100 PPS
> [ 5.294080] usb usb3: default language 0x0409
> [ 5.294093] usb usb3: udev 1, busnum 3, minor = 256
> [ 5.294095] usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
> [ 5.294097] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
> [ 5.294098] usb usb3: Product: OHCI Host Controller
> [ 5.294100] usb usb3: Manufacturer: Linux 3.7.0-rc3upstream-00220-g37b7153 ohci_hcd
> [ 5.294101] usb usb3: SerialNumber: 0000:00:12.0
> [ 5.294371] usb usb3: usb_probe_device
> [ 5.294374] usb usb3: configuration #1 chosen from 1 choice
> [ 5.294386] usb usb3: adding 3-0:1.0 (config #1, interface 0)
> [ 5.294481] hub 3-0:1.0: usb_probe_interface
> [ 5.294482] hub 3-0:1.0: usb_probe_interface - got id
> [ 5.294484] hub 3-0:1.0: USB hub found
> [ 5.294492] hub 3-0:1.0: 5 ports detected
> [ 5.294493] hub 3-0:1.0: standalone hub
> [ 5.294495] hub 3-0:1.0: no power switching (usb 1.0)
> [ 5.294496] hub 3-0:1.0: no over-current protection
> [ 5.294497] hub 3-0:1.0: power on to power good time: 4ms
> [ 5.294505] hub 3-0:1.0: local power source is good
> [ 5.295074] hub 3-0:1.0: trying to enable port power on non-switchable hub
> [ 5.295131] ehci_hcd 0000:00:12.2: HS companion for 0000:00:12.0
> [ 5.295174] xen: registering gsi 18 triggering 0 polarity 1
> [ 5.295178] Already setup the GSI :18
> [ 5.295219] ohci_hcd 0000:00:13.0: OHCI Host Controller
> [ 5.295337] ohci_hcd 0000:00:13.0: new USB bus registered, assigned bus number 4
> [ 5.295397] ohci_hcd 0000:00:13.0: created debug files
> [ 5.295399] ohci_hcd 0000:00:13.0: supports USB remote wakeup
> [ 5.295409] ohci_hcd 0000:00:13.0: irq 18, io mem 0xfeb4e000
> [ 5.549971] hub 3-0:1.0: state 7 ports 5 chg 0000 evt 0000
> [ 5.553975] ohci_hcd 0000:00:13.0: OHCI controller state
> [ 5.553980] ohci_hcd 0000:00:13.0: OHCI 1.0, NO legacy support registers, rh state running
> [ 5.553984] ohci_hcd 0000:00:13.0: control 0x283 RWC HCFS=operational CBSR=3
> [ 5.553987] ohci_hcd 0000:00:13.0: cmdstatus 0x00000 SOC=0
> [ 5.553990] ohci_hcd 0000:00:13.0: intrstatus 0x00000004 SF
> [ 5.553993] ohci_hcd 0000:00:13.0: intrenable 0x8000005a MIE RHSC UE RD WDH
> [ 5.554004] ohci_hcd 0000:00:13.0: hcca frame #0005
> [ 5.554008] ohci_hcd 0000:00:13.0: roothub.a 02001205 POTPGT=2 NOCP NPS NDP=5(5)
> [ 5.554011] ohci_hcd 0000:00:13.0: roothub.b 00000000 PPCM=0000 DR=0000
> [ 5.554014] ohci_hcd 0000:00:13.0: roothub.status 00008000 DRWE
> [ 5.554018] ohci_hcd 0000:00:13.0: roothub.portstatus [0] 0x00000100 PPS
> [ 5.554021] ohci_hcd 0000:00:13.0: roothub.portstatus [1] 0x00000100 PPS
> [ 5.554025] ohci_hcd 0000:00:13.0: roothub.portstatus [2] 0x00000100 PPS
> [ 5.554028] ohci_hcd 0000:00:13.0: roothub.portstatus [3] 0x00000100 PPS
> [ 5.554031] ohci_hcd 0000:00:13.0: roothub.portstatus [4] 0x00000100 PPS
> [ 5.554051] usb usb4: default language 0x0409
> [ 5.554064] usb usb4: udev 1, busnum 4, minor = 384
> [ 5.554066] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
> [ 5.554068] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
> [ 5.554070] usb usb4: Product: OHCI Host Controller
> [ 5.554071] usb usb4: Manufacturer: Linux 3.7.0-rc3upstream-00220-g37b7153 ohci_hcd
> [ 5.554072] usb usb4: SerialNumber: 0000:00:13.0
> [ 5.554538] usb usb4: usb_probe_device
> [ 5.554542] usb usb4: configuration #1 chosen from 1 choice
> [ 5.554559] usb usb4: adding 4-0:1.0 (config #1, interface 0)
> [ 5.554665] hub 4-0:1.0: usb_probe_interface
> [ 5.554667] hub 4-0:1.0: usb_probe_interface - got id
> [ 5.554669] hub 4-0:1.0: USB hub found
> [ 5.554678] hub 4-0:1.0: 5 ports detected
> [ 5.554679] hub 4-0:1.0: standalone hub
> [ 5.554681] hub 4-0:1.0: no power switching (usb 1.0)
> [ 5.554682] hub 4-0:1.0: no over-current protection
> [ 5.554683] hub 4-0:1.0: power on to power good time: 4ms
> [ 5.554691] hub 4-0:1.0: local power source is good
> [ 5.555283] hub 4-0:1.0: trying to enable port power on non-switchable hub
> [ 5.555346] ehci_hcd 0000:00:13.2: HS companion for 0000:00:13.0
> [ 5.555389] xen: registering gsi 18 triggering 0 polarity 1
> [ 5.555394] Already setup the GSI :18
> [ 5.555433] ohci_hcd 0000:00:14.5: OHCI Host Controller
> [ 5.555598] ohci_hcd 0000:00:14.5: new USB bus registered, assigned bus number 5
> [ 5.555670] ohci_hcd 0000:00:14.5: created debug files
> [ 5.555672] ohci_hcd 0000:00:14.5: supports USB remote wakeup
> [ 5.555685] ohci_hcd 0000:00:14.5: irq 18, io mem 0xfeb4c000
> [ 5.809859] hub 4-0:1.0: state 7 ports 5 chg 0000 evt 0000
> [ 5.813841] ohci_hcd 0000:00:14.5: OHCI controller state
> [ 5.813847] ohci_hcd 0000:00:14.5: OHCI 1.0, NO legacy support registers, rh state running
> [ 5.813851] ohci_hcd 0000:00:14.5: control 0x283 RWC HCFS=operational CBSR=3
> [ 5.813854] ohci_hcd 0000:00:14.5: cmdstatus 0x00000 SOC=0
> [ 5.813857] ohci_hcd 0000:00:14.5: intrstatus 0x00000004 SF
> [ 5.813860] ohci_hcd 0000:00:14.5: intrenable 0x8000005a MIE RHSC UE RD WDH
> [ 5.813871] ohci_hcd 0000:00:14.5: hcca frame #0004
> [ 5.813874] ohci_hcd 0000:00:14.5: roothub.a 02001202 POTPGT=2 NOCP NPS NDP=2(2)
> [ 5.813876] ohci_hcd 0000:00:14.5: roothub.b 00000000 PPCM=0000 DR=0000
> [ 5.813880] ohci_hcd 0000:00:14.5: roothub.status 00008000 DRWE
> [ 5.813884] ohci_hcd 0000:00:14.5: roothub.portstatus [0] 0x00000100 PPS
> [ 5.813886] ohci_hcd 0000:00:14.5: roothub.portstatus [1] 0x00000100 PPS
> [ 5.813923] usb usb5: default language 0x0409
> [ 5.813935] usb usb5: udev 1, busnum 5, minor = 512
> [ 5.813938] usb usb5: New USB device found, idVendor=1d6b, idProduct=0001
> [ 5.813939] usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
> [ 5.813941] usb usb5: Product: OHCI Host Controller
> [ 5.813942] usb usb5: Manufacturer: Linux 3.7.0-rc3upstream-00220-g37b7153 ohci_hcd
> [ 5.813944] usb usb5: SerialNumber: 0000:00:14.5
> [ 5.814220] usb usb5: usb_probe_device
> [ 5.814223] usb usb5: configuration #1 chosen from 1 choice
> [ 5.814238] usb usb5: adding 5-0:1.0 (config #1, interface 0)
> [ 5.814373] hub 5-0:1.0: usb_probe_interface
> [ 5.814375] hub 5-0:1.0: usb_probe_interface - got id
> [ 5.814377] hub 5-0:1.0: USB hub found
> [ 5.814390] hub 5-0:1.0: 2 ports detected
> [ 5.814392] hub 5-0:1.0: standalone hub
> [ 5.814393] hub 5-0:1.0: no power switching (usb 1.0)
> [ 5.814394] hub 5-0:1.0: no over-current protection
> [ 5.814396] hub 5-0:1.0: power on to power good time: 4ms
> [ 5.814405] hub 5-0:1.0: local power source is good
> [ 5.814425] hub 5-0:1.0: trying to enable port power on non-switchable hub
> [ 5.814584] uhci_hcd: USB Universal Host Controller Interface driver
> [ 6.009259] hub 5-0:1.0: state 7 ports 2 chg 0000 evt 0000
> [ 6.009372] usbcore: registered new interface driver usblp
> [ 6.009676] i8042: PNP: No PS/2 controller found. Probing ports directly.
> [ 6.010397] serio: i8042 KBD port at 0x60,0x64 irq 1
> [ 6.010409] serio: i8042 AUX port at 0x60,0x64 irq 12
> [ 6.010691] mousedev: PS/2 mouse device common for all mice
> [ 6.011327] rtc_cmos 00:05: RTC can wake from S4
> [ 6.011618] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0
> [ 6.011662] rtc0: alarms up to one month, y3k, 114 bytes nvram
> [ 6.011868] EFI Variables Facility v0.08 2004-May-17
> [ 6.011957] zram: num_devices not specified. Using default: 1
> [ 6.011958] zram: Creating 1 devices ...
> [ 6.077064] Netfilter messages via NETLINK v0.30.
> [ 6.082006] nf_conntrack version 0.5.0 (7194 buckets, 28776 max)
> [ 6.088334] ctnetlink v0.93: registering with nfnetlink.
> [ 6.094201] ip_tables: (C) 2000-2006 Netfilter Core Team
> [ 6.099817] TCP: cubic registered
> [ 6.103279] Initializing XFRM netlink socket
> [ 6.107873] NET: Registered protocol family 10
> [ 6.112784] ip6_tables: (C) 2000-2006 Netfilter Core Team
> [ 6.119125] sit: IPv6 over IPv4 tunneling driver
> [ 6.124870] NET: Registered protocol family 17
> [ 6.129593] Key type dns_resolver registered
> [ 6.135108] PM: Hibernation image not present or could not be loaded.
> [ 6.141853] registered taskstats version 1
> [ 6.146837] Magic number: 0:661:743
> [ 6.151899] Freeing unused kernel memory: 752k freed
> [ 6.157248] Write protecting the kernel read-only data: 10240k
> [ 6.169212] Freeing unused kernel memory: 1768k freed
> [ 6.175129] Freeing unused kernel memory: 168k freed
> [ 6.188257] consoletype (1257) used greatest stack depth: 5272 bytes left
> [ 6.524961] modprobe (1286) used greatest stack depth: 5256 bytes left
> [ 6.547403] core_filesystem (1258) used greatest stack depth: 4952 bytes left
> [ 6.592358] Initialising Xen virtual ethernet driver.
> [ 6.722568] wmi: Mapper loaded
> [ 6.796831] xen: registering gsi 17 triggering 0 polarity 1
> [ 6.802666] Already setup the GSI :17
> [ 6.808260] e1000e: Intel(R) PRO/1000 Network Driver - 2.1.4-k
> [ 6.814354] e1000e: Copyright(c) 1999 - 2012 Intel Corporation.
> [ 6.816573] SCSI subsystem initialized
> [ 6.824498] e1000e 0000:02:00.0: Disabling ASPM L0s L1
> [ 6.824524] xen: registering gsi 16 triggering 0 polarity 1
> [ 6.824531] Already setup the GSI :16
> [ 6.824740] e1000e 0000:02:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
> [ 6.859455] [drm] radeon defaulting to kernel modesetting.
> [ 6.866428] [drm] radeon kernel modesetting enabled.
> [ 6.873121] xen: registering gsi 18 triggering 0 polarity 1
> [ 6.873126] Already setup the GSI :18
> [ 6.883269] atl1c 0000:03:00.0: version 1.0.1.0-NAPI
> [ 6.883447] [drm] initializing kernel modesetting (SUMO 0x1002:0x9640 0x1043:0x84C8).
> [ 6.883509] [drm] register mmio base: 0xFEB00000
> [ 6.883511] [drm] register mmio size: 262144
> [ 6.883646] ATOM BIOS: General
> [ 6.883717] radeon 0000:00:01.0: VRAM: 512M 0x0000000000000000 - 0x000000001FFFFFFF (512M used)
> [ 6.883720] radeon 0000:00:01.0: GTT: 512M 0x0000000020000000 - 0x000000003FFFFFFF
> [ 6.883722] [drm] Detected VRAM RAM=512M, BAR=256M
> [ 6.883725] [drm] RAM width 32bits DDR
> [ 6.884463] [TTM] Zone kernel: Available graphics memory: 461822 kiB
> [ 6.884466] [TTM] Initializing pool allocator
> [ 6.884476] [TTM] Initializing DMA pool allocator
> [ 6.884529] [drm] radeon: 512M of VRAM memory ready
> [ 6.884531] [drm] radeon: 512M of GTT memory ready.
> [ 6.884622] [drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
> [ 6.884625] [drm] Driver supports precise vblank timestamp query.
> [ 6.884802] radeon 0000:00:01.0: radeon: using MSI.
> [ 6.884865] [drm] radeon: irq initialized.
> [ 6.884871] [drm] GART: num cpu pages 131072, num gpu pages 131072
> [ 6.895835] [drm] Loading SUMO Microcode
> [ 6.933742] ACPI: bus type scsi registered
> [ 6.941860] e1000e 0000:02:00.0 eth1: (PCI Express:2.5GT/s:Width x1) 00:1b:21:ab:c6:12
> [ 6.941862] e1000e 0000:02:00.0 eth1: Intel(R) PRO/1000 Network Connection
> [ 6.941882] e1000e 0000:02:00.0 eth1: MAC: 3, PHY: 8, PBA No: E46981-005
> [ 6.997612] ip (1909) used greatest stack depth: 3896 bytes left
> [ 7.048540] libata version 3.00 loaded.
> [ 7.144568] [drm] PCIE GART of 512M enabled (table at 0x0000000000040000).
> [ 7.151898] radeon 0000:00:01.0: WB enabled
> [ 7.156253] radeon 0000:00:01.0: fence driver on ring 0 use gpu addr 0x0000000020000c00 and cpu addr 0xffff880023235c00
> [ 7.185375] [drm] ring test on 0 succeeded in 1 usecs
> [ 7.191817] [drm] ib test on ring 0 succeeded in 0 usecs
> [ 7.209657] [drm] Radeon Display Connectors
> [ 7.214052] [drm] Connector 0:
> [ 7.217253] [drm] VGA-1
> [ 7.220007] [drm] HPD2
> [ 7.222674] [drm] DDC: 0x6440 0x6440 0x6444 0x6444 0x6448 0x6448 0x644c 0x644c
> [ 7.230416] [drm] Encoders:
> [ 7.233539] [drm] CRT1: INTERNAL_UNIPHY2
> [ 7.238001] [drm] CRT1: NUTMEG
> [ 7.238003] [drm] Connector 1:
> [ 7.238005] [drm] HDMI-A-1
> [ 7.238006] [drm] HPD1
> [ 7.238009] [drm] DDC: 0x6430 0x6430 0x6434 0x6434 0x6438 0x6438 0x643c 0x643c
> [ 7.238011] [drm] Encoders:
> [ 7.238011] [drm] DFP1: INTERNAL_UNIPHY2
> [ 7.255949] [drm] Internal thermal controller without fan control
> [ 7.259955] [drm] radeon: power management initialized
> [ 7.335529] No connectors reported connected with modes
> [ 7.340968] [drm] Cannot find any crtc or sizes - going 1024x768
> [ 7.350906] [drm] fb mappable at 0xD0142000
> [ 7.355261] [drm] vram apper at 0xD0000000
> [ 7.359523] [drm] size 3145728
> [ 7.362716] [drm] fb depth is 24
> [ 7.366146] [drm] pitch is 4096
> [ 7.370065] fbcon: radeondrmfb (fb0) is primary device
> [ 7.376982] ttyS1: 1 input overrun(s)
> [ 7.411210] Console: switching to colour frame buffer device 128x48
> [ 7.424071] fb0: radeondrmfb frame buffer device
> [ 7.428949] drm: registered panic notifier
> [ 7.433284] [drm] Initialized radeon 2.24.0 20080528 for 0000:00:01.0 on minor 0
> [ 7.441164] ahci 0000:00:11.0: version 3.0
> [ 7.445546] xen: registering gsi 19 triggering 0 polarity 1
> [ 7.451464] xen: --> pirq=19 -> irq=19 (gsi=19)
> [ 7.456473] ahci 0000:00:11.0: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
> [ 7.465104] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part
> [ 7.478140] scsi0 : ahci
> [ 7.481629] scsi1 : ahci
> [ 7.484929] scsi2 : ahci
> [ 7.488287] scsi3 : ahci
> [ 7.491496] scsi4 : ahci
> [ 7.494907] scsi5 : ahci
> [ 7.498649] ata1: SATA max UDMA/133 abar m2048@0xfeb51000 port 0xfeb51100 irq 71
> [ 7.507585] ata2: SATA max UDMA/133 abar m2048@0xfeb51000 port 0xfeb51180 irq 71
> [ 7.515392] ata3: SATA max UDMA/133 abar m2048@0xfeb51000 port 0xfeb51200 irq 71
> [ 7.523177] ata4: SATA max UDMA/133 abar m2048@0xfeb51000 port 0xfeb51280 irq 71
> [ 7.530982] ata5: SATA max UDMA/133 abar m2048@0xfeb51000 port 0xfeb51300 irq 71
> [ 7.538786] ata6: SATA max UDMA/133 abar m2048@0xfeb51000 port 0xfeb51380 irq 71
> [ 7.850775] ata1: SATA link down (SStatus 0 SControl 300)
> [ 7.856613] ata2: SATA link down (SStatus 0 SControl 300)
> [ 7.862497] ata6: SATA link down (SStatus 0 SControl 300)
> [ 7.871228] ata3: SATA link down (SStatus 0 SControl 300)
> [ 7.879887] ata5: SATA link down (SStatus 0 SControl 300)
> [ 8.038131] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
> [ 8.062593] ata4.00: ATA-7: WDC WD800AAJS-18TDA0, 01.00A03, max UDMA/133
> [ 8.072417] ata4.00: 156250000 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
> [ 8.082336] ata4.00: failed to get Identify Device Data, Emask 0x1
> [ 8.092297] ata4.00: failed to get Identify Device Data, Emask 0x1
> [ 8.100682] ata4.00: configured for UDMA/133
> [ 8.107497] scsi 3:0:0:0: Direct-Access ATA WDC WD800AAJS-18 01.0 PQ: 0 ANSI: 5
> [ 8.129843] sd 3:0:0:0: [sda] 156250000 512-byte logical blocks: (80.0 GB/74.5 GiB)
> [ 8.140007] sd 3:0:0:0: [sda] Write Protect is off
> [ 8.146945] sd 3:0:0:0: [sda] Mode Sense: 00 3a 00 00
> [ 8.154244] sd 3:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> [ 8.199771] sda: sda1 sda2 sda3 sda4
> [ 8.210772] sd 3:0:0:0: [sda] Attached SCSI disk
> [ 8.224663] sd 3:0:0:0: Attached scsi generic sg0 type 0
> [ 8.944409] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
> [ 8.963120] device eth0 entered promiscuous mode
> [ 9.181754] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
> [ 10.053518] atl1c 0000:03:00.0: atl1c: eth0 NIC Link is Up<1000 Mbps Full Duplex>
> [ 10.064785] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [ 11.555156] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> [ 11.566749] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
> [ 13.964892] switch: port 1(eth0) entered forwarding state
> [ 13.973524] switch: port 1(eth0) entered forwarding state
> [ 16.232334] Loading iSCSI transport class v2.0-870.
> [ 16.248793] iscsi: registered transport (tcp)
> [ 16.306470] Event-channel device installed.
> [ 19.630065] mount.nfs (3143) used greatest stack depth: 3224 bytes left
> [ 21.157586] device-mapper: ioctl: 4.23.0-ioctl (2012-07-25) initialised: dm-devel@redhat.com
> [ 21.170420] device-mapper: multipath: version 1.5.0 loaded
> [ 21.450531] scsi6 : iSCSI Initiator over TCP/IP
> [ 21.731710] scsi 6:0:0:0: Direct-Access LIO-ORG IBLOCK 4.0 PQ: 0 ANSI: 5
> [ 21.747167] sd 6:0:0:0: [sdb] 503316480 512-byte logical blocks: (257 GB/240 GiB)
> [ 21.747211] sd 6:0:0:0: Attached scsi generic sg1 type 0
> [ 21.748648] scsi 6:0:0:1: Direct-Access LIO-ORG IBLOCK 4.0 PQ: 0 ANSI: 5
> [ 21.752968] sd 6:0:0:1: [sdc] 167772160 512-byte logical blocks: (85.8 GB/80.0 GiB)
> [ 21.753042] sd 6:0:0:1: Attached scsi generic sg2 type 0
> [ 21.755647] sd 6:0:0:1: [sdc] Write Protect is off
> [ 21.755655] sd 6:0:0:1: [sdc] Mode Sense: 2f 00 00 00
> [ 21.756638] sd 6:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
> [ 21.761219] scsi 6:0:0:2: Direct-Access LIO-ORG IBLOCK 4.0 PQ: 0 ANSI: 5
> [ 21.770871] sd 6:0:0:2: Attached scsi generic sg3 type 0
> [ 21.771168] sd 6:0:0:2: [sdd] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
> [ 21.775163] ttyS1: 1 input overrun(s)
> [ 21.776400] sd 6:0:0:2: [sdd] Write Protect is off
> [ 21.776405] sd 6:0:0:2: [sdd] Mode Sense: 2f 00 00 00
> [ 21.776884] sd 6:0:0:2: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
> [ 21.780001] sdd: unknown partition table
> [ 21.781272] sd 6:0:0:2: [sdd] Attached SCSI disk
> [ 21.804226] sdc: sdc1 sdc2 < sdc5 >
> [ 21.806353] sd 6:0:0:1: [sdc] Attached SCSI disk
> [ 21.918632] sd 6:0:0:0: [sdb] Write Protect is off
> [ 21.926413] sd 6:0:0:0: [sdb] Mode Sense: 2f 00 00 00
> [ 21.935417] sd 6:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
> [ 21.963859] sdb: unknown partition table
> [ 21.974284] sd 6:0:0:0: [sdb] Attached SCSI disk
> [ 28.589544] bio: create slab <bio-1> at 1
> [ 28.997728] switch: port 1(eth0) entered forwarding state
> [ 57.703132] device vif1.0 entered promiscuous mode
> [ 57.715328] IPv6: ADDRCONF(NETDEV_UP): vif1.0: link is not ready
> [ 59.760909] IPv6: ADDRCONF(NETDEV_CHANGE): vif1.0: link becomes ready
> [ 59.770384] switch: port 2(vif1.0) entered forwarding state
> [ 59.778900] switch: port 2(vif1.0) entered forwarding state
> [ 59.876529] xen-blkback:ring-ref 10, event-channel 18, protocol 2 (x86_32-abi) persistent grants
> [ 65.135453] switch: port 2(vif1.0) entered disabled state
> [ 65.145638] device vif1.0 left promiscuous mode
> [ 65.154458] switch: port 2(vif1.0) entered disabled state
> [ 70.836602] device vif2.0 entered promiscuous mode
> [ 70.848191] IPv6: ADDRCONF(NETDEV_UP): vif2.0: link is not ready
> [ 72.285945] IPv6: ADDRCONF(NETDEV_CHANGE): vif2.0: link becomes ready
> [ 72.295310] switch: port 2(vif2.0) entered forwarding state
> [ 72.303597] switch: port 2(vif2.0) entered forwarding state
> [ 72.403140] xen-blkback:ring-ref 10, event-channel 11, protocol 1 (x86_64-abi) persistent grants
> [ 72.544101] ------------[ cut here ]------------
> [ 72.552932] kernel BUG at /home/konrad/linux/drivers/block/xen-blkback/blkback.c:589!
> [ 72.563680] invalid opcode: 0000 [#1] SMP
> [ 72.570865] Modules linked in: dm_multipath dm_mod xen_evtchn iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi libcrc32c crc32c sg sd_mod ahci libahci libata radeon e1000e scsi_mod atl1c fbcon ttm tileblit font bitblit softcursor drm_kms_helper wmi xen_blkfront xen_netfront fb_sys_fops sysimgblt sysfillrect syscopyarea xenfs xen_privcmd [last unloaded: dump_dma]
> [ 72.617251] CPU 0
> [ 72.619173] Pid: 3823, comm: blkback.2.xvda Tainted: G O 3.7.0-rc3upstream-00220-g37b7153 #1 System manufacturer System Product Name/F1A75-M
> [ 72.641606] RIP: e030:[<ffffffff81409766>] [<ffffffff81409766>] xen_blkbk_map+0x696/0x6e0
> [ 72.653181] RSP: e02b:ffff880027dd3728 EFLAGS: 00010246
> [ 72.661651] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
> [ 72.672090] RDX: ffff8800232d7f40 RSI: 0000000000000000 RDI: ffff880027dd3d88
> [ 72.682437] RBP: ffff880027dd39e8 R08: 0000000000000000 R09: 0000000000000000
> [ 72.692835] R10: 0000000000000001 R11: dead000000200200 R12: 0000000000000000
> [ 72.703261] R13: 0000000000000000 R14: ffff88002b5e7070 R15: 0000000000000000
> [ 72.713560] FS: 00007f6cc62d5700(0000) GS:ffff88003e000000(0000) knlGS:0000000000000000
> [ 72.724921] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 72.733811] CR2: 00000000006dd384 CR3: 0000000027b8f000 CR4: 0000000000000660
> [ 72.744131] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 72.754517] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 72.764882] Process blkback.2.xvda (pid: 3823, threadinfo ffff880027dd2000, task ffff88002bb15040)
> [ 72.777140] Stack:
> [ 72.782184] ffff880027dd3738 ffff880026081af0 ffff880027dd3798 ffffffff810ac7df
> [ 72.792960] ffff880027dd3798 ffffffff8104c506 0000000000000117 ffff88002160d030
> [ 72.803712] ffff880027dd3a38 ffff88002b5e7120 ffff88002160d000 ffff880027dd3d88
> [ 72.814469] Call Trace:
> [ 72.820094] [<ffffffff810ac7df>] ? __queue_work+0xff/0x420
> [ 72.829023] [<ffffffff8104c506>] ? xen_spin_lock_flags+0xb6/0x120
> [ 72.838462] [<ffffffff810acb61>] ? queue_work_on+0x31/0x50
> [ 72.847279] [<ffffffff81636eb9>] ? _raw_spin_unlock_irqrestore+0x19/0x30
> [ 72.854408] ttyS1: 2 input overrun(s)
> [ 72.864352] [<ffffffff8107e068>] ? pvclock_clocksource_read+0x58/0xd0
> [ 72.874471] [<ffffffff812ea543>] ? cpumask_next_and+0x23/0x40
> [ 72.883656] [<ffffffff812ea543>] ? cpumask_next_and+0x23/0x40
> [ 72.892716] [<ffffffff810cc5e7>] ? update_sd_lb_stats+0x157/0x6c0
> [ 72.902178] [<ffffffff81636e90>] ? _raw_spin_lock_irq+0x20/0x30
> [ 72.911486] [<ffffffff810cd441>] ? find_busiest_group+0x31/0x4d0
> [ 72.920849] [<ffffffff81409e87>] dispatch_rw_block_io+0x377/0x600
> [ 72.930187] [<ffffffff8107e068>] ? pvclock_clocksource_read+0x58/0xd0
> [ 72.939957] [<ffffffff8103e0c0>] ? xen_mc_flush+0xc0/0x1f0
> [ 72.948743] [<ffffffff8103c9e9>] ? xen_end_context_switch+0x19/0x20
> [ 72.958251] [<ffffffff8107e068>] ? pvclock_clocksource_read+0x58/0xd0
> [ 72.967917] [<ffffffff8107e068>] ? pvclock_clocksource_read+0x58/0xd0
> [ 72.977617] [<ffffffff8107e068>] ? pvclock_clocksource_read+0x58/0xd0
> [ 72.987474] [<ffffffff81044359>] ? xen_clocksource_read+0x39/0x50
> [ 72.997278] [<ffffffff8104c506>] ? xen_spin_lock_flags+0xb6/0x120
> [ 73.006476] [<ffffffff8140a32e>] xen_blkif_schedule+0x21e/0xa00
> [ 73.015493] [<ffffffff81111442>] ? irq_to_desc+0x12/0x20
> [ 73.023833] [<ffffffff81114779>] ? irq_get_irq_data+0x9/0x10
> [ 73.032418] [<ffffffff81382909>] ? info_for_irq+0x9/0x20
> [ 73.040554] [<ffffffff81383cb9>] ? notify_remote_via_irq+0x29/0x50
> [ 73.049523] [<ffffffff810c844d>] ? sched_clock_cpu+0xcd/0x110
> [ 73.058024] [<ffffffff8107e068>] ? pvclock_clocksource_read+0x58/0xd0
> [ 73.067191] [<ffffffff8103e0c0>] ? xen_mc_flush+0xc0/0x1f0
> [ 73.075338] [<ffffffff81635e9e>] ? __schedule+0x3be/0x7c0
> [ 73.083311] [<ffffffff810b52a0>] ? wake_up_bit+0x40/0x40
> [ 73.091108] [<ffffffff8140a110>] ? dispatch_rw_block_io+0x600/0x600
> [ 73.099902] [<ffffffff810b4b16>] kthread+0xc6/0xd0
> [ 73.107124] [<ffffffff8103c9e9>] ? xen_end_context_switch+0x19/0x20
> [ 73.115845] [<ffffffff810b4a50>] ? kthread_freezable_should_stop+0x80/0x80
> [ 73.125266] [<ffffffff8163f1fc>] ret_from_fork+0x7c/0xb0
> [ 73.133089] [<ffffffff810b4a50>] ? kthread_freezable_should_stop+0x80/0x80
> [ 73.142576] Code: 48 89 d7 e8 ad 66 d8 ff 4a c7 84 3d 70 ff ff ff 00 00 00 00 4c 8b 85 60 fd ff ff 41 8b b0 e4 fd ff ff 41 83 cd 01 e9 ef fb ff ff <0f> 0b eb fe 48 8d 95 10 ff ff ff 48 8d bd b0 fd ff ff 31 f6 44
> [ 73.167611] RIP [<ffffffff81409766>] xen_blkbk_map+0x696/0x6e0
> [ 73.176081] RSP <ffff880027dd3728>
> [ 73.182024] ---[ end trace 914a52d8b62134db ]---
> [ 87.339441] switch: port 2(vif2.0) entered forwarding state
> [ 315.067569] device tap3.0 entered promiscuous mode
> [ 315.074965] switch: port 3(tap3.0) entered forwarding state
> [ 315.083116] switch: port 3(tap3.0) entered forwarding state
> [ 315.142543] switch: port 3(tap3.0) entered disabled state
> [ 315.161150] switch: port 3(tap3.0) entered forwarding state
> [ 315.169097] switch: port 3(tap3.0) entered forwarding state
> [ 330.162411] switch: port 3(tap3.0) entered forwarding state
> [ 415.483626] switch: port 3(tap3.0) entered disabled state
> [ 415.491439] device tap3.0 left promiscuous mode
> [ 415.498191] switch: port 3(tap3.0) entered disabled state
> [ 658.839306] device tap4.0 entered promiscuous mode
> [ 658.846451] switch: port 3(tap4.0) entered forwarding state
> [ 658.854354] switch: port 3(tap4.0) entered forwarding state
> [ 658.923751] switch: port 3(tap4.0) entered disabled state
> [ 658.942642] switch: port 3(tap4.0) entered forwarding state
> [ 658.950492] switch: port 3(tap4.0) entered forwarding state
> [ 673.990219] switch: port 3(tap4.0) entered forwarding state
> [ 759.263762] switch: port 3(tap4.0) entered disabled state
> [ 759.271529] device tap4.0 left promiscuous mode
> [ 759.278257] switch: port 3(tap4.0) entered disabled state
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v2] Persistent grant maps for xen blk drivers
2012-10-30 18:33 ` Roger Pau Monné
@ 2012-10-30 20:38 ` Konrad Rzeszutek Wilk
0 siblings, 0 replies; 13+ messages in thread
From: Konrad Rzeszutek Wilk @ 2012-10-30 20:38 UTC (permalink / raw)
To: Roger Pau Monné; +Cc: xen-devel, linux-kernel
On Tue, Oct 30, 2012 at 07:33:16PM +0100, Roger Pau Monné wrote:
> On 30/10/12 18:01, Konrad Rzeszutek Wilk wrote:
> > On Wed, Oct 24, 2012 at 06:58:45PM +0200, Roger Pau Monne wrote:
> >> This patch implements persistent grants for the xen-blk{front,back}
> >> mechanism. The effect of this change is to reduce the number of unmap
> >> operations performed, since they cause a (costly) TLB shootdown. This
> >> allows the I/O performance to scale better when a large number of VMs
> >> are performing I/O.
> >>
> >> Previously, the blkfront driver was supplied a bvec[] from the request
> >> queue. This was granted to dom0; dom0 performed the I/O and wrote
> >> directly into the grant-mapped memory and unmapped it; blkfront then
> >> removed foreign access for that grant. The cost of unmapping scales
> >> badly with the number of CPUs in Dom0. An experiment showed that when
> >> Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
> >> ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
> >> (at which point 650,000 IOPS are being performed in total). If more
> >> than 5 guests are used, the performance declines. By 10 guests, only
> >> 400,000 IOPS are being performed.
> >>
> >> This patch improves performance by only unmapping when the connection
> >> between blkfront and back is broken.
> >>
> >> On startup blkfront notifies blkback that it is using persistent
> >> grants, and blkback will do the same. If blkback is not capable of
> >> persistent mapping, blkfront will still use the same grants, since it
> >> is compatible with the previous protocol, and simplifies the code
> >> complexity in blkfront.
> >>
> >> To perform a read, in persistent mode, blkfront uses a separate pool
> >> of pages that it maps to dom0. When a request comes in, blkfront
> >> transmutes the request so that blkback will write into one of these
> >> free pages. Blkback keeps note of which grefs it has already
> >> mapped. When a new ring request comes to blkback, it looks to see if
> >> it has already mapped that page. If so, it will not map it again. If
> >> the page hasn't been previously mapped, it is mapped now, and a record
> >> is kept of this mapping. Blkback proceeds as usual. When blkfront is
> >> notified that blkback has completed a request, it memcpy's from the
> >> shared memory, into the bvec supplied. A record that the {gref, page}
> >> tuple is mapped, and not inflight is kept.
> >>
> >> Writes are similar, except that the memcpy is peformed from the
> >> supplied bvecs, into the shared pages, before the request is put onto
> >> the ring.
> >>
> >> Blkback stores a mapping of grefs=>{page mapped to by gref} in
> >> a red-black tree. As the grefs are not known apriori, and provide no
> >> guarantees on their ordering, we have to perform a search
> >> through this tree to find the page, for every gref we receive. This
> >> operation takes O(log n) time in the worst case. In blkfront grants
> >> are stored using a single linked list.
> >>
> >> The maximum number of grants that blkback will persistenly map is
> >> currently set to RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, to
> >> prevent a malicios guest from attempting a DoS, by supplying fresh
> >> grefs, causing the Dom0 kernel to map excessively. If a guest
> >> is using persistent grants and exceeds the maximum number of grants to
> >> map persistenly the newly passed grefs will be mapped and unmaped.
> >> Using this approach, we can have requests that mix persistent and
> >> non-persistent grants, and we need to handle them correctly.
> >> This allows us to set the maximum number of persistent grants to a
> >> lower value than RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, although
> >> setting it will lead to unpredictable performance.
> >>
> >> In writing this patch, the question arrises as to if the additional
> >> cost of performing memcpys in the guest (to/from the pool of granted
> >> pages) outweigh the gains of not performing TLB shootdowns. The answer
> >> to that question is `no'. There appears to be very little, if any
> >> additional cost to the guest of using persistent grants. There is
> >> perhaps a small saving, from the reduced number of hypercalls
> >> performed in granting, and ending foreign access.
> >>
> >> Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
> >> Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
> >> Cc: <konrad.wilk@oracle.com>
> >> Cc: <linux-kernel@vger.kernel.org>
> >> ---
> >> Changes since v1:
> >> * Changed the unmap_seg array to a bitmap.
> >> * Only report using persistent grants in blkfront if blkback supports
> >> it.
> >> * Reword some comments.
> >> * Fix a bug when setting the handler, index j was not incremented
> >> correctly.
> >> * Check that the tree of grants in blkback is not empty before
> >> iterating over it when doing the cleanup.
> >> * Rebase on top of linux-net.
> >
> > I fixed the 'new_map = [1|0]' you had in and altered it to use 'true'
> > or 'false', but when running some tests (with a 64-bit PV guest) I got it
> > to bug.
>
> Thanks for the testing. I'm going to rebase on top of your linux-next
> branch and see if I can reproduce it. Did you run any kind of specific
> test/benchmark? I've been running with this patch for a long time (on
None. Just booted a guest with a phy:/dev/vg_guest/blah.
> top of your previous linux-next branch), and I haven't been able to get
> it to bug.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v2] Persistent grant maps for xen blk drivers
2012-09-21 18:41 ` Konrad Rzeszutek Wilk
2012-09-21 18:56 ` Konrad Rzeszutek Wilk
@ 2012-09-27 15:49 ` Oliver Chick
1 sibling, 0 replies; 13+ messages in thread
From: Oliver Chick @ 2012-09-27 15:49 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk; +Cc: xen-devel, linux-kernel
Hi Konrad,
I have applied my patch to your #linux-next, and compiled. I don't
experience the same problem that you do - both PV and HVM boot and work.
Do you have anything special in your setup that might help me reproduce
the problem?
On Fri, 2012-09-21 at 19:41 +0100, Konrad Rzeszutek Wilk wrote:
> On Fri, Sep 21, 2012 at 04:52:47PM +0100, Oliver Chick wrote:
> > This patch implements persistent grants for the xen-blk{front,back}
> > mechanism. The effect of this change is to reduce the number of unmap
> > operations performed, since they cause a (costly) TLB shootdown. This
> > allows the I/O performance to scale better when a large number of VMs
> > are performing I/O.
>
> So I applied your patch to my #linux-next and I got this (the dom0 is the
> same as the domU)
> I get this when using an PVHVM (*)and PV guest:
> # fdisk /dev/xvda
> [ 28.594049] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
> [ 28.595028] IP: [<ffffffffa003fe9a>] blkif_interrupt+0x3aa/0x5d0 [xen_blkfront]
> [ 28.595028] PGD dcaf0067 PUD 7fb5e067 PMD 0
> [ 28.595028] Oops: 0000 [#1] SMP
> [ 28.595028] Modules linked in: iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi libcrc32c crc32c sg sr_mod cdrom ata_generic ata_piix crc32c_intel libata scsi_mod xen_blkfront xen_netfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xenfs xen_privcmd [last unloaded: dump_dma]
> [ 28.595028] CPU 0
> [ 28.595028] Pid: 0, comm: swapper/0 Tainted: G O 3.6.0-rc6upstream-00196-ge45cba2 #1 Xen HVM domU
> [ 28.595028] RIP: 0010:[<ffffffffa003fe9a>] [<ffffffffa003fe9a>] blkif_interrupt+0x3aa/0x5d0 [xen_blkfront]
> [ 28.595028] RSP: 0018:ffff88010fc03da8 EFLAGS: 00010002
> [ 28.595028] RAX: 0000000000000000 RBX: ffffffff81a00000 RCX: ffff88010af5f140
> [ 28.595028] RDX: ffff88010af5f080 RSI: ffff8800dbbf8210 RDI: ffff8800dc679c00
> [ 28.595028] RBP: ffff88010fc03e28 R08: 0000000000000001 R09: 0000000000000011
> [ 28.595028] R10: 0000000000000000 R11: 0000000000000000 R12: 6db6db6db6db6db7
> [ 28.595028] R13: ffff88010af5f1a8 R14: 0000000000000004 R15: 0000000000000000
> [ 28.595028] FS: 0000000000000000(0000) GS:ffff88010fc00000(0000) knlGS:0000000000000000
> [ 28.595028] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 28.595028] CR2: 0000000000000008 CR3: 00000000d4ac0000 CR4: 00000000000006f0
> [ 28.595028] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 28.595028] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 28.595028] Process swapper/0 (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a13420)
> [ 28.595028] Stack:
> [ 28.595028] 0000000000000086 ffff8800d9951060 ffff8800d9953000 0000000881376d09
> [ 28.595028] 0000000000000008 0000000a81376f2d 0000000000000000 ffff8800dbbf8210
> [ 28.595028] ffff8800dbbf8000 ffff88010af5f140 ffff88010fc03e08 ffff8800dcc4e3c0
> [ 28.595028] Call Trace:
> [ 28.595028] <IRQ>
> [ 28.595028] [<ffffffff8110dcdc>] handle_irq_event_percpu+0x7c/0x240
> [ 28.595028] [<ffffffff8110df02>] handle_irq_event+0x62/0x90
> [ 28.595028] [<ffffffff8111178e>] handle_edge_irq+0x8e/0x160
> [ 28.595028] [<ffffffff81376ac4>] __xen_evtchn_do_upcall+0x1c4/0x2a0
> [ 28.595028] [<ffffffff813775ba>] xen_evtchn_do_upcall+0x2a/0x40
> [ 28.595028] [<ffffffff816314ba>] xen_hvm_callback_vector+0x6a/0x70
> [ 28.595028] <EOI>
> [ 28.595028] [<ffffffff8107b3a6>] ? native_safe_halt+0x6/0x10
> [ 28.595028] [<ffffffff810555da>] default_idle+0x4a/0x210
> [ 28.595028] [<ffffffff81054d79>] cpu_idle+0x99/0xe0
> [ 28.595028] [<ffffffff816015da>] rest_init+0x8a/0xa0
> [ 28.595028] [<ffffffff81abcd72>] start_kernel+0x383/0x390
> [ 28.595028] [<ffffffff81abc80d>] ? kernel_init+0x1e8/0x1e8
> [ 28.595028] [<ffffffff81abc356>] x86_64_start_reservations+0x131/0x136
> [ 28.595028] [<ffffffff81abc45e>] x86_64_start_kernel+0x103/0x112
> [ 28.595028] Code: 00 49 83 c5 10 41 8b 45 08 41 03 45 0c 3d 00 10 00 00 0f 87 7e 01 00 00 48 8b 75 b8 49 63 c6 41 83 c6 01 48 8b 84 c6 d0 00 00 00 <48> 8b 40 08 83 43 1c 01 48 8d 14 c5 00 00 00 00 48 c1 e0 06 48
> [ 28.595028] RIP [<ffffffffa003fe9a>] blkif_interrupt+0x3aa/0x5d0 [xen_blkfront]
> [ 28.595028] RSP <ffff88010fc03da8>
> [ 28.595028] CR2: 0000000000000008
> [ 28.595028] ---[ end trace 49f5630a34777570 ]---
> [ 28.595028] Kernel panic - not syncing: Fatal exception in interrupt
> [ 28.595028] Rebooting in 60 seconds..
>
> *: With a PVHVM guest I get
>
> [ 261.927218] privcmd_fault: vma=ffff88002a31dce8 7f4edc095000-7f4edc195000, pgoff=c8, uv=00007f4edc15d000
>
> thought if I applied your patch on top of v3.6-rc6 I didn't see the privcmd_fault but
> I did see the guest crash.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v2] Persistent grant maps for xen blk drivers
2012-09-24 15:06 ` Konrad Rzeszutek Wilk
@ 2012-09-24 15:21 ` Andres Lagar-Cavilla
0 siblings, 0 replies; 13+ messages in thread
From: Andres Lagar-Cavilla @ 2012-09-24 15:21 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk
Cc: Andres Lagar-Cavilla, Oliver Chick, xen-devel, linux-kernel
On Sep 24, 2012, at 11:06 AM, Konrad Rzeszutek Wilk wrote:
> On Mon, Sep 24, 2012 at 10:38:48AM -0400, Andres Lagar-Cavilla wrote:
>> On Sep 21, 2012, at 4:46 PM, Konrad Rzeszutek Wilk wrote:
>>
>>> On Fri, Sep 21, 2012 at 02:56:22PM -0400, Konrad Rzeszutek Wilk wrote:
>>>>> *: With a PVHVM guest I get
>>>>>
>>>>> [ 261.927218] privcmd_fault: vma=ffff88002a31dce8 7f4edc095000-7f4edc195000, pgoff=c8, uv=00007f4edc15d000
>>>>>
>>>>> thought if I applied your patch on top of v3.6-rc6 I didn't see the privcmd_fault but
>>>>> I did see the guest crash.
>>>>
>>>> And that is due to c571898ffc24a1768e1b2dabeac0fc7dd4c14601 which I've reverted in my
>>>> #linux-next branch
>>>
>>> Nevermind. Andres' patch by itself (so without yours) works just fine. There is
>>> something your patch and his aren't agreeing on.
>>
>> Apart from interacting badly in combination, would either patch in isolation work well?
>
> "work well" is not exactly the right phase I would use.
>
> Your patch by itself (so on top v3.6-rc6) works great.
> On top of #linux-next (so v3.6-rc6 + lot of other patches for v3.7) works great.
>
> Oliver's patch for blkback/blkfront on top of v3.6-rc6 falls flat on its face in the guest.
> Oliver's patch for blkback/blkfront on top of v3.6-rc6 + lot of other patches for v3.7 - your patch)
> falls flat on its face in the guest.
>
> Oliver's patch for blkback/blkfront on top of v3.6-rc6 + lot of other patches for v3.7 + your patch)
> falls flat on its face in the backend with the privcmd_fault.
>
> I think I need to figure on what tree does Oliver's patch work properly and
> figure out why it dies first in the guest and then find out why in the backend
> it dies with your patch.
>
> Your patch is still on the train for v3.7
I saw. Great!
>
>>
>> I can think of only one hunk in my patch disagreeing with blkback stuff:
>
> I can try it out.. but I am not going to get to it today or in the next
> few days.
It would seem from your testing matrix above the issue lies somewhere deeper within the blk* patches. I'll watch from the sidelines, please keep me in the loop.
Thanks,
Andres
>
>> diff --git a/drivers/xen/grant-table.c b/drivers/xen/grant-table.c
>> index eea81cf..f5681c8 100644
>> --- a/drivers/xen/grant-table.c
>> +++ b/drivers/xen/grant-table.c
>> @@ -836,6 +883,12 @@ int gnttab_map_refs(struct gnttab_map_grant_ref *map_ops,
>> if (ret)
>> return ret;
>>
>> + /* Retry eagain maps */
>> + for (i = 0; i < count; i++)
>> + if (map_ops[i].status == GNTST_eagain)
>> + gnttab_retry_eagain_gop(GNTTABOP_map_grant_ref, map_ops + i,
>> + &map_ops[i].status, __func__);
>> +
>> if (xen_feature(XENFEAT_auto_translated_physmap))
>> return ret;
>>
>> How would you like to proceed?
>> Andres
>>
>>>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v2] Persistent grant maps for xen blk drivers
2012-09-24 14:38 ` Andres Lagar-Cavilla
@ 2012-09-24 15:06 ` Konrad Rzeszutek Wilk
2012-09-24 15:21 ` Andres Lagar-Cavilla
0 siblings, 1 reply; 13+ messages in thread
From: Konrad Rzeszutek Wilk @ 2012-09-24 15:06 UTC (permalink / raw)
To: Andres Lagar-Cavilla; +Cc: Oliver Chick, xen-devel, linux-kernel
On Mon, Sep 24, 2012 at 10:38:48AM -0400, Andres Lagar-Cavilla wrote:
> On Sep 21, 2012, at 4:46 PM, Konrad Rzeszutek Wilk wrote:
>
> > On Fri, Sep 21, 2012 at 02:56:22PM -0400, Konrad Rzeszutek Wilk wrote:
> >>> *: With a PVHVM guest I get
> >>>
> >>> [ 261.927218] privcmd_fault: vma=ffff88002a31dce8 7f4edc095000-7f4edc195000, pgoff=c8, uv=00007f4edc15d000
> >>>
> >>> thought if I applied your patch on top of v3.6-rc6 I didn't see the privcmd_fault but
> >>> I did see the guest crash.
> >>
> >> And that is due to c571898ffc24a1768e1b2dabeac0fc7dd4c14601 which I've reverted in my
> >> #linux-next branch
> >
> > Nevermind. Andres' patch by itself (so without yours) works just fine. There is
> > something your patch and his aren't agreeing on.
>
> Apart from interacting badly in combination, would either patch in isolation work well?
"work well" is not exactly the right phase I would use.
Your patch by itself (so on top v3.6-rc6) works great.
On top of #linux-next (so v3.6-rc6 + lot of other patches for v3.7) works great.
Oliver's patch for blkback/blkfront on top of v3.6-rc6 falls flat on its face in the guest.
Oliver's patch for blkback/blkfront on top of v3.6-rc6 + lot of other patches for v3.7 - your patch)
falls flat on its face in the guest.
Oliver's patch for blkback/blkfront on top of v3.6-rc6 + lot of other patches for v3.7 + your patch)
falls flat on its face in the backend with the privcmd_fault.
I think I need to figure on what tree does Oliver's patch work properly and
figure out why it dies first in the guest and then find out why in the backend
it dies with your patch.
Your patch is still on the train for v3.7
>
> I can think of only one hunk in my patch disagreeing with blkback stuff:
I can try it out.. but I am not going to get to it today or in the next
few days.
> diff --git a/drivers/xen/grant-table.c b/drivers/xen/grant-table.c
> index eea81cf..f5681c8 100644
> --- a/drivers/xen/grant-table.c
> +++ b/drivers/xen/grant-table.c
> @@ -836,6 +883,12 @@ int gnttab_map_refs(struct gnttab_map_grant_ref *map_ops,
> if (ret)
> return ret;
>
> + /* Retry eagain maps */
> + for (i = 0; i < count; i++)
> + if (map_ops[i].status == GNTST_eagain)
> + gnttab_retry_eagain_gop(GNTTABOP_map_grant_ref, map_ops + i,
> + &map_ops[i].status, __func__);
> +
> if (xen_feature(XENFEAT_auto_translated_physmap))
> return ret;
>
> How would you like to proceed?
> Andres
>
> >
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v2] Persistent grant maps for xen blk drivers
2012-09-21 20:46 ` Konrad Rzeszutek Wilk
@ 2012-09-24 14:38 ` Andres Lagar-Cavilla
2012-09-24 15:06 ` Konrad Rzeszutek Wilk
0 siblings, 1 reply; 13+ messages in thread
From: Andres Lagar-Cavilla @ 2012-09-24 14:38 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk; +Cc: Oliver Chick, xen-devel, linux-kernel
On Sep 21, 2012, at 4:46 PM, Konrad Rzeszutek Wilk wrote:
> On Fri, Sep 21, 2012 at 02:56:22PM -0400, Konrad Rzeszutek Wilk wrote:
>>> *: With a PVHVM guest I get
>>>
>>> [ 261.927218] privcmd_fault: vma=ffff88002a31dce8 7f4edc095000-7f4edc195000, pgoff=c8, uv=00007f4edc15d000
>>>
>>> thought if I applied your patch on top of v3.6-rc6 I didn't see the privcmd_fault but
>>> I did see the guest crash.
>>
>> And that is due to c571898ffc24a1768e1b2dabeac0fc7dd4c14601 which I've reverted in my
>> #linux-next branch
>
> Nevermind. Andres' patch by itself (so without yours) works just fine. There is
> something your patch and his aren't agreeing on.
Apart from interacting badly in combination, would either patch in isolation work well?
I can think of only one hunk in my patch disagreeing with blkback stuff:
diff --git a/drivers/xen/grant-table.c b/drivers/xen/grant-table.c
index eea81cf..f5681c8 100644
--- a/drivers/xen/grant-table.c
+++ b/drivers/xen/grant-table.c
@@ -836,6 +883,12 @@ int gnttab_map_refs(struct gnttab_map_grant_ref *map_ops,
if (ret)
return ret;
+ /* Retry eagain maps */
+ for (i = 0; i < count; i++)
+ if (map_ops[i].status == GNTST_eagain)
+ gnttab_retry_eagain_gop(GNTTABOP_map_grant_ref, map_ops + i,
+ &map_ops[i].status, __func__);
+
if (xen_feature(XENFEAT_auto_translated_physmap))
return ret;
How would you like to proceed?
Andres
>
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH v2] Persistent grant maps for xen blk drivers
2012-09-21 18:56 ` Konrad Rzeszutek Wilk
@ 2012-09-21 20:46 ` Konrad Rzeszutek Wilk
2012-09-24 14:38 ` Andres Lagar-Cavilla
0 siblings, 1 reply; 13+ messages in thread
From: Konrad Rzeszutek Wilk @ 2012-09-21 20:46 UTC (permalink / raw)
To: Oliver Chick, andres.lagarcavilla; +Cc: xen-devel, linux-kernel
On Fri, Sep 21, 2012 at 02:56:22PM -0400, Konrad Rzeszutek Wilk wrote:
> > *: With a PVHVM guest I get
> >
> > [ 261.927218] privcmd_fault: vma=ffff88002a31dce8 7f4edc095000-7f4edc195000, pgoff=c8, uv=00007f4edc15d000
> >
> > thought if I applied your patch on top of v3.6-rc6 I didn't see the privcmd_fault but
> > I did see the guest crash.
>
> And that is due to c571898ffc24a1768e1b2dabeac0fc7dd4c14601 which I've reverted in my
> #linux-next branch
Nevermind. Andres' patch by itself (so without yours) works just fine. There is
something your patch and his aren't agreeing on.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v2] Persistent grant maps for xen blk drivers
2012-09-21 18:41 ` Konrad Rzeszutek Wilk
@ 2012-09-21 18:56 ` Konrad Rzeszutek Wilk
2012-09-21 20:46 ` Konrad Rzeszutek Wilk
2012-09-27 15:49 ` Oliver Chick
1 sibling, 1 reply; 13+ messages in thread
From: Konrad Rzeszutek Wilk @ 2012-09-21 18:56 UTC (permalink / raw)
To: Oliver Chick; +Cc: xen-devel, linux-kernel
> *: With a PVHVM guest I get
>
> [ 261.927218] privcmd_fault: vma=ffff88002a31dce8 7f4edc095000-7f4edc195000, pgoff=c8, uv=00007f4edc15d000
>
> thought if I applied your patch on top of v3.6-rc6 I didn't see the privcmd_fault but
> I did see the guest crash.
And that is due to c571898ffc24a1768e1b2dabeac0fc7dd4c14601 which I've reverted in my
#linux-next branch
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v2] Persistent grant maps for xen blk drivers
2012-09-21 15:52 Oliver Chick
@ 2012-09-21 18:41 ` Konrad Rzeszutek Wilk
2012-09-21 18:56 ` Konrad Rzeszutek Wilk
2012-09-27 15:49 ` Oliver Chick
0 siblings, 2 replies; 13+ messages in thread
From: Konrad Rzeszutek Wilk @ 2012-09-21 18:41 UTC (permalink / raw)
To: Oliver Chick; +Cc: xen-devel, linux-kernel
On Fri, Sep 21, 2012 at 04:52:47PM +0100, Oliver Chick wrote:
> This patch implements persistent grants for the xen-blk{front,back}
> mechanism. The effect of this change is to reduce the number of unmap
> operations performed, since they cause a (costly) TLB shootdown. This
> allows the I/O performance to scale better when a large number of VMs
> are performing I/O.
So I applied your patch to my #linux-next and I got this (the dom0 is the
same as the domU)
I get this when using an PVHVM (*)and PV guest:
# fdisk /dev/xvda
[ 28.594049] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
[ 28.595028] IP: [<ffffffffa003fe9a>] blkif_interrupt+0x3aa/0x5d0 [xen_blkfront]
[ 28.595028] PGD dcaf0067 PUD 7fb5e067 PMD 0
[ 28.595028] Oops: 0000 [#1] SMP
[ 28.595028] Modules linked in: iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi libcrc32c crc32c sg sr_mod cdrom ata_generic ata_piix crc32c_intel libata scsi_mod xen_blkfront xen_netfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xenfs xen_privcmd [last unloaded: dump_dma]
[ 28.595028] CPU 0
[ 28.595028] Pid: 0, comm: swapper/0 Tainted: G O 3.6.0-rc6upstream-00196-ge45cba2 #1 Xen HVM domU
[ 28.595028] RIP: 0010:[<ffffffffa003fe9a>] [<ffffffffa003fe9a>] blkif_interrupt+0x3aa/0x5d0 [xen_blkfront]
[ 28.595028] RSP: 0018:ffff88010fc03da8 EFLAGS: 00010002
[ 28.595028] RAX: 0000000000000000 RBX: ffffffff81a00000 RCX: ffff88010af5f140
[ 28.595028] RDX: ffff88010af5f080 RSI: ffff8800dbbf8210 RDI: ffff8800dc679c00
[ 28.595028] RBP: ffff88010fc03e28 R08: 0000000000000001 R09: 0000000000000011
[ 28.595028] R10: 0000000000000000 R11: 0000000000000000 R12: 6db6db6db6db6db7
[ 28.595028] R13: ffff88010af5f1a8 R14: 0000000000000004 R15: 0000000000000000
[ 28.595028] FS: 0000000000000000(0000) GS:ffff88010fc00000(0000) knlGS:0000000000000000
[ 28.595028] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 28.595028] CR2: 0000000000000008 CR3: 00000000d4ac0000 CR4: 00000000000006f0
[ 28.595028] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 28.595028] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 28.595028] Process swapper/0 (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a13420)
[ 28.595028] Stack:
[ 28.595028] 0000000000000086 ffff8800d9951060 ffff8800d9953000 0000000881376d09
[ 28.595028] 0000000000000008 0000000a81376f2d 0000000000000000 ffff8800dbbf8210
[ 28.595028] ffff8800dbbf8000 ffff88010af5f140 ffff88010fc03e08 ffff8800dcc4e3c0
[ 28.595028] Call Trace:
[ 28.595028] <IRQ>
[ 28.595028] [<ffffffff8110dcdc>] handle_irq_event_percpu+0x7c/0x240
[ 28.595028] [<ffffffff8110df02>] handle_irq_event+0x62/0x90
[ 28.595028] [<ffffffff8111178e>] handle_edge_irq+0x8e/0x160
[ 28.595028] [<ffffffff81376ac4>] __xen_evtchn_do_upcall+0x1c4/0x2a0
[ 28.595028] [<ffffffff813775ba>] xen_evtchn_do_upcall+0x2a/0x40
[ 28.595028] [<ffffffff816314ba>] xen_hvm_callback_vector+0x6a/0x70
[ 28.595028] <EOI>
[ 28.595028] [<ffffffff8107b3a6>] ? native_safe_halt+0x6/0x10
[ 28.595028] [<ffffffff810555da>] default_idle+0x4a/0x210
[ 28.595028] [<ffffffff81054d79>] cpu_idle+0x99/0xe0
[ 28.595028] [<ffffffff816015da>] rest_init+0x8a/0xa0
[ 28.595028] [<ffffffff81abcd72>] start_kernel+0x383/0x390
[ 28.595028] [<ffffffff81abc80d>] ? kernel_init+0x1e8/0x1e8
[ 28.595028] [<ffffffff81abc356>] x86_64_start_reservations+0x131/0x136
[ 28.595028] [<ffffffff81abc45e>] x86_64_start_kernel+0x103/0x112
[ 28.595028] Code: 00 49 83 c5 10 41 8b 45 08 41 03 45 0c 3d 00 10 00 00 0f 87 7e 01 00 00 48 8b 75 b8 49 63 c6 41 83 c6 01 48 8b 84 c6 d0 00 00 00 <48> 8b 40 08 83 43 1c 01 48 8d 14 c5 00 00 00 00 48 c1 e0 06 48
[ 28.595028] RIP [<ffffffffa003fe9a>] blkif_interrupt+0x3aa/0x5d0 [xen_blkfront]
[ 28.595028] RSP <ffff88010fc03da8>
[ 28.595028] CR2: 0000000000000008
[ 28.595028] ---[ end trace 49f5630a34777570 ]---
[ 28.595028] Kernel panic - not syncing: Fatal exception in interrupt
[ 28.595028] Rebooting in 60 seconds..
*: With a PVHVM guest I get
[ 261.927218] privcmd_fault: vma=ffff88002a31dce8 7f4edc095000-7f4edc195000, pgoff=c8, uv=00007f4edc15d000
thought if I applied your patch on top of v3.6-rc6 I didn't see the privcmd_fault but
I did see the guest crash.
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v2] Persistent grant maps for xen blk drivers
@ 2012-09-21 15:52 Oliver Chick
2012-09-21 18:41 ` Konrad Rzeszutek Wilk
0 siblings, 1 reply; 13+ messages in thread
From: Oliver Chick @ 2012-09-21 15:52 UTC (permalink / raw)
To: xen-devel, linux-kernel, konrad.wilk; +Cc: Oliver Chick
This patch implements persistent grants for the xen-blk{front,back}
mechanism. The effect of this change is to reduce the number of unmap
operations performed, since they cause a (costly) TLB shootdown. This
allows the I/O performance to scale better when a large number of VMs
are performing I/O.
Previously, the blkfront driver was supplied a bvec[] from the request
queue. This was granted to dom0; dom0 performed the I/O and wrote
directly into the grant-mapped memory and unmapped it; blkfront then
removed foreign access for that grant. The cost of unmapping scales
badly with the number of CPUs in Dom0. An experiment showed that when
Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
(at which point 650,000 IOPS are being performed in total). If more
than 5 guests are used, the performance declines. By 10 guests, only
400,000 IOPS are being performed.
This patch improves performance by only unmapping when the connection
between blkfront and back is broken.
On startup, blk{front,back} use xenbus to communicate their ability to
perform 'feature-persistent'. Iff both ends have this ability,
persistent mode is used.
To perform a read, in persistent mode, blkfront uses a separate pool
of pages that it maps to dom0. When a request comes in, blkfront
transmutes the request so that blkback will write into one of these
free pages. Blkback keeps note of which grefs it has already
mapped. When a new ring request comes to blkback, it looks to see if
it has already mapped that page. If so, it will not map it again. If
the page hasn't been previously mapped, it is mapped now, and a record
is kept of this mapping. Blkback proceeds as usual. When blkfront is
notified that blkback has completed a request, it memcpy's from the
shared memory, into the bvec supplied. A record that the {gref, page}
tuple is mapped, and not inflight is kept.
Writes are similar, except that the memcpy is peformed from the
supplied bvecs, into the shared pages, before the request is put onto
the ring.
Blkback has to store a mapping of grefs=>{page mapped to by gref} in
an array. As the grefs are not known apriori, and provide no
guarantees on their ordering, we have to perform a linear search
through this array to find the page, for every gref we receive. The
overhead of this is low, however future work might want to use a more
efficient data structure to reduce this O(n) operation.
We (ijc, and myself) have introduced a new constant,
BLKIF_MAX_PERSISTENT_REQUESTS_PER_DEV. This is to prevent a malicious
guest from attempting a DoS, by supplying fresh grefs, causing the
Dom0 kernel from to map excessively. This is currently set to 64---the
maximum number of entires in the ring. As the number of inflight
requests <= size of ring, 64 is also the maximum sensible size (for
single page rings---this constant may need to be made dynamic when
multipage rings takeoff). This introduces a maximum overhead of 2.75MB
of mapped memory, per block device. If the guest exceeds the limit,
it is either buggy or malicious. We treat this in one of two ways:
1) If we have mapped < BLKIF_MAX_SEGMENTS_PER_REQUEST *
BLKIF_MAX_PERSISTENT_REQUESTS_PER_DEV pages, we will persistently map
the grefs. This can occur is previous requests have not used all
BLKIF_MAX_SEGMENTS_PER_REQUEST segments.
2) Otherwise, we revert to non-persistent grants for all future grefs.
In writing this patch, the question arrises as to if the additional
cost of performing memcpys in the guest (to/from the pool of granted
pages) outweigh the gains of not performing TLB shootdowns. The answer
to that question is `no'. There appears to be very little, if any
additional cost to the guest of using persistent grants. There is
perhaps a small saving, from the reduced number of hypercalls
performed in granting, and ending foreign access.
Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
---
Changes since v1:
* Maximum number of persistent grants per device now 64, rather than
256, as this is the actual maxmimum request in a (1 page) ring.
* If blkfront supplies more grefs than it is meant to, to blkback,
blkback will fail to process them. This gives more predictable
pefromance.
* Use of `pers' as an abbreviation has been removed, in favour of the
more clear `persistent'.
* Fixed potential leak where we allocate space to store information
about persistent grants, but fail to alloc a page for the grant.
* Moved frontend persistent gnt feature information into vbd
* Check that offset+length < page size in blkfront.
* Minor variable renames, as suggested by Konrad
* Added some comments documenting code, as suggested by Konrad
drivers/block/xen-blkback/blkback.c | 166 ++++++++++++++++++++++++---
drivers/block/xen-blkback/common.h | 16 +++
drivers/block/xen-blkback/xenbus.c | 21 +++-
drivers/block/xen-blkfront.c | 211 +++++++++++++++++++++++++++++++----
include/xen/interface/io/blkif.h | 13 +++
5 files changed, 385 insertions(+), 42 deletions(-)
diff --git a/drivers/block/xen-blkback/blkback.c b/drivers/block/xen-blkback/blkback.c
index 73f196c..8ac8d3f 100644
--- a/drivers/block/xen-blkback/blkback.c
+++ b/drivers/block/xen-blkback/blkback.c
@@ -78,6 +78,7 @@ struct pending_req {
unsigned short operation;
int status;
struct list_head free_list;
+ unsigned int flag_persistent:1;
};
#define BLKBACK_INVALID_HANDLE (~0)
@@ -128,6 +129,26 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif,
static void make_response(struct xen_blkif *blkif, u64 id,
unsigned short op, int st);
+static void add_persistent_gnt(struct persistent_gnt *persistent_gnt,
+ struct xen_blkif *blkif)
+{
+ BUG_ON(blkif->persistent_gnt_c >=
+ BLKIF_MAX_PERSISTENT_REQUESTS_PER_DEV *
+ BLKIF_MAX_SEGMENTS_PER_REQUEST);
+ blkif->persistent_gnts[blkif->persistent_gnt_c++] = persistent_gnt;
+}
+
+static struct persistent_gnt *get_persistent_gnt(struct xen_blkif *blkif,
+ grant_ref_t gref)
+{
+ int i;
+
+ for (i = 0; i < blkif->persistent_gnt_c; i++)
+ if (gref == blkif->persistent_gnts[i]->gnt)
+ return blkif->persistent_gnts[i];
+ return NULL;
+}
+
/*
* Retrieve from the 'pending_reqs' a free pending_req structure to be used.
*/
@@ -274,6 +295,12 @@ int xen_blkif_schedule(void *arg)
{
struct xen_blkif *blkif = arg;
struct xen_vbd *vbd = &blkif->vbd;
+ struct gnttab_unmap_grant_ref unmap[BLKIF_MAX_SEGMENTS_PER_REQUEST];
+ struct page *pages[BLKIF_MAX_SEGMENTS_PER_REQUEST];
+ struct persistent_gnt *persistent_gnt;
+ int i;
+ int ret = 0;
+ int segs_to_unmap;
xen_blkif_get(blkif);
@@ -301,6 +328,31 @@ int xen_blkif_schedule(void *arg)
print_stats(blkif);
}
+ /* Free all persistent grant pages */
+
+ while ((segs_to_unmap = min(BLKIF_MAX_SEGMENTS_PER_REQUEST,
+ blkif->persistent_gnt_c))) {
+
+ for (i = 0; i < segs_to_unmap; i++) {
+ persistent_gnt = blkif->persistent_gnts
+ [blkif->persistent_gnt_c - i - 1];
+
+ gnttab_set_unmap_op(&unmap[i],
+ pfn_to_kaddr(page_to_pfn(
+ persistent_gnt->page)),
+ GNTMAP_host_map,
+ persistent_gnt->handle);
+
+ pages[i] = persistent_gnt->page;
+ }
+
+ ret = gnttab_unmap_refs(unmap, pages, segs_to_unmap, false);
+ BUG_ON(ret);
+
+ blkif->persistent_gnt_c -= segs_to_unmap;
+
+ }
+
if (log_stats)
print_stats(blkif);
@@ -343,13 +395,31 @@ static void xen_blkbk_unmap(struct pending_req *req)
static int xen_blkbk_map(struct blkif_request *req,
struct pending_req *pending_req,
- struct seg_buf seg[])
+ struct seg_buf seg[],
+ struct page *pages[])
{
struct gnttab_map_grant_ref map[BLKIF_MAX_SEGMENTS_PER_REQUEST];
+ struct persistent_gnt
+ *new_persistent_gnts[BLKIF_MAX_SEGMENTS_PER_REQUEST];
+ struct persistent_gnt *persistent_gnts[BLKIF_MAX_SEGMENTS_PER_REQUEST];
+ struct page *pages_to_gnt[BLKIF_MAX_SEGMENTS_PER_REQUEST];
+ struct persistent_gnt *persistent_gnt;
+ phys_addr_t addr;
int i;
+ int new_map;
int nseg = req->u.rw.nr_segments;
+ int segs_to_map = 0;
int ret = 0;
+ int use_persistent_gnts;
+
+ use_persistent_gnts = (pending_req->blkif->vbd.feature_gnt_persistent);
+
+ if (pending_req->blkif->persistent_gnt_c >=
+ BLKIF_MAX_SEGMENTS_PER_REQUEST *
+ BLKIF_MAX_PERSISTENT_REQUESTS_PER_DEV)
+ return -EIO;
+ pending_req->flag_persistent = use_persistent_gnts;
/*
* Fill out preq.nr_sects with proper amount of sectors, and setup
* assign map[..] with the PFN of the page in our domain with the
@@ -358,36 +428,97 @@ static int xen_blkbk_map(struct blkif_request *req,
for (i = 0; i < nseg; i++) {
uint32_t flags;
+ if (use_persistent_gnts) {
+ persistent_gnt = get_persistent_gnt(
+ pending_req->blkif,
+ req->u.rw.seg[i].gref);
+ if (!persistent_gnt) {
+ new_map = 1;
+ persistent_gnt = kmalloc(
+ sizeof(struct persistent_gnt),
+ GFP_KERNEL);
+ if (!persistent_gnt)
+ return -ENOMEM;
+ persistent_gnt->page = alloc_page(GFP_KERNEL);
+ if (!persistent_gnt->page) {
+ kfree(persistent_gnt);
+ return -ENOMEM;
+ }
+ persistent_gnt->gnt = req->u.rw.seg[i].gref;
+
+ pages_to_gnt[segs_to_map] =
+ persistent_gnt->page;
+ new_persistent_gnts[segs_to_map] =
+ persistent_gnt;
+
+ add_persistent_gnt(persistent_gnt,
+ pending_req->blkif);
+
+ } else {
+ new_map = 0;
+ }
+ pages[i] = persistent_gnt->page;
+ addr = (unsigned long) pfn_to_kaddr(
+ page_to_pfn(persistent_gnt->page));
+ persistent_gnts[i] = persistent_gnt;
+ } else {
+ new_map = 1;
+ pages[i] = blkbk->pending_page(pending_req, i);
+ addr = vaddr(pending_req, i);
+ pages_to_gnt[i] = blkbk->pending_page(pending_req, i);
+ }
+
flags = GNTMAP_host_map;
- if (pending_req->operation != BLKIF_OP_READ)
+ if (!use_persistent_gnts &&
+ (pending_req->operation != BLKIF_OP_READ))
flags |= GNTMAP_readonly;
- gnttab_set_map_op(&map[i], vaddr(pending_req, i), flags,
- req->u.rw.seg[i].gref,
- pending_req->blkif->domid);
+ if (new_map) {
+ gnttab_set_map_op(&map[segs_to_map++], addr,
+ flags, req->u.rw.seg[i].gref,
+ pending_req->blkif->domid);
+ }
}
- ret = gnttab_map_refs(map, NULL, &blkbk->pending_page(pending_req, 0), nseg);
- BUG_ON(ret);
+ if (segs_to_map) {
+ ret = gnttab_map_refs(map, NULL, pages_to_gnt, segs_to_map);
+ BUG_ON(ret);
+ }
/*
* Now swizzle the MFN in our domain with the MFN from the other domain
* so that when we access vaddr(pending_req,i) it has the contents of
* the page from the other domain.
*/
- for (i = 0; i < nseg; i++) {
+ for (i = 0; i < segs_to_map; i++) {
if (unlikely(map[i].status != 0)) {
pr_debug(DRV_PFX "invalid buffer -- could not remap it\n");
map[i].handle = BLKBACK_INVALID_HANDLE;
ret |= 1;
}
- pending_handle(pending_req, i) = map[i].handle;
+ if (use_persistent_gnts) {
+ /* store the `out' values from map */
+ pending_req->blkif->persistent_gnts
+ [pending_req->blkif->persistent_gnt_c - segs_to_map +
+ i]->handle = map[i].handle;
+ new_persistent_gnts[i]->dev_bus_addr =
+ map[i].dev_bus_addr;
+ }
if (ret)
continue;
-
- seg[i].buf = map[i].dev_bus_addr |
- (req->u.rw.seg[i].first_sect << 9);
+ }
+ for (i = 0; i < nseg; i++) {
+ if (use_persistent_gnts) {
+ pending_handle(pending_req, i) =
+ persistent_gnts[i]->handle;
+ seg[i].buf = persistent_gnts[i]->dev_bus_addr |
+ (req->u.rw.seg[i].first_sect << 9);
+ } else {
+ pending_handle(pending_req, i) = map[i].handle;
+ seg[i].buf = map[i].dev_bus_addr |
+ (req->u.rw.seg[i].first_sect << 9);
+ }
}
return ret;
}
@@ -468,7 +599,8 @@ static void __end_block_io_op(struct pending_req *pending_req, int error)
* the proper response on the ring.
*/
if (atomic_dec_and_test(&pending_req->pendcnt)) {
- xen_blkbk_unmap(pending_req);
+ if (!pending_req->flag_persistent)
+ xen_blkbk_unmap(pending_req);
make_response(pending_req->blkif, pending_req->id,
pending_req->operation, pending_req->status);
xen_blkif_put(pending_req->blkif);
@@ -590,6 +722,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif,
int operation;
struct blk_plug plug;
bool drain = false;
+ struct page *pages[BLKIF_MAX_SEGMENTS_PER_REQUEST];
switch (req->operation) {
case BLKIF_OP_READ:
@@ -676,7 +809,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif,
* the hypercall to unmap the grants - that is all done in
* xen_blkbk_unmap.
*/
- if (xen_blkbk_map(req, pending_req, seg))
+ if (xen_blkbk_map(req, pending_req, seg, pages))
goto fail_flush;
/*
@@ -688,7 +821,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif,
for (i = 0; i < nseg; i++) {
while ((bio == NULL) ||
(bio_add_page(bio,
- blkbk->pending_page(pending_req, i),
+ pages[i],
seg[i].nsec << 9,
seg[i].buf & ~PAGE_MASK) == 0)) {
@@ -743,7 +876,8 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif,
return 0;
fail_flush:
- xen_blkbk_unmap(pending_req);
+ if (!blkif->vbd.feature_gnt_persistent)
+ xen_blkbk_unmap(pending_req);
fail_response:
/* Haven't submitted any bio's yet. */
make_response(blkif, req->u.rw.id, req->operation, BLKIF_RSP_ERROR);
diff --git a/drivers/block/xen-blkback/common.h b/drivers/block/xen-blkback/common.h
index 9ad3b5e..0b7e049 100644
--- a/drivers/block/xen-blkback/common.h
+++ b/drivers/block/xen-blkback/common.h
@@ -160,10 +160,20 @@ struct xen_vbd {
sector_t size;
bool flush_support;
bool discard_secure;
+
+ unsigned int feature_gnt_persistent:1;
};
struct backend_info;
+
+struct persistent_gnt {
+ struct page *page;
+ grant_ref_t gnt;
+ grant_handle_t handle;
+ uint64_t dev_bus_addr;
+};
+
struct xen_blkif {
/* Unique identifier for this interface. */
domid_t domid;
@@ -190,6 +200,12 @@ struct xen_blkif {
struct task_struct *xenblkd;
unsigned int waiting_reqs;
+ /* frontend feature information */
+ struct persistent_gnt *persistent_gnts
+ [BLKIF_MAX_PERSISTENT_REQUESTS_PER_DEV
+ * BLKIF_MAX_SEGMENTS_PER_REQUEST];
+ unsigned int persistent_gnt_c;
+
/* statistics */
unsigned long st_print;
int st_rd_req;
diff --git a/drivers/block/xen-blkback/xenbus.c b/drivers/block/xen-blkback/xenbus.c
index 4f66171..473e416 100644
--- a/drivers/block/xen-blkback/xenbus.c
+++ b/drivers/block/xen-blkback/xenbus.c
@@ -681,6 +681,13 @@ again:
goto abort;
}
+ err = xenbus_printf(xbt, dev->nodename, "feature-persistent-grants",
+ "%d", 1);
+ if (err) {
+ dev_warn(&dev->dev, "writing persistent grants feature");
+ goto abort;
+ }
+
/* FIXME: use a typename instead */
err = xenbus_printf(xbt, dev->nodename, "info", "%u",
be->blkif->vbd.type |
@@ -721,6 +728,7 @@ static int connect_ring(struct backend_info *be)
struct xenbus_device *dev = be->dev;
unsigned long ring_ref;
unsigned int evtchn;
+ u8 pers_grants;
char protocol[64] = "";
int err;
@@ -750,8 +758,17 @@ static int connect_ring(struct backend_info *be)
xenbus_dev_fatal(dev, err, "unknown fe protocol %s", protocol);
return -1;
}
- pr_info(DRV_PFX "ring-ref %ld, event-channel %d, protocol %d (%s)\n",
- ring_ref, evtchn, be->blkif->blk_protocol, protocol);
+ err = xenbus_gather(XBT_NIL, dev->otherend,
+ "feature-persistent-grants", "%d",
+ &pers_grants, NULL);
+ if (err)
+ pers_grants = 0;
+
+ be->blkif->vbd.feature_gnt_persistent = pers_grants;
+
+ pr_info(DRV_PFX "ring-ref %ld, event-channel %d, protocol %d (%s) persistent %d\n",
+ ring_ref, evtchn, be->blkif->blk_protocol, protocol,
+ pers_grants);
/* Map the shared frame, irq etc. */
err = xen_blkif_map(be->blkif, ring_ref, evtchn);
diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c
index 2c2d2e5..9606c5a 100644
--- a/drivers/block/xen-blkfront.c
+++ b/drivers/block/xen-blkfront.c
@@ -64,10 +64,17 @@ enum blkif_state {
BLKIF_STATE_SUSPENDED,
};
+struct gnt_list {
+ grant_ref_t gref;
+ unsigned long pfn;
+ struct gnt_list *tail;
+};
+
struct blk_shadow {
struct blkif_request req;
struct request *request;
unsigned long frame[BLKIF_MAX_SEGMENTS_PER_REQUEST];
+ struct gnt_list *grants_used[BLKIF_MAX_SEGMENTS_PER_REQUEST];
};
static DEFINE_MUTEX(blkfront_mutex);
@@ -97,11 +104,14 @@ struct blkfront_info
struct work_struct work;
struct gnttab_free_callback callback;
struct blk_shadow shadow[BLK_RING_SIZE];
+ struct gnt_list *persistent_gnts;
+ unsigned int persistent_gnts_c;
unsigned long shadow_free;
unsigned int feature_flush;
unsigned int flush_op;
unsigned int feature_discard:1;
unsigned int feature_secdiscard:1;
+ unsigned int feature_persistent:1;
unsigned int discard_granularity;
unsigned int discard_alignment;
int is_ready;
@@ -287,21 +297,42 @@ static int blkif_queue_request(struct request *req)
unsigned long id;
unsigned int fsect, lsect;
int i, ref;
+
+ /*
+ * stores if we have negotiated with the backend, agreeing to use
+ * persistent grants.
+ */
+ int use_persistent_gnts;
+
+ /*
+ * Used when doing persistent grants to store if we are able to queue
+ * the request by just using existing persistent grants (0), or if we
+ * have to get new grants, as there are not sufficiently many free.
+ */
+ int new_persistent_gnts;
grant_ref_t gref_head;
+ struct page *granted_page;
+ struct gnt_list *gnt_list_entry;
struct scatterlist *sg;
if (unlikely(info->connected != BLKIF_STATE_CONNECTED))
return 1;
- if (gnttab_alloc_grant_references(
- BLKIF_MAX_SEGMENTS_PER_REQUEST, &gref_head) < 0) {
- gnttab_request_free_callback(
- &info->callback,
- blkif_restart_queue_callback,
- info,
- BLKIF_MAX_SEGMENTS_PER_REQUEST);
- return 1;
- }
+ use_persistent_gnts = info->feature_persistent;
+
+ if (info->persistent_gnts_c < BLKIF_MAX_SEGMENTS_PER_REQUEST) {
+ new_persistent_gnts = 1;
+ if (gnttab_alloc_grant_references(
+ BLKIF_MAX_SEGMENTS_PER_REQUEST, &gref_head) < 0) {
+ gnttab_request_free_callback(
+ &info->callback,
+ blkif_restart_queue_callback,
+ info,
+ BLKIF_MAX_SEGMENTS_PER_REQUEST);
+ return 1;
+ }
+ } else
+ new_persistent_gnts = 0;
/* Fill out a communications ring structure. */
ring_req = RING_GET_REQUEST(&info->ring, info->ring.req_prod_pvt);
@@ -341,20 +372,83 @@ static int blkif_queue_request(struct request *req)
BLKIF_MAX_SEGMENTS_PER_REQUEST);
for_each_sg(info->sg, sg, ring_req->u.rw.nr_segments, i) {
- buffer_mfn = pfn_to_mfn(page_to_pfn(sg_page(sg)));
fsect = sg->offset >> 9;
lsect = fsect + (sg->length >> 9) - 1;
- /* install a grant reference. */
- ref = gnttab_claim_grant_reference(&gref_head);
- BUG_ON(ref == -ENOSPC);
- gnttab_grant_foreign_access_ref(
+ if (use_persistent_gnts && info->persistent_gnts_c) {
+ gnt_list_entry = info->persistent_gnts;
+
+ info->persistent_gnts = info->persistent_gnts->
+ tail;
+ ref = gnt_list_entry->gref;
+ buffer_mfn = pfn_to_mfn(gnt_list_entry->pfn);
+ info->persistent_gnts_c--;
+ } else {
+ ref = gnttab_claim_grant_reference(&gref_head);
+ BUG_ON(ref == -ENOSPC);
+
+ if (use_persistent_gnts) {
+ gnt_list_entry =
+ kmalloc(sizeof(struct gnt_list),
+ GFP_ATOMIC);
+ if (!gnt_list_entry)
+ return -ENOMEM;
+
+ granted_page = alloc_page(GFP_ATOMIC);
+ if (!granted_page) {
+ kfree(gnt_list_entry);
+ return -ENOMEM;
+ }
+
+ gnt_list_entry->pfn =
+ page_to_pfn(granted_page);
+ gnt_list_entry->gref = ref;
+ } else
+ granted_page = sg_page(sg);
+
+ buffer_mfn = pfn_to_mfn(page_to_pfn(
+ granted_page));
+ gnttab_grant_foreign_access_ref(
ref,
info->xbdev->otherend_id,
buffer_mfn,
+ !use_persistent_gnts &&
rq_data_dir(req));
+ }
+
+ if (use_persistent_gnts)
+ info->shadow[id].grants_used[i] =
+ gnt_list_entry;
+
+ if (use_persistent_gnts && rq_data_dir(req)) {
+ char *bvec_data;
+ void *shared_data;
+
+ BUG_ON(sg->offset + sg->length > PAGE_SIZE);
+
+ shared_data = kmap_atomic(
+ pfn_to_page(gnt_list_entry->pfn));
+ bvec_data = kmap_atomic(sg_page(sg));
+
+ /*
+ * this does not wipe data stored outside the
+ * range sg->offset..sg->offset+sg->length.
+ * Therefore, blkback *could* see data from
+ * previous requests. This is OK as long as
+ * persistent grants are shared with just one
+ * domain. It may need refactoring if This
+ * changes
+ */
+ memcpy(shared_data + sg->offset,
+ bvec_data + sg->offset,
+ sg->length);
+
+ kunmap_atomic(bvec_data);
+ kunmap_atomic(shared_data);
+ }
info->shadow[id].frame[i] = mfn_to_pfn(buffer_mfn);
+
ring_req->u.rw.seg[i] =
(struct blkif_request_segment) {
.gref = ref,
@@ -368,7 +462,8 @@ static int blkif_queue_request(struct request *req)
/* Keep a private copy so we can reissue requests when recovering. */
info->shadow[id].req = *ring_req;
- gnttab_free_grant_references(gref_head);
+ if (new_persistent_gnts)
+ gnttab_free_grant_references(gref_head);
return 0;
}
@@ -480,12 +575,13 @@ static int xlvbd_init_blk_queue(struct gendisk *gd, u16 sector_size)
static void xlvbd_flush(struct blkfront_info *info)
{
blk_queue_flush(info->rq, info->feature_flush);
- printk(KERN_INFO "blkfront: %s: %s: %s\n",
+ printk(KERN_INFO "blkfront: %s: %s: %s %s\n",
info->gd->disk_name,
info->flush_op == BLKIF_OP_WRITE_BARRIER ?
"barrier" : (info->flush_op == BLKIF_OP_FLUSH_DISKCACHE ?
"flush diskcache" : "barrier or flush"),
- info->feature_flush ? "enabled" : "disabled");
+ info->feature_flush ? "enabled" : "disabled",
+ info->feature_persistent ? "persistent" : "non-persistent");
}
static int xen_translate_vdev(int vdevice, int *minor, unsigned int *offset)
@@ -707,6 +803,8 @@ static void blkif_restart_queue(struct work_struct *work)
static void blkif_free(struct blkfront_info *info, int suspend)
{
+ struct gnt_list *persistent_gnt;
+
/* Prevent new requests being issued until we fix things up. */
spin_lock_irq(&info->io_lock);
info->connected = suspend ?
@@ -714,6 +812,15 @@ static void blkif_free(struct blkfront_info *info, int suspend)
/* No more blkif_request(). */
if (info->rq)
blk_stop_queue(info->rq);
+
+ /* Remove all persistent grants */
+ while (info->persistent_gnts) {
+ persistent_gnt = info->persistent_gnts;
+ info->persistent_gnts = persistent_gnt->tail;
+ gnttab_end_foreign_access(persistent_gnt->gref, 0, 0UL);
+ kfree(persistent_gnt);
+ }
+
/* No more gnttab callback work. */
gnttab_cancel_free_callback(&info->callback);
spin_unlock_irq(&info->io_lock);
@@ -734,13 +841,47 @@ static void blkif_free(struct blkfront_info *info, int suspend)
}
-static void blkif_completion(struct blk_shadow *s)
+static void blkif_completion(struct blk_shadow *s, struct blkfront_info *info,
+ struct blkif_response *bret)
{
int i;
- /* Do not let BLKIF_OP_DISCARD as nr_segment is in the same place
- * flag. */
- for (i = 0; i < s->req.u.rw.nr_segments; i++)
- gnttab_end_foreign_access(s->req.u.rw.seg[i].gref, 0, 0UL);
+ struct gnt_list *new_gnt_list_entry;
+ struct bio_vec *bvec;
+ struct req_iterator iter;
+ unsigned long flags;
+ char *bvec_data;
+ void *shared_data;
+
+
+ if (info->feature_persistent == 0) {
+ /* Do not let BLKIF_OP_DISCARD as nr_segment is in the same
+ * place flag. */
+ for (i = 0; i < s->req.u.rw.nr_segments; i++)
+ gnttab_end_foreign_access(s->req.u.rw.seg[i].gref,
+ 0, 0UL);
+ return;
+ }
+
+ i = 0;
+ if (bret->operation == BLKIF_OP_READ)
+ rq_for_each_segment(bvec, s->request, iter) {
+ BUG_ON(bvec->bv_offset + bvec->bv_len > PAGE_SIZE);
+
+ shared_data = kmap_atomic
+ (pfn_to_page(s->grants_used[i++]->pfn));
+ bvec_data = bvec_kmap_irq(bvec, &flags);
+ memcpy(bvec_data, shared_data + bvec->bv_offset,
+ bvec->bv_len);
+ bvec_kunmap_irq(bvec_data, &flags);
+ kunmap_atomic(shared_data);
+ }
+ /* Add the persistent grant into the list of free grants */
+ for (i = 0; i < s->req.u.rw.nr_segments; i++) {
+ new_gnt_list_entry = s->grants_used[i];
+ new_gnt_list_entry->tail = info->persistent_gnts;
+ info->persistent_gnts = new_gnt_list_entry;
+ info->persistent_gnts_c++;
+ }
}
static irqreturn_t blkif_interrupt(int irq, void *dev_id)
@@ -783,7 +924,7 @@ static irqreturn_t blkif_interrupt(int irq, void *dev_id)
req = info->shadow[id].request;
if (bret->operation != BLKIF_OP_DISCARD)
- blkif_completion(&info->shadow[id]);
+ blkif_completion(&info->shadow[id], info, bret);
if (add_id_to_freelist(info, id)) {
WARN(1, "%s: response to %s (id %ld) couldn't be recycled!\n",
@@ -942,6 +1083,13 @@ again:
message = "writing protocol";
goto abort_transaction;
}
+ err = xenbus_printf(xbt, dev->nodename,
+ "feature-persistent-grants", "%d", 1);
+ if (err) {
+ dev_warn(&dev->dev,
+ "writing persistent grants feature to xenbus");
+ info->feature_persistent = 0;
+ }
err = xenbus_transaction_end(xbt, 0);
if (err) {
@@ -1029,6 +1177,7 @@ static int blkfront_probe(struct xenbus_device *dev,
spin_lock_init(&info->io_lock);
info->xbdev = dev;
info->vdevice = vdevice;
+ info->persistent_gnts_c = 0;
info->connected = BLKIF_STATE_DISCONNECTED;
INIT_WORK(&info->work, blkif_restart_queue);
@@ -1093,6 +1242,7 @@ static int blkif_recover(struct blkfront_info *info)
req->u.rw.seg[j].gref,
info->xbdev->otherend_id,
pfn_to_mfn(info->shadow[req->u.rw.id].frame[j]),
+ !info->feature_persistent &&
rq_data_dir(info->shadow[req->u.rw.id].request));
}
info->shadow[req->u.rw.id].req = *req;
@@ -1225,7 +1375,7 @@ static void blkfront_connect(struct blkfront_info *info)
unsigned long sector_size;
unsigned int binfo;
int err;
- int barrier, flush, discard;
+ int barrier, flush, discard, persistent;
switch (info->connected) {
case BLKIF_STATE_CONNECTED:
@@ -1296,6 +1446,19 @@ static void blkfront_connect(struct blkfront_info *info)
info->flush_op = BLKIF_OP_FLUSH_DISKCACHE;
}
+ /*
+ * Are we dealing with an old blkback that will unmap
+ * all grefs?
+ */
+ err = xenbus_gather(XBT_NIL, info->xbdev->otherend,
+ "feature-persistent-grants", "%d", &persistent,
+ NULL);
+
+ if (err)
+ info->feature_persistent = 0;
+ else
+ info->feature_persistent = persistent;
+
err = xenbus_gather(XBT_NIL, info->xbdev->otherend,
"feature-discard", "%d", &discard,
NULL);
diff --git a/include/xen/interface/io/blkif.h b/include/xen/interface/io/blkif.h
index ee338bf..9048df1 100644
--- a/include/xen/interface/io/blkif.h
+++ b/include/xen/interface/io/blkif.h
@@ -109,6 +109,19 @@ typedef uint64_t blkif_sector_t;
*/
#define BLKIF_MAX_SEGMENTS_PER_REQUEST 11
+/*
+ * Maximum number of inflight requests that can be sent by from
+ * blkfront, and will be guaranteed to be persistently mapped.
+ * BLKIF_MAX_SEGMENTS_PER_REQUEST *
+ * BLKIF_MAX_PERSISTENT_REQUESTS_PER_DEV is the maximum number of
+ * pages that blkback will persistently map. This is set to be the size
+ * of the ring, as this is a limit on the number of requests that can
+ * be inflight at any one time. 64 imposes an overhead of 2.75MB of
+ * mapped kernel space per interface.
+ */
+#define BLKIF_MAX_PERSISTENT_REQUESTS_PER_DEV 64
+
+
struct blkif_request_rw {
uint8_t nr_segments; /* number of segments */
blkif_vdev_t handle; /* only for read/write requests */
--
1.7.9.5
^ permalink raw reply related [flat|nested] 13+ messages in thread
end of thread, other threads:[~2012-10-30 20:51 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-10-24 16:58 [PATCH v2] Persistent grant maps for xen blk drivers Roger Pau Monne
2012-10-29 13:57 ` Konrad Rzeszutek Wilk
2012-10-30 17:01 ` Konrad Rzeszutek Wilk
2012-10-30 18:33 ` Roger Pau Monné
2012-10-30 20:38 ` Konrad Rzeszutek Wilk
-- strict thread matches above, loose matches on Subject: below --
2012-09-21 15:52 Oliver Chick
2012-09-21 18:41 ` Konrad Rzeszutek Wilk
2012-09-21 18:56 ` Konrad Rzeszutek Wilk
2012-09-21 20:46 ` Konrad Rzeszutek Wilk
2012-09-24 14:38 ` Andres Lagar-Cavilla
2012-09-24 15:06 ` Konrad Rzeszutek Wilk
2012-09-24 15:21 ` Andres Lagar-Cavilla
2012-09-27 15:49 ` Oliver Chick
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).