[RFC PATCH v2, part3 00/11] Introduce PCI bus lock and state machine

[RFC PATCH v2, part3 00/11] Introduce PCI bus lock and state machine

[Jiang Liu]

Hi all,
	About one year ago, I started to solve issues triggered by concurrent
PCI device hotplug operations, and several different solutions have been
sent to community for comments. Now we have made important changes to the
solution, so start again with a "v1".

This is the second take to resolve race conditions when hot-plugging PCI
devices/host bridges. Instead of using a globla lock to serialize all hotplug
operations as in previous version, now we introduce a state machine and lock
mechanism for PCI buses to serialize hotplug operations. For discussions
related to previous version, please refer to:
http://comments.gmane.org/gmane.linux.kernel.pci/15007

This patch-set is still in early stages, so sending it out just requesting
for comments. Any comments are welcomed, especially about whether it's the
right/suitable way to solve these race condition issues.

There are multiple methods to trigger PCI hotplug requests/operations
concurrently, such as:
1. Sysfs interfaces exported by the PCI core subsystem
	/sys/devices/pcissss:bb/ssss:bb:dd.f/.../remove
	/sys/devices/pcissss:bb/ssss:bb:dd.f/.../rescan
	/sys/devices/pcissss:bb/ssss:bb:dd.f/.../pci_bus/ssss:bb/rescan
	/sys/bus/pci/rescan
2. Sysfs interfaces exported by the PCI hotplug subsystem
	/sys/bus/pci/slots/xx/power
3. PCI hotplug events triggered by PCI Hotplug Controllers
4. ACPI hotplug events for PCI host bridges
5. Driver binding/unbinding events
	binding/unbinding pci drivers with SR-IOV support

With current implementation, the PCI core subsystem doesn't support
concurrent hotplug operations yet. The existing pci_bus_sem lock only
protects several lists in struct pci_bus, such as children list,
devices list, but it doesn't protect the pci_bus or pci_dev structure
themselves.

Let's take pci_remove_bus_device() as an example, which are used by
PCI hotplug drivers to hot-remove PCI devices.  Currently all these
are free running without any protection, so it can't support reentrance.
pci_remove_bus_device()
    ->pci_stop_bus_device()
        ->pci_stop_bus_device()
            ->pci_stop_bus_devices()
        ->pci_stop_dev()

Patch 1-4:
1) Introduce PCI bus lock and state machine
2) Introduce a state machine for PCI devices to enable reentrance

Patch 5-8:
1) Enhance PCI core to support new PCI bus lock and state machine
2) Refine xen-pcifront code with new PCI interfaces

Patch 9-10:
1) Give example about how to enhance PCI hotplug drivers with PCI bus
   lcok and state machine

Patch 11:
1) Give an example about how to enhance arch specific PCI code with PCI
   bus lock and state machine

Jiang Liu (11):
  PCI: introduce bus lock and state machine to serialize PCI hotplug
    operations
  PCI: implement state machine for PCI bus
  PCI: introduce a state machine to manage PCI device lifecycle
  PCI: introduce helper function pci_stop_and_remove_device()
  PCI: enhance PCI core logic to support PCI bus lock
  PCI, sysfs: use PCI bus lock to serialize hotplug operations triggered
    by sysfs
  PCI, xen-pcifront: use new PCI interfaces to simplify implementation
  PCI, xen-pcifront: use PCI bus lock to protect PCI device hotplug
  PCI, acpiphp: use PCI bus lock to protect PCI device hotplug
  PCI, pciehp: use PCI bus lock to protect PCI device hotplug
  PCI, ACPI, pci_root: use PCI bus lock to protect PCI device hotplug

 arch/ia64/pci/pci.c                |   2 +-
 arch/x86/pci/acpi.c                |   3 +-
 drivers/acpi/pci_root.c            |  14 +-
 drivers/pci/Kconfig                |   3 +
 drivers/pci/bus.c                  | 274 ++++++++++++++++++++++++++++++++++++-
 drivers/pci/hotplug-pci.c          |   4 +
 drivers/pci/hotplug/acpiphp_glue.c |  16 ++-
 drivers/pci/hotplug/pciehp_pci.c   |  15 ++
 drivers/pci/pci-sysfs.c            |  52 ++++---
 drivers/pci/probe.c                |  33 ++++-
 drivers/pci/remove.c               | 151 +++++++++++++-------
 drivers/pci/xen-pcifront.c         |  95 +++++++------
 include/linux/pci.h                |  63 ++++++++-
 13 files changed, 588 insertions(+), 137 deletions(-)

-- 
1.8.1.2

[RFC PATCH v2, part3 01/11] PCI: introduce bus lock and state machine to serialize PCI hotplug operations

[Jiang Liu]

There are multiple ways to trigger concurrent PCI hotplug operations for
a specific PCI bus, but we have no way to serialize those PCI hotplug
operations yet and thus it may break the PCI hotplug logic. This patch
introduces a lock mechanism and a state machine for PCI bus to serialize
PCI hotplug operations.

The state machine is as below:
        _________________________________________________________________
       |                                                                 v
INITIALIZED->REGISTERED->STARTED->STOPPING->STOPPED->REMOVING->REMOVED->DESTOYED
                  |_____________________________________^

 INITIALIZED:	the bus structure has just been allocated
 REGISTERED:	the device for the bus has been registered into the
		device tree
 STARTED:	PCI devices under the bus has been started
 STOPPING:	start to stop devices on the bus
 STOPPED:	child PCI devices under the bus has been stopped
 REMOVING:	start to remove the PCI bus
 REMOVED:	the bus has been removed from the PCI tree
 DESTROYED:	memory for the bus has been freed, just for debug

The interfaces to manage PCI bus state are:
static inline int pci_bus_get_state(const struct pci_bus *bus);
extern void pci_bus_change_state(struct pci_bus *bus, int new_state,
				 int old_state, bool unlock);

And another three interfaces for PCI bus lock:
extern int pci_bus_lock(struct pci_bus *bus, int states, bool recursive);
extern int pci_bus_lock_timeout(struct pci_bus *bus, int states,
				bool recursive, long timeout);
extern void pci_bus_unlock(struct pci_bus *bus, bool recursive);

The locking rules are simple:
1) The bus must be locked when changing state of child devices.
2) The bus must be locked when changing state of itself.

A typical usage looks like:
if (pci_bus_lock(bus, PCI_BUS_STATE_STARTED, true) == 0) {
	do_pci_hotplug();
	pci_bus_unlock(bus, true);
}

Note:
The PCI_BUS_LOCK config option is a temporary solution to avoid breaking
bisects, it will be removed when all architectures have been enhanced to
support the new PCI bus state and lock mechanism.

Signed-off-by: Jiang Liu <jiang.liu@huawei.com>
Signed-off-by: Yijing Wang <wangyijing@huawei.com>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: linux-pci@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
 drivers/pci/Kconfig |   3 +
 drivers/pci/bus.c   | 251 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/pci.h |  47 ++++++++++
 3 files changed, 301 insertions(+)

diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig
index 6d51aa6..92e69bd 100644
--- a/drivers/pci/Kconfig
+++ b/drivers/pci/Kconfig
@@ -119,3 +119,6 @@ config PCI_IOAPIC
 config PCI_LABEL
 	def_bool y if (DMI || ACPI)
 	select NLS
+
+config PCI_BUS_LOCK
+	def_bool n
diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
index f7e1f76..b232775 100644
--- a/drivers/pci/bus.c
+++ b/drivers/pci/bus.c
@@ -15,9 +15,12 @@
 #include <linux/proc_fs.h>
 #include <linux/init.h>
 #include <linux/slab.h>
+#include <linux/sched.h>
 
 #include "pci.h"
 
+static DECLARE_WAIT_QUEUE_HEAD(pci_bus_state_wait_queue);
+
 void pci_add_resource_offset(struct list_head *resources, struct resource *res,
 			     resource_size_t offset)
 {
@@ -297,6 +300,254 @@ void pci_bus_put(struct pci_bus *bus)
 }
 EXPORT_SYMBOL(pci_bus_put);
 
+/**
+ * pci_bus_change_state - Change state of @bus from @old_state to @new_state
+ * @bus: the PCI bus to change state
+ * @old_state: old state of @bus
+ * @new_state: new state for @bus
+ * @unlock: unlock @bus if true
+ *
+ * Change state of @bus from @old_state to @new_state, and unlock @bus
+ * if @unlock is true.
+ * Note:
+ * - @new_state must be bigger than @old_state
+ * - must be called with @bus locked.
+ */
+void pci_bus_change_state(struct pci_bus *bus, int old_state, int new_state,
+			  bool unlock)
+{
+	int t;
+
+	BUG_ON(!pci_bus_is_locked(bus));
+	BUG_ON(new_state < old_state || pci_bus_get_state(bus) != old_state ||
+	       (new_state & ~PCI_BUS_STATE_MASK));
+
+	old_state |= PCI_BUS_STATE_LOCK;
+	if (!unlock)
+		new_state |= PCI_BUS_STATE_LOCK;
+
+	do {
+		t = atomic_read(&bus->state);
+		t &= ~(PCI_BUS_STATE_MASK | PCI_BUS_STATE_LOCK);
+	} while (atomic_cmpxchg(&bus->state, t | old_state, t | new_state)
+		 != (t | old_state));
+
+	wake_up_all(&pci_bus_state_wait_queue);
+}
+
+/* Wait for bus to reach one of the specified states. */
+static bool pci_bus_wait_for_states(struct pci_bus *bus, int states)
+{
+	int t = atomic_read(&bus->state);
+
+	/* Bus state is bigger than any of the specified states. */
+	if ((t & PCI_BUS_STATE_MASK) > states)
+		return true;
+
+	/* Bus is in one of the specified states and unlocked. */
+	if ((t & states) && !(t & PCI_BUS_STATE_LOCK))
+		return true;
+
+	return false;
+}
+
+static int pci_bus_lock_one(struct pci_bus *bus, int states, long *to)
+{
+	int t;
+
+	BUG_ON(states & ~PCI_BUS_STATE_MASK);
+	do {
+		do {
+			if (*to <= 0)
+				return -ETIMEDOUT;
+			*to = wait_event_timeout(pci_bus_state_wait_queue,
+				pci_bus_wait_for_states(bus, states), *to);
+			t = atomic_read(&bus->state);
+			if (((t & PCI_BUS_STATE_MASK) > states))
+				return -EBUSY;
+		} while (!(t & states));
+
+		t &= ~PCI_BUS_STATE_LOCK;
+	} while (atomic_cmpxchg(&bus->state, t , t | PCI_BUS_STATE_LOCK) != t);
+
+	return 0;
+}
+
+static int ___pci_bus_lock(struct pci_bus *bus, int states, bool recursive,
+			   long *to, struct pci_bus **marker)
+{
+	int ret;
+	struct pci_bus *child;
+
+	ret = pci_bus_lock_one(bus, states, to);
+	if (ret < 0) {
+		*marker = bus;
+	} else if (recursive) {
+		list_for_each_entry(child, &bus->children, node) {
+			ret = ___pci_bus_lock(child, states, recursive,
+					      to, marker);
+			if (ret < 0)
+				break;
+		}
+	}
+
+	return ret;
+}
+
+static int __pci_bus_unlock(struct pci_bus *bus, bool recursive,
+			    struct pci_bus *mark)
+{
+	int t, ret = 0;
+	struct pci_bus *child;
+
+	if (unlikely(bus == mark))
+		return 1;
+
+	if (recursive)
+		list_for_each_entry(child, &bus->children, node) {
+			ret = __pci_bus_unlock(child, recursive, mark);
+			if (ret)
+				break;
+		}
+
+	BUG_ON(!pci_bus_is_locked(bus));
+	do {
+		t = atomic_read(&bus->state);
+	} while (atomic_cmpxchg(&bus->state, t, t & ~PCI_BUS_STATE_LOCK) != t);
+
+	return ret;
+}
+
+static int __pci_bus_lock(struct pci_bus *bus, int states, bool recursive,
+			  long *to)
+{
+	int ret;
+	struct pci_bus *mark = NULL;
+
+	ret = ___pci_bus_lock(bus, states, recursive, to, &mark);
+	if (ret < 0) {
+		__pci_bus_unlock(bus, recursive, mark);
+		wake_up_all(&pci_bus_state_wait_queue);
+	}
+
+	return ret;
+}
+
+/**
+ * pci_bus_lock - wait for @bus to reach one of the specified states
+ *		  and then lock it.
+ * @bus: the PCI bus to lock
+ * @states: specified states to wait for
+ * @recursive: lock all the subtree if true
+ *
+ * Wait for @bus to reach one of the states specified by @states and then
+ * lock it. Resursively lock all the subtree if @recursive is true.
+ *
+ * Return values:
+ * - 0 on success
+ * - -EBUSY if failed to lock the bus
+ */
+int pci_bus_lock(struct pci_bus *bus, int states, bool recursive)
+{
+	int ret;
+	long to;
+
+	/*
+	 * When recursively locking the sub-tree starting from @bus,
+	 * __pci_bus_lock() may return -EBUSY if some child buses are under
+	 * hot-removing, so keep retrying until success or @bus is out of
+	 * requested @states.
+	 */
+	do {
+		to = HZ / 10;
+		ret = __pci_bus_lock(bus, states, recursive, &to);
+		if (ret == 0)
+			return 0;
+
+		/*
+		 * Prevent a deadlock scenario that thread A waits for
+		 * all sysfs files to be released while holding PCI bus
+		 * locks, and Thread B tries to acquire PCI bus locks
+		 * in a sysfs handler. These checks break the deadlock
+		 * condition.
+		 */
+		if (pci_bus_get_state(bus) > states)
+			return -EBUSY;
+		if (to > 0)
+			schedule_timeout_interruptible(to);
+	} while (true);
+}
+EXPORT_SYMBOL(pci_bus_lock);
+
+/**
+ * pci_bus_lock_timeout - wait for @bus to reach one of the
+ *			  specified states and then lock it.
+ * @bus: the PCI bus to lock
+ * @states: specified states to wait for
+ * @recursive: lock all the subtree if true
+ * @timeout: maximum number of jiffies to wait
+ *
+ * Wait for @bus to reach one of the states specified by @states and then
+ * lock it. Resursively lock all the subtree if @recursive is true.
+ *
+ * Return values:
+ * - 0 on success
+ * - -ETIMEDOUT if timeouts
+ * - -EBUSY if failed to lock the bus
+ */
+int pci_bus_lock_timeout(struct pci_bus *bus, int states, bool recursive,
+			 long timeout)
+{
+	int ret;
+	long to;
+
+	if (timeout <= 0)
+		return -ETIMEDOUT;
+
+	/*
+	 * When recursively locking the sub-tree starting from @bus,
+	 * __pci_bus_lock() may return -EBUSY if some child buses are under
+	 * hot-removing, so keep retrying until success or @bus is out of
+	 * requested @states.
+	 */
+	do {
+		to = (timeout > HZ / 10) ? (HZ / 10) : timeout;
+		timeout -= to;
+		ret = __pci_bus_lock(bus, states, recursive, &to);
+		if (ret == 0)
+			return 0;
+
+		/*
+		 * Prevent a deadlock scenario that thread A waits for
+		 * all sysfs files to be released while holding PCI bus
+		 * locks, and Thread B tries to acquire PCI bus locks
+		 * in a sysfs handler. These checks break the deadlock
+		 * condition.
+		 */
+		if (pci_bus_get_state(bus) > states)
+			return -EBUSY;
+		if (to > 0)
+			schedule_timeout_interruptible(to);
+	} while (timeout > 0);
+
+	return -ETIMEDOUT;
+}
+EXPORT_SYMBOL(pci_bus_lock_timeout);
+
+/**
+ * pci_bus_unlock - unlock @bus and wake up waiters
+ * bus: PCI bus to unlock
+ * recursive: unlock subtree if recursive is true
+ *
+ * Unlock PCI bus @bus and wake up waiters. Must be called with @bus locked.
+ */
+void pci_bus_unlock(struct pci_bus *bus, bool recursive)
+{
+	__pci_bus_unlock(bus, recursive, NULL);
+	wake_up_all(&pci_bus_state_wait_queue);
+}
+EXPORT_SYMBOL(pci_bus_unlock);
+
 EXPORT_SYMBOL(pci_bus_alloc_resource);
 EXPORT_SYMBOL_GPL(pci_bus_add_device);
 EXPORT_SYMBOL(pci_bus_add_devices);
diff --git a/include/linux/pci.h b/include/linux/pci.h
index c88d4e6..cfe075c 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -450,8 +450,55 @@ struct pci_bus {
 	struct bin_attribute	*legacy_io; /* legacy I/O for this bus */
 	struct bin_attribute	*legacy_mem; /* legacy mem */
 	unsigned int		is_added:1;
+	atomic_t		state;
 };
 
+/*
+ * PCI bus state machine:
+ *         _________________________________________________________________
+ *        |                                                                 v
+ * INITIALIZED->REGISTERED->STARTED->STOPPING->STOPPED->REMOVING->REMOVED->DESTOYED
+ *                   |_____________________________________^
+ */
+#define PCI_BUS_STATE_UNKNOWN		0x0	/* invalid state */
+#define PCI_BUS_STATE_INITIALIZED	0x1	/* bus initialized */
+#define PCI_BUS_STATE_REGISTERED	0x2	/* bus added into device tree */
+#define PCI_BUS_STATE_STARTED		0x4	/* working state */
+#define PCI_BUS_STATE_STOPPING		0x8	/* start to stop children */
+#define PCI_BUS_STATE_STOPPED		0x10	/* child devices stopped */
+#define PCI_BUS_STATE_REMOVING		0x20	/* start to remove PCI */
+#define PCI_BUS_STATE_REMOVED		0x40	/* bus removed */
+#define PCI_BUS_STATE_DESTROYED		0x80	/* used for debug only */
+#define PCI_BUS_STATE_MASK		0xFF
+
+#ifdef	CONFIG_PCI_BUS_LOCK
+#define	PCI_BUS_STATE_LOCK		0x10000	/* for pci core only */
+
+static inline bool pci_bus_is_locked(const struct pci_bus *bus)
+{
+	return !!(atomic_read(&bus->state) & PCI_BUS_STATE_LOCK);
+}
+#else /* CONFIG_PCI_BUS_LOCK */
+#define	PCI_BUS_STATE_LOCK		0x00000	/* for pci core only */
+
+static inline bool pci_bus_is_locked(const struct pci_bus *bus)
+{
+	return true;
+}
+#endif /* CONFIG_PCI_BUS_LOCK */
+
+static inline int pci_bus_get_state(const struct pci_bus *bus)
+{
+	return atomic_read(&bus->state) & PCI_BUS_STATE_MASK;
+}
+
+void pci_bus_change_state(struct pci_bus *bus, int new_state,
+			  int old_state, bool unlock);
+int pci_bus_lock(struct pci_bus *bus, int states, bool recursive);
+int pci_bus_lock_timeout(struct pci_bus *bus, int states,
+			 bool recursive, long timeout);
+void pci_bus_unlock(struct pci_bus *bus, bool recursive);
+
 #define pci_bus_b(n)	list_entry(n, struct pci_bus, node)
 #define to_pci_bus(n)	container_of(n, struct pci_bus, dev)
 #define for_each_pci_bus(b)	for (b = NULL; (b = pci_get_next_bus(b)); )
-- 
1.8.1.2

[RFC PATCH v2, part3 02/11] PCI: implement state machine for PCI bus

[Jiang Liu]

Enhance the PCI core to implement the state machine for PCI buses.
It also enhances PCI bus removal logic by using the state machine.

The state machine will be used to protect PCI buses from concurrent
hotplug operations.

Signed-off-by: Jiang Liu <jiang.liu@huawei.com>
Signed-off-by: Yijing Wang <wangyijing@huawei.com>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: linux-pci@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
 drivers/pci/bus.c    |   9 +++++
 drivers/pci/probe.c  |   8 ++++
 drivers/pci/remove.c | 101 ++++++++++++++++++++++++++++++---------------------
 3 files changed, 76 insertions(+), 42 deletions(-)

diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
index b232775..8ea5972 100644
--- a/drivers/pci/bus.c
+++ b/drivers/pci/bus.c
@@ -200,6 +200,15 @@ void pci_bus_add_devices(const struct pci_bus *bus)
 	struct pci_bus *child;
 	int retval;
 
+	BUG_ON(!pci_bus_is_locked(bus));
+	/* change bus to STARTED state before adding devices */
+	if (pci_bus_get_state(bus) == PCI_BUS_STATE_REGISTERED)
+		pci_bus_change_state((struct pci_bus *)bus,
+			PCI_BUS_STATE_REGISTERED, PCI_BUS_STATE_STARTED, false);
+	/* Return if @bus is going to be removed */
+	if (pci_bus_get_state(bus) != PCI_BUS_STATE_STARTED)
+		return;
+
 	list_for_each_entry(dev, &bus->devices, bus_list) {
 		/* Skip already-added devices */
 		if (dev->is_added)
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index cc5e432..4e24d93 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -90,6 +90,7 @@ static void release_pcibus_dev(struct device *dev)
 		put_device(pci_bus->bridge);
 	pci_bus_remove_resources(pci_bus);
 	pci_release_bus_of_node(pci_bus);
+	atomic_set(&pci_bus->state, PCI_BUS_STATE_DESTROYED);
 	kfree(pci_bus);
 }
 
@@ -471,6 +472,8 @@ static struct pci_bus *pci_alloc_bus(struct pci_ops *ops, void *sd, int bus)
 		b->busn_res.end = 0xff;
 		b->busn_res.flags = IORESOURCE_BUS;
 		b->dev.class = &pcibus_class;
+		atomic_set(&b->state,
+			   PCI_BUS_STATE_INITIALIZED | PCI_BUS_STATE_LOCK);
 		dev_set_name(&b->dev, "%04x:%02x", pci_domain_nr(b), bus);
 		device_initialize(&b->dev);
 	}
@@ -688,6 +691,8 @@ add_dev:
 
 	/* Create legacy_io and legacy_mem files for this bus */
 	pci_create_legacy_files(child);
+	pci_bus_change_state(child, PCI_BUS_STATE_INITIALIZED,
+			     PCI_BUS_STATE_REGISTERED, false);
 
 	return child;
 }
@@ -1760,6 +1765,9 @@ struct pci_bus *pci_create_root_bus(struct device *parent, int bus,
 	list_add_tail(&b->node, &pci_root_buses);
 	up_write(&pci_bus_sem);
 
+	pci_bus_change_state(b, PCI_BUS_STATE_INITIALIZED,
+			     PCI_BUS_STATE_REGISTERED, false);
+
 	return b;
 
 class_dev_reg_err:
diff --git a/drivers/pci/remove.c b/drivers/pci/remove.c
index b0ce875..effe4ff 100644
--- a/drivers/pci/remove.c
+++ b/drivers/pci/remove.c
@@ -3,6 +3,9 @@
 #include <linux/pci-aspm.h>
 #include "pci.h"
 
+static void pci_stop_bus_device(struct pci_dev *dev);
+static void pci_remove_bus_device(struct pci_dev *dev);
+
 static void pci_free_resources(struct pci_dev *dev)
 {
 	int i;
@@ -44,53 +47,75 @@ static void pci_destroy_dev(struct pci_dev *dev)
 
 void pci_remove_bus(struct pci_bus *bus)
 {
-	pci_proc_detach_bus(bus);
+	int state = pci_bus_get_state(bus);
+
+	BUG_ON(!pci_bus_is_locked(bus) || state == PCI_BUS_STATE_REMOVING);
+	if (state >= PCI_BUS_STATE_REMOVED)
+		return;
+
+	pci_bus_change_state(bus, state, PCI_BUS_STATE_REMOVING, false);
 
+	pci_proc_detach_bus(bus);
 	down_write(&pci_bus_sem);
 	list_del(&bus->node);
 	pci_bus_release_busn_res(bus);
 	up_write(&pci_bus_sem);
 	pci_remove_legacy_files(bus);
 	pcibios_remove_bus(bus);
+
 	device_del(&bus->dev);
+	pci_bus_change_state(bus, PCI_BUS_STATE_REMOVING,
+			     PCI_BUS_STATE_REMOVED, true);
 	put_device(&bus->dev);
 }
 EXPORT_SYMBOL(pci_remove_bus);
 
-static void pci_stop_bus_device(struct pci_dev *dev)
+static void pci_bus_stop_devices(struct pci_bus *bus)
 {
-	struct pci_bus *bus = dev->subordinate;
 	struct pci_dev *child, *tmp;
+	int state = pci_bus_get_state(bus);
+
+	BUG_ON(!pci_bus_is_locked(bus) || state == PCI_BUS_STATE_STOPPING);
+	if (state >= PCI_BUS_STATE_STOPPED)
+		return;
 
+	pci_bus_change_state(bus, state, PCI_BUS_STATE_STOPPING, false);
 	/*
 	 * Stopping an SR-IOV PF device removes all the associated VFs,
 	 * which will update the bus->devices list and confuse the
 	 * iterator.  Therefore, iterate in reverse so we remove the VFs
 	 * first, then the PF.
 	 */
-	if (bus) {
-		list_for_each_entry_safe_reverse(child, tmp,
-						 &bus->devices, bus_list)
-			pci_stop_bus_device(child);
-	}
-
-	pci_stop_dev(dev);
+	list_for_each_entry_safe_reverse(child, tmp, &bus->devices, bus_list)
+		pci_stop_bus_device(child);
+	pci_bus_change_state(bus, PCI_BUS_STATE_STOPPING,
+			     PCI_BUS_STATE_STOPPED, false);
 }
 
-static void pci_remove_bus_device(struct pci_dev *dev)
+static void pci_bus_remove_bus(struct pci_bus *bus)
 {
-	struct pci_bus *bus = dev->subordinate;
 	struct pci_dev *child, *tmp;
 
-	if (bus) {
-		list_for_each_entry_safe(child, tmp,
-					 &bus->devices, bus_list)
+	if (pci_bus_get_state(bus) < PCI_BUS_STATE_REMOVED) {
+		list_for_each_entry_safe(child, tmp, &bus->devices, bus_list)
 			pci_remove_bus_device(child);
-
 		pci_remove_bus(bus);
-		dev->subordinate = NULL;
 	}
+}
 
+static void pci_stop_bus_device(struct pci_dev *dev)
+{
+	if (dev->subordinate)
+		pci_bus_stop_devices(dev->subordinate);
+	pci_stop_dev(dev);
+}
+
+static void pci_remove_bus_device(struct pci_dev *dev)
+{
+	if (dev->subordinate) {
+		pci_bus_remove_bus(dev->subordinate);
+		dev->subordinate = NULL;
+	}
 	pci_destroy_dev(dev);
 }
 
@@ -108,6 +133,7 @@ static void pci_remove_bus_device(struct pci_dev *dev)
  */
 void pci_stop_and_remove_bus_device(struct pci_dev *dev)
 {
+	BUG_ON(!pci_bus_is_locked(dev->bus));
 	pci_stop_bus_device(dev);
 	pci_remove_bus_device(dev);
 }
@@ -115,36 +141,27 @@ EXPORT_SYMBOL(pci_stop_and_remove_bus_device);
 
 void pci_stop_root_bus(struct pci_bus *bus)
 {
-	struct pci_dev *child, *tmp;
-	struct pci_host_bridge *host_bridge;
-
-	if (!pci_is_root_bus(bus))
-		return;
-
-	host_bridge = to_pci_host_bridge(bus->bridge);
-	list_for_each_entry_safe_reverse(child, tmp,
-					 &bus->devices, bus_list)
-		pci_stop_bus_device(child);
-
-	/* stop the host bridge */
-	device_del(&host_bridge->dev);
+	BUG_ON(!pci_bus_is_locked(bus));
+	if (pci_is_root_bus(bus) &&
+	    pci_bus_get_state(bus) < PCI_BUS_STATE_STOPPED) {
+		pci_bus_stop_devices(bus);
+		/* stop the host bridge */
+		device_del(bus->bridge);
+	}
 }
 
 void pci_remove_root_bus(struct pci_bus *bus)
 {
-	struct pci_dev *child, *tmp;
 	struct pci_host_bridge *host_bridge;
 
-	if (!pci_is_root_bus(bus))
-		return;
-
-	host_bridge = to_pci_host_bridge(bus->bridge);
-	list_for_each_entry_safe(child, tmp,
-				 &bus->devices, bus_list)
-		pci_remove_bus_device(child);
-	pci_remove_bus(bus);
-	host_bridge->bus = NULL;
+	BUG_ON(!pci_bus_is_locked(bus));
+	if (pci_is_root_bus(bus) &&
+	    pci_bus_get_state(bus) < PCI_BUS_STATE_REMOVED) {
+		host_bridge = to_pci_host_bridge(bus->bridge);
+		pci_bus_remove_bus(bus);
+		host_bridge->bus = NULL;
+		/* remove the host bridge */
+		put_device(&host_bridge->dev);
+	}
 
-	/* remove the host bridge */
-	put_device(&host_bridge->dev);
 }
-- 
1.8.1.2

[RFC PATCH v2, part3 03/11] PCI: introduce a state machine to manage PCI device lifecycle

[Jiang Liu]

struct pci_dev has a one bit flag (is_added) to store PCI device state,
but it's not enough to protect PCI devices from concurrent PCI hotplug
operations. So extend the one bit is_added flag as a three bits
dev_state field and introduce a simple state machine for PCI device
objects as:
	INITIALIZED -> STARTED -> STOPPING -> STOPPED -> REMOVED

The state machine is used to protect PCI devices from multiple invokes
of pci_stop_dev() and pci_destroy_dev() for the same PCI device object.

Signed-off-by: Jiang Liu <jiang.liu@huawei.com>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>
Cc: Yijing Wang <wangyijing@huawei.com>
Cc: linux-pci@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
 drivers/pci/bus.c                  | 14 ++++++++------
 drivers/pci/hotplug/acpiphp_glue.c |  2 +-
 drivers/pci/probe.c                |  7 +++----
 drivers/pci/remove.c               | 31 ++++++++++++++++++++-----------
 include/linux/pci.h                | 15 ++++++++++++++-
 5 files changed, 46 insertions(+), 23 deletions(-)

diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
index 8ea5972..913608e 100644
--- a/drivers/pci/bus.c
+++ b/drivers/pci/bus.c
@@ -183,7 +183,7 @@ int pci_bus_add_device(struct pci_dev *dev)
 	retval = device_attach(&dev->dev);
 	WARN_ON(retval < 0);
 
-	dev->is_added = 1;
+	dev->dev_state = PCI_DEV_STATE_STARTED;
 
 	return 0;
 }
@@ -211,7 +211,7 @@ void pci_bus_add_devices(const struct pci_bus *bus)
 
 	list_for_each_entry(dev, &bus->devices, bus_list) {
 		/* Skip already-added devices */
-		if (dev->is_added)
+		if (dev->dev_state >= PCI_DEV_STATE_STARTED)
 			continue;
 		retval = pci_bus_add_device(dev);
 		if (retval)
@@ -220,10 +220,12 @@ void pci_bus_add_devices(const struct pci_bus *bus)
 	}
 
 	list_for_each_entry(dev, &bus->devices, bus_list) {
-		BUG_ON(!dev->is_added);
-		child = dev->subordinate;
-		if (child)
-			pci_bus_add_devices(child);
+		BUG_ON(dev->dev_state < PCI_DEV_STATE_STARTED);
+		if (dev->dev_state == PCI_DEV_STATE_STARTED) {
+			child = dev->subordinate;
+			if (child)
+				pci_bus_add_devices(child);
+		}
 	}
 }
 
diff --git a/drivers/pci/hotplug/acpiphp_glue.c b/drivers/pci/hotplug/acpiphp_glue.c
index 96fed19..e330331 100644
--- a/drivers/pci/hotplug/acpiphp_glue.c
+++ b/drivers/pci/hotplug/acpiphp_glue.c
@@ -708,7 +708,7 @@ static int __ref enable_device(struct acpiphp_slot *slot)
 
 	list_for_each_entry(dev, &bus->devices, bus_list) {
 		/* Assume that newly added devices are powered on already. */
-		if (!dev->is_added)
+		if (dev->dev_state == PCI_DEV_STATE_INITIALIZED)
 			dev->current_state = PCI_D0;
 	}
 
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 4e24d93..9fcf3a3 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1433,8 +1433,7 @@ static int only_one_child(struct pci_bus *bus)
  * @devfn: slot number to scan (must have zero function.)
  *
  * Scan a PCI slot on the specified PCI bus for devices, adding
- * discovered devices to the @bus->devices list.  New devices
- * will not have is_added set.
+ * discovered devices to the @bus->devices list.
  *
  * Returns the number of new devices found.
  */
@@ -1449,13 +1448,13 @@ int pci_scan_slot(struct pci_bus *bus, int devfn)
 	dev = pci_scan_single_device(bus, devfn);
 	if (!dev)
 		return 0;
-	if (!dev->is_added)
+	if (dev->dev_state == PCI_DEV_STATE_INITIALIZED)
 		nr++;
 
 	for (fn = next_fn(bus, dev, 0); fn > 0; fn = next_fn(bus, dev, fn)) {
 		dev = pci_scan_single_device(bus, devfn + fn);
 		if (dev) {
-			if (!dev->is_added)
+			if (dev->dev_state == PCI_DEV_STATE_INITIALIZED)
 				nr++;
 			dev->multifunction = 1;
 		}
diff --git a/drivers/pci/remove.c b/drivers/pci/remove.c
index effe4ff..2f01f72 100644
--- a/drivers/pci/remove.c
+++ b/drivers/pci/remove.c
@@ -10,7 +10,7 @@ static void pci_free_resources(struct pci_dev *dev)
 {
 	int i;
 
- 	msi_remove_pci_irq_vectors(dev);
+	msi_remove_pci_irq_vectors(dev);
 
 	pci_cleanup_rom(dev);
 	for (i = 0; i < PCI_NUM_RESOURCES; i++) {
@@ -22,27 +22,36 @@ static void pci_free_resources(struct pci_dev *dev)
 
 static void pci_stop_dev(struct pci_dev *dev)
 {
-	pci_pme_active(dev, false);
+	bool started;
+
+	BUG_ON(dev->dev_state == PCI_DEV_STATE_STOPPING);
+	if (dev->dev_state >= PCI_DEV_STATE_STOPPED)
+		return;
 
-	if (dev->is_added) {
+	started = dev->dev_state == PCI_DEV_STATE_STARTED;
+	dev->dev_state = PCI_DEV_STATE_STOPPING;
+	pci_pme_active(dev, false);
+	if (started) {
 		pci_proc_detach_device(dev);
 		pci_remove_sysfs_dev_files(dev);
 		device_del(&dev->dev);
-		dev->is_added = 0;
 	}
-
 	if (dev->bus->self)
 		pcie_aspm_exit_link_state(dev);
+
+	dev->dev_state = PCI_DEV_STATE_STOPPED;
 }
 
 static void pci_destroy_dev(struct pci_dev *dev)
 {
-	down_write(&pci_bus_sem);
-	list_del(&dev->bus_list);
-	up_write(&pci_bus_sem);
-
-	pci_free_resources(dev);
-	put_device(&dev->dev);
+	if (dev->dev_state < PCI_DEV_STATE_REMOVED) {
+		down_write(&pci_bus_sem);
+		list_del(&dev->bus_list);
+		up_write(&pci_bus_sem);
+		pci_free_resources(dev);
+		put_device(&dev->dev);
+		dev->dev_state = PCI_DEV_STATE_REMOVED;
+	}
 }
 
 void pci_remove_bus(struct pci_bus *bus)
diff --git a/include/linux/pci.h b/include/linux/pci.h
index cfe075c..d3f61f8 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -210,6 +210,14 @@ enum pci_bus_speed {
 	PCI_SPEED_UNKNOWN		= 0xff,
 };
 
+enum pci_dev_state {
+	PCI_DEV_STATE_INITIALIZED = 0,
+	PCI_DEV_STATE_STARTED = 1,
+	PCI_DEV_STATE_STOPPING = 2,
+	PCI_DEV_STATE_STOPPED = 3,
+	PCI_DEV_STATE_REMOVED = 4,
+};
+
 struct pci_cap_saved_data {
 	char cap_nr;
 	unsigned int size;
@@ -307,7 +315,7 @@ struct pci_dev {
 	unsigned int	transparent:1;	/* Transparent PCI bridge */
 	unsigned int	multifunction:1;/* Part of multi-function device */
 	/* keep track of device state */
-	unsigned int	is_added:1;
+	unsigned int	dev_state:3;
 	unsigned int	is_busmaster:1; /* device is busmaster */
 	unsigned int	no_msi:1;	/* device may not use msi */
 	unsigned int	block_cfg_access:1;	/* config space access is blocked */
@@ -367,6 +375,11 @@ static inline struct pci_dev *pci_physfn(struct pci_dev *dev)
 struct pci_dev *pci_alloc_dev(struct pci_bus *bus);
 struct pci_dev * __deprecated alloc_pci_dev(void);
 
+static inline int pci_dev_get_state(struct pci_dev *pdev)
+{
+	return (volatile int)pdev->dev_state;
+}
+
 #define	to_pci_dev(n) container_of(n, struct pci_dev, dev)
 #define for_each_pci_dev(d) while ((d = pci_get_device(PCI_ANY_ID, PCI_ANY_ID, d)) != NULL)
 
-- 
1.8.1.2

[RFC PATCH v2, part3 04/11] PCI: introduce helper function pci_stop_and_remove_device()

[Jiang Liu]

Introduce a helper function pci_stop_and_remove_device(), which
removes a PCI device in a safe way by locking parent and all
affected descendant PCI buses.

Signed-off-by: Jiang Liu <jiang.liu@huawei.com>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: linux-pci@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
 drivers/pci/remove.c | 19 +++++++++++++++++++
 include/linux/pci.h  |  1 +
 2 files changed, 20 insertions(+)

diff --git a/drivers/pci/remove.c b/drivers/pci/remove.c
index 2f01f72..8e9b272 100644
--- a/drivers/pci/remove.c
+++ b/drivers/pci/remove.c
@@ -1,6 +1,7 @@
 #include <linux/pci.h>
 #include <linux/module.h>
 #include <linux/pci-aspm.h>
+#include <linux/delay.h>
 #include "pci.h"
 
 static void pci_stop_bus_device(struct pci_dev *dev);
@@ -148,6 +149,24 @@ void pci_stop_and_remove_bus_device(struct pci_dev *dev)
 }
 EXPORT_SYMBOL(pci_stop_and_remove_bus_device);
 
+void pci_stop_and_remove_device(struct pci_dev *pdev)
+{
+	int ret;
+	struct pci_bus *bus = pdev->bus;
+
+	do {
+		ret = pci_bus_lock_timeout(bus, PCI_BUS_STATE_STOPPING - 1,
+					   true, HZ / 10);
+		if (ret == 0) {
+			pci_stop_and_remove_bus_device(pdev);
+			pci_bus_unlock(bus, true);
+			break;
+		}
+	} while (pci_bus_get_state(bus) <= PCI_BUS_STATE_STARTED &&
+		 pci_dev_get_state(pdev) <= PCI_DEV_STATE_STARTED);
+}
+EXPORT_SYMBOL(pci_stop_and_remove_device);
+
 void pci_stop_root_bus(struct pci_bus *bus)
 {
 	BUG_ON(!pci_bus_is_locked(bus));
diff --git a/include/linux/pci.h b/include/linux/pci.h
index d3f61f8..45d4aea 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -817,6 +817,7 @@ struct pci_dev *pci_dev_get(struct pci_dev *dev);
 void pci_dev_put(struct pci_dev *dev);
 void pci_remove_bus(struct pci_bus *b);
 void pci_stop_and_remove_bus_device(struct pci_dev *dev);
+void pci_stop_and_remove_device(struct pci_dev *dev);
 void pci_stop_root_bus(struct pci_bus *bus);
 void pci_remove_root_bus(struct pci_bus *bus);
 void pci_setup_cardbus(struct pci_bus *bus);
-- 
1.8.1.2

[RFC PATCH v2, part3 05/11] PCI: enhance PCI core logic to support PCI bus lock

[Jiang Liu]

Every new PCI bus will be created with the PCI_BUS_STATE_LOCK bit set,
so a typical PCI device hot-add flow is:
	pci_bus_lock(parent, PCI_BUS_STATE_STARTED, true);
	pci_scan_slot(parent, PCI_DEVFN(0, 0));
	pci_bus_add_devices(parent);
	pci_bus_unlock(parent, true);	// recursively unlock all PCI
					// buses, including new PCI buses

This patch also changes behaviors of interfaces to create PCI root buses:
	pci_create_root_bus(): return with new PCI root bus locked
	pci_scan_bus_parented(): return with new PCI root bus locked
	pci_scan_root_bus(): return with new PCI root bus unlocked
	pci_scan_bus(): return with new PCI root bus unlocked

pci_scan_bus_parented() has already been marked as __deprecated and
all in-tree callers have been converted to use other interfaces.
So following-on patches will enhance callers of pci_create_root_bus()
to unlock returned root buses.

Signed-off-by: Jiang Liu <jiang.liu@huawei.com>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: linux-pci@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
 drivers/pci/hotplug-pci.c |  4 ++++
 drivers/pci/probe.c       | 18 ++++++++++++++++++
 2 files changed, 22 insertions(+)

diff --git a/drivers/pci/hotplug-pci.c b/drivers/pci/hotplug-pci.c
index 18533ed..8faa3c2 100644
--- a/drivers/pci/hotplug-pci.c
+++ b/drivers/pci/hotplug-pci.c
@@ -10,6 +10,10 @@ int __ref pci_hp_add_bridge(struct pci_dev *dev)
 	int pass, busnr, start = parent->busn_res.start;
 	int end = parent->busn_res.end;
 
+	BUG_ON(!pci_bus_is_locked(parent));
+	if (pci_bus_get_state(parent) > PCI_BUS_STATE_STARTED)
+		return 0;
+
 	for (busnr = start; busnr <= end; busnr++) {
 		if (!pci_bus_exists(pci_domain_nr(parent), busnr))
 			break;
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 9fcf3a3..0ddf5b8 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -745,6 +745,10 @@ int pci_scan_bridge(struct pci_bus *bus, struct pci_dev *dev, int max, int pass)
 	u8 primary, secondary, subordinate;
 	int broken = 0;
 
+	BUG_ON(!pci_bus_is_locked(bus));
+	if (pci_bus_get_state(bus) > PCI_BUS_STATE_STARTED)
+		return max;
+
 	pci_read_config_dword(dev, PCI_PRIMARY_BUS, &buses);
 	primary = buses & 0xFF;
 	secondary = (buses >> 8) & 0xFF;
@@ -799,6 +803,8 @@ int pci_scan_bridge(struct pci_bus *bus, struct pci_dev *dev, int max, int pass)
 			pci_bus_insert_busn_res(child, secondary, subordinate);
 			child->bridge_ctl = bctl;
 			pci_bus_get(child);
+		} else {
+			BUG_ON(!pci_bus_is_locked(child));
 		}
 
 		cmax = pci_scan_child_bus(child);
@@ -837,6 +843,8 @@ int pci_scan_bridge(struct pci_bus *bus, struct pci_dev *dev, int max, int pass)
 				goto out;
 			pci_bus_insert_busn_res(child, max, 0xff);
 			pci_bus_get(child);
+		} else {
+			BUG_ON(!pci_bus_is_locked(child));
 		}
 		buses = (buses & 0xff000000)
 		      | ((unsigned int)(child->primary)     <<  0)
@@ -1322,6 +1330,7 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
 {
 	int ret;
 
+	BUG_ON(!pci_bus_is_locked(bus));
 	device_initialize(&dev->dev);
 	dev->dev.release = pci_release_dev;
 
@@ -1369,6 +1378,10 @@ struct pci_dev *__ref pci_scan_single_device(struct pci_bus *bus, int devfn)
 {
 	struct pci_dev *dev;
 
+	BUG_ON(!pci_bus_is_locked(bus));
+	if (pci_bus_get_state(bus) > PCI_BUS_STATE_STARTED)
+		return NULL;
+
 	dev = pci_get_slot(bus, devfn);
 	if (dev) {
 		pci_dev_put(dev);
@@ -1624,6 +1637,8 @@ unsigned int pci_scan_child_bus(struct pci_bus *bus)
 	unsigned int devfn, pass, max = bus->busn_res.start;
 	struct pci_dev *dev;
 
+	BUG_ON(!pci_bus_is_locked(bus));
+	BUG_ON(pci_bus_get_state(bus) > PCI_BUS_STATE_STARTED);
 	dev_dbg(&bus->dev, "scanning bus\n");
 
 	/* Go find them, Rover! */
@@ -1872,6 +1887,8 @@ struct pci_bus *pci_scan_root_bus(struct device *parent, int bus,
 		pci_bus_update_busn_res_end(b, max);
 
 	pci_bus_add_devices(b);
+	pci_bus_unlock(b, true);
+
 	return b;
 }
 EXPORT_SYMBOL(pci_scan_root_bus);
@@ -1908,6 +1925,7 @@ struct pci_bus *pci_scan_bus(int bus, struct pci_ops *ops,
 	if (b) {
 		pci_scan_child_bus(b);
 		pci_bus_add_devices(b);
+		pci_bus_unlock(b, true);
 	} else {
 		pci_free_resource_list(&resources);
 	}
-- 
1.8.1.2

[RFC PATCH v2, part3 06/11] PCI, sysfs: use PCI bus lock to serialize hotplug operations triggered by sysfs

[Jiang Liu]

Use PCI bus lock to serialize hotplug operations triggered by pci-sysfs,
and remove the redundant local mutex pci_remove_rescan_mutex.

This also fixes the bug reported by Gu Zheng as:
	echo -n 1 > /sys/bus/pci/devices/0000\:10\:00.0/remove ; echo -n 1 >
/sys/bus/pci/devices/0000\:1a\:01.0/remove

will cause kernel crash as bus get freed.

[  418.946462] CPU 4
[  418.968377] Pid: 512, comm: kworker/u:2 Tainted: G        W    3.8.0 #2
FUJITSU-SV PRIMEQUEST 1800E/SB
[  419.081763] RIP: 0010:[<ffffffff8137972e>]  [<ffffffff8137972e>]
pci_bus_read_config_word+0x5e/0x90
[  420.494137] Call Trace:
[  420.523326]  [<ffffffff813851ef>] ? remove_callback+0x1f/0x40
[  420.591984]  [<ffffffff8138044b>] pci_pme_active+0x4b/0x1c0
[  420.658545]  [<ffffffff8137d8e7>] pci_stop_bus_device+0x57/0xb0
[  420.729259]  [<ffffffff8137dab6>] pci_stop_and_remove_bus_device+0x16/0x30
[  420.811392]  [<ffffffff813851fb>] remove_callback+0x2b/0x40
[  420.877955]  [<ffffffff81257a56>] sysfs_schedule_callback_work+0x26/0x70

https://bugzilla.kernel.org/show_bug.cgi?id=54411

Signed-off-by: Jiang Liu <jiang.liu@huawei.com>
Reported-by: Gu Zheng <guz.fnst@cn.fujitsu.com>
Cc: linux-pci@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
 drivers/pci/pci-sysfs.c | 52 ++++++++++++++++++++++++++++++++-----------------
 1 file changed, 34 insertions(+), 18 deletions(-)

diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c
index fcc4bb2..91ff11e 100644
--- a/drivers/pci/pci-sysfs.c
+++ b/drivers/pci/pci-sysfs.c
@@ -284,7 +284,6 @@ msi_bus_store(struct device *dev, struct device_attribute *attr,
 	return count;
 }
 
-static DEFINE_MUTEX(pci_remove_rescan_mutex);
 static ssize_t bus_rescan_store(struct bus_type *bus, const char *buf,
 				size_t count)
 {
@@ -293,13 +292,15 @@ static ssize_t bus_rescan_store(struct bus_type *bus, const char *buf,
 
 	if (strict_strtoul(buf, 0, &val) < 0)
 		return -EINVAL;
+	if (!val)
+		return count;
 
-	if (val) {
-		mutex_lock(&pci_remove_rescan_mutex);
-		for_each_pci_root_bus(b)
+	for_each_pci_root_bus(b)
+		if (pci_bus_lock(b, PCI_BUS_STATE_STOPPING - 1, true) == 0) {
 			pci_rescan_bus(b);
-		mutex_unlock(&pci_remove_rescan_mutex);
-	}
+			pci_bus_unlock(b, true);
+		}
+
 	return count;
 }
 
@@ -312,27 +313,41 @@ static ssize_t
 dev_rescan_store(struct device *dev, struct device_attribute *attr,
 		 const char *buf, size_t count)
 {
+	int ret;
 	unsigned long val;
 	struct pci_dev *pdev = to_pci_dev(dev);
 
 	if (strict_strtoul(buf, 0, &val) < 0)
 		return -EINVAL;
+	if (!val)
+		return count;
+
+	do {
+		ret = pci_bus_lock_timeout(pdev->bus,
+				PCI_BUS_STATE_STOPPING - 1, true, HZ);
+		if (ret == 0) {
+			pci_rescan_bus(pdev->bus);
+			pci_bus_unlock(pdev->bus, true);
+			break;
+		}
+		/*
+		 * Prevent a deadlock scenario that thread A waits for
+		 * all sysfs files to be released while holding PCI bus
+		 * locks, and Thread B tries to acquire PCI bus locks
+		 * in a sysfs handler. These checks break the deadlock
+		 * condition.
+		 */
+		if (pci_dev_get_state(pdev) >= PCI_DEV_STATE_STOPPING ||
+		    pci_bus_get_state(pdev->bus) >= PCI_BUS_STATE_STOPPING)
+			return -EBUSY;
+	} while (true);
 
-	if (val) {
-		mutex_lock(&pci_remove_rescan_mutex);
-		pci_rescan_bus(pdev->bus);
-		mutex_unlock(&pci_remove_rescan_mutex);
-	}
 	return count;
 }
 
 static void remove_callback(struct device *dev)
 {
-	struct pci_dev *pdev = to_pci_dev(dev);
-
-	mutex_lock(&pci_remove_rescan_mutex);
-	pci_stop_and_remove_bus_device(pdev);
-	mutex_unlock(&pci_remove_rescan_mutex);
+	pci_stop_and_remove_device(to_pci_dev(dev));
 }
 
 static ssize_t
@@ -366,12 +381,13 @@ dev_bus_rescan_store(struct device *dev, struct device_attribute *attr,
 		return -EINVAL;
 
 	if (val) {
-		mutex_lock(&pci_remove_rescan_mutex);
+		if (pci_bus_lock(bus, PCI_BUS_STATE_STOPPING - 1, true) < 0)
+			return -EBUSY;
 		if (!pci_is_root_bus(bus) && list_empty(&bus->devices))
 			pci_rescan_bus_bridge_resize(bus->self);
 		else
 			pci_rescan_bus(bus);
-		mutex_unlock(&pci_remove_rescan_mutex);
+		pci_bus_unlock(bus, true);
 	}
 	return count;
 }
-- 
1.8.1.2

[RFC PATCH v2, part3 07/11] PCI, xen-pcifront: use new PCI interfaces to simplify implementation

[Jiang Liu]

Use new PCI interfaces to simplify xen-pcifront implementation:
1) Use pci_create_root_bus() instead of pci_scan_bus_parented()
   because pci_scan_bus_parented() is marked as __deprecated.This
   also gets rid of a duplicated call of pci_bus_start_devices().
2) Use pci_stop_root_bus() and pci_remove_root_bus() instead of
   open-coded private implementation.
3) Use pci_set_host_bridge_release() to release data structures
   associated with PCI root buses.
4) Use pci_bus_get()/pci_bus_put() to manage PCI root bus reference
   count.

This is also a preparation for coming PCI bus lock enhancement.

Signed-off-by: Jiang Liu <jiang.liu@huawei.com>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Jeremy Fitzhardinge <jeremy@goop.org>
Cc: xen-devel@lists.xensource.com
Cc: virtualization@lists.linux-foundation.org
Cc: linux-pci@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
 drivers/pci/xen-pcifront.c | 81 ++++++++++++++++++++++------------------------
 1 file changed, 39 insertions(+), 42 deletions(-)

diff --git a/drivers/pci/xen-pcifront.c b/drivers/pci/xen-pcifront.c
index 816cf94..6aa2c0f 100644
--- a/drivers/pci/xen-pcifront.c
+++ b/drivers/pci/xen-pcifront.c
@@ -25,11 +25,6 @@
 #define INVALID_GRANT_REF (0)
 #define INVALID_EVTCHN    (-1)
 
-struct pci_bus_entry {
-	struct list_head list;
-	struct pci_bus *bus;
-};
-
 #define _PDEVB_op_active		(0)
 #define PDEVB_op_active			(1 << (_PDEVB_op_active))
 
@@ -47,12 +42,12 @@ struct pcifront_device {
 	struct xen_pci_sharedinfo *sh_info;
 	struct work_struct op_work;
 	unsigned long flags;
-
 };
 
 struct pcifront_sd {
 	int domain;
 	struct pcifront_device *pdev;
+	struct resource busn_res;
 };
 
 static inline struct pcifront_device *
@@ -67,6 +62,12 @@ static inline void pcifront_init_sd(struct pcifront_sd *sd,
 {
 	sd->domain = domain;
 	sd->pdev = pdev;
+
+	/* Xen pci-backend doesn't export P2P bridges */
+	sd->busn_res.start = bus;
+	sd->busn_res.end = bus;
+	sd->busn_res.flags = IORESOURCE_BUS;
+	sd->busn_res.name = "PCI busn";
 }
 
 static DEFINE_SPINLOCK(pcifront_dev_lock);
@@ -441,12 +442,19 @@ static int pcifront_scan_bus(struct pcifront_device *pdev,
 	return 0;
 }
 
+static void pcifront_release_sd(struct pci_host_bridge *bridge)
+{
+	struct pcifront_sd *sd = bridge->release_data;
+
+	kfree(sd);
+}
+
 static int pcifront_scan_root(struct pcifront_device *pdev,
 				 unsigned int domain, unsigned int bus)
 {
 	struct pci_bus *b;
 	struct pcifront_sd *sd = NULL;
-	struct pci_bus_entry *bus_entry = NULL;
+	LIST_HEAD(resources);
 	int err = 0;
 
 #ifndef CONFIG_PCI_DOMAINS
@@ -463,16 +471,18 @@ static int pcifront_scan_root(struct pcifront_device *pdev,
 	dev_info(&pdev->xdev->dev, "Creating PCI Frontend Bus %04x:%02x\n",
 		 domain, bus);
 
-	bus_entry = kmalloc(sizeof(*bus_entry), GFP_KERNEL);
-	sd = kmalloc(sizeof(*sd), GFP_KERNEL);
-	if (!bus_entry || !sd) {
+	sd = kzalloc(sizeof(*sd), GFP_KERNEL);
+	if (!sd) {
 		err = -ENOMEM;
 		goto err_out;
 	}
 	pcifront_init_sd(sd, domain, bus, pdev);
 
-	b = pci_scan_bus_parented(&pdev->xdev->dev, bus,
-				  &pcifront_bus_ops, sd);
+	pci_add_resource(&resources, &ioport_resource);
+	pci_add_resource(&resources, &iomem_resource);
+	pci_add_resource(&resources, &sd->busn_res);
+	b = pci_create_root_bus(&pdev->xdev->dev, bus, &pcifront_bus_ops,
+				sd, &resources);
 	if (!b) {
 		dev_err(&pdev->xdev->dev,
 			"Error creating PCI Frontend Bus!\n");
@@ -480,12 +490,14 @@ static int pcifront_scan_root(struct pcifront_device *pdev,
 		goto err_out;
 	}
 
-	bus_entry->bus = b;
+	pci_set_host_bridge_release(to_pci_host_bridge(b->bridge),
+				    pcifront_release_sd, sd);
 
-	list_add(&bus_entry->list, &pdev->root_buses);
-
-	/* pci_scan_bus_parented skips devices which do not have a have
-	* devfn==0. The pcifront_scan_bus enumerates all devfn. */
+	/*
+	 * Every PCI physical device should have function 0, but that's not
+	 * true for xen.
+	 * So use pcifront_scan_bus() instead of pci_scan_child_bus().
+	 */
 	err = pcifront_scan_bus(pdev, domain, bus, b);
 
 	/* Claim resources before going "live" with our devices */
@@ -497,7 +509,6 @@ static int pcifront_scan_root(struct pcifront_device *pdev,
 	return err;
 
 err_out:
-	kfree(bus_entry);
 	kfree(sd);
 
 	return err;
@@ -539,35 +550,19 @@ static int pcifront_rescan_root(struct pcifront_device *pdev,
 	return err;
 }
 
-static void free_root_bus_devs(struct pci_bus *bus)
-{
-	struct pci_dev *dev;
-
-	while (!list_empty(&bus->devices)) {
-		dev = container_of(bus->devices.next, struct pci_dev,
-				   bus_list);
-		dev_dbg(&dev->dev, "removing device\n");
-		pci_stop_and_remove_bus_device(dev);
-	}
-}
-
 static void pcifront_free_roots(struct pcifront_device *pdev)
 {
-	struct pci_bus_entry *bus_entry, *t;
+	struct pcifront_sd *sd;
+	struct pci_bus *bus;
 
 	dev_dbg(&pdev->xdev->dev, "cleaning up root buses\n");
 
-	list_for_each_entry_safe(bus_entry, t, &pdev->root_buses, list) {
-		list_del(&bus_entry->list);
-
-		free_root_bus_devs(bus_entry->bus);
-
-		kfree(bus_entry->bus->sysdata);
-
-		device_unregister(bus_entry->bus->bridge);
-		pci_remove_bus(bus_entry->bus);
-
-		kfree(bus_entry);
+	for_each_pci_root_bus(bus) {
+		sd = bus->sysdata;
+		if (sd->pdev == pdev) {
+			pci_stop_root_bus(bus);
+			pci_remove_root_bus(bus);
+		}
 	}
 }
 
@@ -670,6 +665,7 @@ static irqreturn_t pcifront_handler_aer(int irq, void *dev)
 	schedule_pcifront_aer_op(pdev);
 	return IRQ_HANDLED;
 }
+
 static int pcifront_connect_and_init_dma(struct pcifront_device *pdev)
 {
 	int err = 0;
@@ -705,6 +701,7 @@ static void pcifront_disconnect(struct pcifront_device *pdev)
 
 	spin_unlock(&pcifront_dev_lock);
 }
+
 static struct pcifront_device *alloc_pdev(struct xenbus_device *xdev)
 {
 	struct pcifront_device *pdev;
-- 
1.8.1.2

[RFC PATCH v2, part3 08/11] PCI, xen-pcifront: use PCI bus lock to protect PCI device hotplug

[Jiang Liu]

Use PCI bus lock to protect concurrent PCI device hotplug operations.

Signed-off-by: Jiang Liu <jiang.liu@huawei.com>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Jeremy Fitzhardinge <jeremy@goop.org>
Cc: xen-devel@lists.xensource.com
Cc: virtualization@lists.linux-foundation.org
Cc: linux-pci@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
 drivers/pci/xen-pcifront.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/xen-pcifront.c b/drivers/pci/xen-pcifront.c
index 6aa2c0f..2b213b3 100644
--- a/drivers/pci/xen-pcifront.c
+++ b/drivers/pci/xen-pcifront.c
@@ -505,6 +505,7 @@ static int pcifront_scan_root(struct pcifront_device *pdev,
 
 	/* Create SysFS and notify udev of the devices. Aka: "going live" */
 	pci_bus_add_devices(b);
+	pci_bus_unlock(b, true);
 
 	return err;
 
@@ -538,6 +539,11 @@ static int pcifront_rescan_root(struct pcifront_device *pdev,
 		/* If the bus is unknown, create it. */
 		return pcifront_scan_root(pdev, domain, bus);
 
+	if (pci_bus_lock(b, PCI_BUS_STATE_STOPPING - 1, true) < 0) {
+		err = -EBUSY;
+		goto out;
+	}
+
 	err = pcifront_scan_bus(pdev, domain, bus, b);
 
 	/* Claim resources before going "live" with our devices */
@@ -545,6 +551,9 @@ static int pcifront_rescan_root(struct pcifront_device *pdev,
 
 	/* Create SysFS and notify udev of the devices. Aka: "going live" */
 	pci_bus_add_devices(b);
+	pci_bus_unlock(b, true);
+
+out:
 	pci_bus_put(b);
 
 	return err;
@@ -559,8 +568,11 @@ static void pcifront_free_roots(struct pcifront_device *pdev)
 
 	for_each_pci_root_bus(bus) {
 		sd = bus->sysdata;
-		if (sd->pdev == pdev) {
+		if (sd->pdev != pdev)
+			continue;
+		if (pci_bus_lock(bus, PCI_BUS_STATE_REMOVED - 1, true) == 0) {
 			pci_stop_root_bus(bus);
+			/* it also unlocks PCI buses */
 			pci_remove_root_bus(bus);
 		}
 	}
@@ -1042,7 +1054,7 @@ static int pcifront_detach_devices(struct pcifront_device *pdev)
 				domain, bus, slot, func);
 			continue;
 		}
-		pci_stop_and_remove_bus_device(pci_dev);
+		pci_stop_and_remove_device(pci_dev);
 		pci_dev_put(pci_dev);
 
 		dev_dbg(&pdev->xdev->dev,
-- 
1.8.1.2

[RFC PATCH v2, part3 09/11] PCI, acpiphp: use PCI bus lock to protect PCI device hotplug

[Jiang Liu]

Use PCI bus lock to protect concurrent PCI device hotplug operations.

Signed-off-by: Jiang Liu <jiang.liu@huawei.com>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>
Cc: Yijing Wang <wangyijing@huawei.com>
Cc: linux-pci@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
 drivers/pci/hotplug/acpiphp_glue.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/hotplug/acpiphp_glue.c b/drivers/pci/hotplug/acpiphp_glue.c
index e330331..d105dcd 100644
--- a/drivers/pci/hotplug/acpiphp_glue.c
+++ b/drivers/pci/hotplug/acpiphp_glue.c
@@ -674,6 +674,9 @@ static int __ref enable_device(struct acpiphp_slot *slot)
 	if (slot->flags & SLOT_ENABLED)
 		goto err_exit;
 
+	if (pci_bus_lock(bus, PCI_BUS_STATE_STOPPING - 1, true) < 0)
+		goto err_exit;
+
 	list_for_each_entry(func, &slot->funcs, sibling)
 		acpiphp_bus_add(func);
 
@@ -681,7 +684,7 @@ static int __ref enable_device(struct acpiphp_slot *slot)
 	if (num == 0) {
 		/* Maybe only part of funcs are added. */
 		dbg("No new device found\n");
-		goto err_exit;
+		goto out_unlock;
 	}
 
 	max = acpiphp_max_busnr(bus);
@@ -726,8 +729,9 @@ static int __ref enable_device(struct acpiphp_slot *slot)
 		}
 	}
 
-
- err_exit:
+out_unlock:
+	pci_bus_unlock(bus, true);
+err_exit:
 	return 0;
 }
 
@@ -757,7 +761,10 @@ static int disable_device(struct acpiphp_slot *slot)
 {
 	struct acpiphp_func *func;
 	struct pci_dev *pdev;
+	struct pci_bus *bus = slot->bridge->pci_bus;
 
+	if (pci_bus_lock(bus, PCI_BUS_STATE_STOPPING - 1, true) < 0)
+		return -EBUSY;
 	/*
 	 * enable_device() enumerates all functions in this device via
 	 * pci_scan_slot(), whether they have associated ACPI hotplug
@@ -768,6 +775,7 @@ static int disable_device(struct acpiphp_slot *slot)
 		pci_stop_and_remove_bus_device(pdev);
 		pci_dev_put(pdev);
 	}
+	pci_bus_unlock(bus, true);
 
 	list_for_each_entry(func, &slot->funcs, sibling) {
 		acpiphp_bus_trim(func->handle);
-- 
1.8.1.2

[RFC PATCH v2, part3 10/11] PCI, pciehp: use PCI bus lock to protect PCI device hotplug

[Jiang Liu]

Use PCI bus lock to protect concurrent PCI device hotplug operations.

Signed-off-by: Jiang Liu <jiang.liu@huawei.com>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: Yijing Wang <wangyijing@huawei.com>
Cc: linux-pci@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
 drivers/pci/hotplug/pciehp_pci.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/drivers/pci/hotplug/pciehp_pci.c b/drivers/pci/hotplug/pciehp_pci.c
index aac7a40..ddc29ae 100644
--- a/drivers/pci/hotplug/pciehp_pci.c
+++ b/drivers/pci/hotplug/pciehp_pci.c
@@ -51,8 +51,14 @@ int pciehp_configure_device(struct slot *p_slot)
 		return -EINVAL;
 	}
 
+	if (pci_bus_lock(parent, PCI_BUS_STATE_STOPPING - 1, true) < 0) {
+		ctrl_err(ctrl, "Parent bus has been removed\n");
+		return -EBUSY;
+	}
+
 	num = pci_scan_slot(parent, PCI_DEVFN(0, 0));
 	if (num == 0) {
+		pci_bus_unlock(parent, true);
 		ctrl_err(ctrl, "No new device found\n");
 		return -ENODEV;
 	}
@@ -72,6 +78,7 @@ int pciehp_configure_device(struct slot *p_slot)
 	}
 
 	pci_bus_add_devices(parent);
+	pci_bus_unlock(parent, true);
 
 	return 0;
 }
@@ -88,6 +95,12 @@ int pciehp_unconfigure_device(struct slot *p_slot)
 
 	ctrl_dbg(ctrl, "%s: domain:bus:dev = %04x:%02x:00\n",
 		 __func__, pci_domain_nr(parent), parent->number);
+
+	if (pci_bus_lock(parent, PCI_BUS_STATE_REMOVED - 1, true) < 0) {
+		ctrl_dbg(ctrl, "Parent bus has been removed\n");
+		return 0;
+	}
+
 	ret = pciehp_get_adapter_status(p_slot, &presence);
 	if (ret)
 		presence = 0;
@@ -119,5 +132,7 @@ int pciehp_unconfigure_device(struct slot *p_slot)
 		pci_dev_put(dev);
 	}
 
+	pci_bus_unlock(parent, true);
+
 	return rc;
 }
-- 
1.8.1.2

[RFC PATCH v2, part3 11/11] PCI, ACPI, pci_root: use PCI bus lock to protect PCI device hotplug

[Jiang Liu]

Use PCI bus lock to protect concurrent PCI device hotplug operations.

Signed-off-by: Jiang Liu <jiang.liu@huawei.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: x86@kernel.org
Cc: Len Brown <lenb@kernel.org>
Cc: "Rafael J. Wysocki" <rjw@sisk.pl>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: Feng Tang <feng.tang@intel.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-acpi@vger.kernel.org
---
 arch/ia64/pci/pci.c     |  2 +-
 arch/x86/pci/acpi.c     |  3 ++-
 drivers/acpi/pci_root.c | 14 ++++++++------
 drivers/pci/Kconfig     |  2 +-
 4 files changed, 12 insertions(+), 9 deletions(-)

diff --git a/arch/ia64/pci/pci.c b/arch/ia64/pci/pci.c
index de1474f..6e91e87 100644
--- a/arch/ia64/pci/pci.c
+++ b/arch/ia64/pci/pci.c
@@ -384,7 +384,7 @@ struct pci_bus *pci_acpi_scan_root(struct acpi_pci_root *root)
 	}
 
 	pci_scan_child_bus(pbus);
-	return pbus;
+	return pci_bus_get(pbus);
 
 out3:
 	kfree(controller->window);
diff --git a/arch/x86/pci/acpi.c b/arch/x86/pci/acpi.c
index b972f04..6dddc06 100644
--- a/arch/x86/pci/acpi.c
+++ b/arch/x86/pci/acpi.c
@@ -534,7 +534,7 @@ struct pci_bus *pci_acpi_scan_root(struct acpi_pci_root *root)
 		 */
 		memcpy(bus->sysdata, sd, sizeof(*sd));
 		kfree(info);
-		pci_bus_put(bus);
+		BUG_ON(pci_bus_lock(bus, PCI_BUS_STATE_REMOVED - 1, true) < 0);
 	} else {
 		probe_pci_root_info(info, device, busnum, domain);
 
@@ -561,6 +561,7 @@ struct pci_bus *pci_acpi_scan_root(struct acpi_pci_root *root)
 			pci_set_host_bridge_release(
 				to_pci_host_bridge(bus->bridge),
 				release_pci_root_info, info);
+			pci_bus_get(bus);
 		} else {
 			pci_free_resource_list(&resources);
 			__release_pci_root_info(info);
diff --git a/drivers/acpi/pci_root.c b/drivers/acpi/pci_root.c
index 21dda5a..eb01cc7 100644
--- a/drivers/acpi/pci_root.c
+++ b/drivers/acpi/pci_root.c
@@ -535,6 +535,7 @@ static int acpi_pci_root_add(struct acpi_device *device,
 	}
 
 	pci_bus_add_devices(root->bus);
+	pci_bus_unlock(root->bus, true);
 	return 1;
 
 end:
@@ -546,13 +547,14 @@ static void acpi_pci_root_remove(struct acpi_device *device)
 {
 	struct acpi_pci_root *root = acpi_driver_data(device);
 
-	pci_stop_root_bus(root->bus);
-
-	device_set_run_wake(root->bus->bridge, false);
-	pci_acpi_remove_bus_pm_notifier(device);
-
-	pci_remove_root_bus(root->bus);
+	if (pci_bus_lock(root->bus, PCI_BUS_STATE_REMOVED - 1, true) == 0) {
+		pci_stop_root_bus(root->bus);
+		device_set_run_wake(root->bus->bridge, false);
+		pci_acpi_remove_bus_pm_notifier(device);
+		pci_remove_root_bus(root->bus);
+	}
 
+	pci_bus_put(root->bus);
 	kfree(root);
 }
 
diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig
index 92e69bd..5ae3504 100644
--- a/drivers/pci/Kconfig
+++ b/drivers/pci/Kconfig
@@ -121,4 +121,4 @@ config PCI_LABEL
 	select NLS
 
 config PCI_BUS_LOCK
-	def_bool n
+	def_bool y if (X86 || IA64)
-- 
1.8.1.2

Re: [RFC PATCH v2, part3 11/11] PCI, ACPI, pci_root: use PCI bus lock to protect PCI device hotplug

[Rafael J. Wysocki]

On Thursday, May 16, 2013 11:50:59 PM Jiang Liu wrote:
> Use PCI bus lock to protect concurrent PCI device hotplug operations.
> 
> Signed-off-by: Jiang Liu <jiang.liu@huawei.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: "H. Peter Anvin" <hpa@zytor.com>
> Cc: x86@kernel.org
> Cc: Len Brown <lenb@kernel.org>
> Cc: "Rafael J. Wysocki" <rjw@sisk.pl>
> Cc: Yinghai Lu <yinghai@kernel.org>
> Cc: Feng Tang <feng.tang@intel.com>
> Cc: linux-kernel@vger.kernel.org
> Cc: linux-acpi@vger.kernel.org

Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

> ---
>  arch/ia64/pci/pci.c     |  2 +-
>  arch/x86/pci/acpi.c     |  3 ++-
>  drivers/acpi/pci_root.c | 14 ++++++++------
>  drivers/pci/Kconfig     |  2 +-
>  4 files changed, 12 insertions(+), 9 deletions(-)
> 
> diff --git a/arch/ia64/pci/pci.c b/arch/ia64/pci/pci.c
> index de1474f..6e91e87 100644
> --- a/arch/ia64/pci/pci.c
> +++ b/arch/ia64/pci/pci.c
> @@ -384,7 +384,7 @@ struct pci_bus *pci_acpi_scan_root(struct acpi_pci_root *root)
>  	}
>  
>  	pci_scan_child_bus(pbus);
> -	return pbus;
> +	return pci_bus_get(pbus);
>  
>  out3:
>  	kfree(controller->window);
> diff --git a/arch/x86/pci/acpi.c b/arch/x86/pci/acpi.c
> index b972f04..6dddc06 100644
> --- a/arch/x86/pci/acpi.c
> +++ b/arch/x86/pci/acpi.c
> @@ -534,7 +534,7 @@ struct pci_bus *pci_acpi_scan_root(struct acpi_pci_root *root)
>  		 */
>  		memcpy(bus->sysdata, sd, sizeof(*sd));
>  		kfree(info);
> -		pci_bus_put(bus);
> +		BUG_ON(pci_bus_lock(bus, PCI_BUS_STATE_REMOVED - 1, true) < 0);
>  	} else {
>  		probe_pci_root_info(info, device, busnum, domain);
>  
> @@ -561,6 +561,7 @@ struct pci_bus *pci_acpi_scan_root(struct acpi_pci_root *root)
>  			pci_set_host_bridge_release(
>  				to_pci_host_bridge(bus->bridge),
>  				release_pci_root_info, info);
> +			pci_bus_get(bus);
>  		} else {
>  			pci_free_resource_list(&resources);
>  			__release_pci_root_info(info);
> diff --git a/drivers/acpi/pci_root.c b/drivers/acpi/pci_root.c
> index 21dda5a..eb01cc7 100644
> --- a/drivers/acpi/pci_root.c
> +++ b/drivers/acpi/pci_root.c
> @@ -535,6 +535,7 @@ static int acpi_pci_root_add(struct acpi_device *device,
>  	}
>  
>  	pci_bus_add_devices(root->bus);
> +	pci_bus_unlock(root->bus, true);
>  	return 1;
>  
>  end:
> @@ -546,13 +547,14 @@ static void acpi_pci_root_remove(struct acpi_device *device)
>  {
>  	struct acpi_pci_root *root = acpi_driver_data(device);
>  
> -	pci_stop_root_bus(root->bus);
> -
> -	device_set_run_wake(root->bus->bridge, false);
> -	pci_acpi_remove_bus_pm_notifier(device);
> -
> -	pci_remove_root_bus(root->bus);
> +	if (pci_bus_lock(root->bus, PCI_BUS_STATE_REMOVED - 1, true) == 0) {
> +		pci_stop_root_bus(root->bus);
> +		device_set_run_wake(root->bus->bridge, false);
> +		pci_acpi_remove_bus_pm_notifier(device);
> +		pci_remove_root_bus(root->bus);
> +	}
>  
> +	pci_bus_put(root->bus);
>  	kfree(root);
>  }
>  
> diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig
> index 92e69bd..5ae3504 100644
> --- a/drivers/pci/Kconfig
> +++ b/drivers/pci/Kconfig
> @@ -121,4 +121,4 @@ config PCI_LABEL
>  	select NLS
>  
>  config PCI_BUS_LOCK
> -	def_bool n
> +	def_bool y if (X86 || IA64)
> 
-- 
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.

Re: [RFC PATCH v2, part3 00/11] Introduce PCI bus lock and state machine

[Gu Zheng]

On 05/16/2013 11:50 PM, Jiang Liu wrote:

> Hi all,
> 	About one year ago, I started to solve issues triggered by concurrent
> PCI device hotplug operations, and several different solutions have been
> sent to community for comments. Now we have made important changes to the
> solution, so start again with a "v1".
> 
> This is the second take to resolve race conditions when hot-plugging PCI
> devices/host bridges. Instead of using a globla lock to serialize all hotplug
> operations as in previous version, now we introduce a state machine and lock
> mechanism for PCI buses to serialize hotplug operations. For discussions
> related to previous version, please refer to:
> http://comments.gmane.org/gmane.linux.kernel.pci/15007
> 
> This patch-set is still in early stages, so sending it out just requesting
> for comments. Any comments are welcomed, especially about whether it's the
> right/suitable way to solve these race condition issues.
> 
> There are multiple methods to trigger PCI hotplug requests/operations
> concurrently, such as:
> 1. Sysfs interfaces exported by the PCI core subsystem
> 	/sys/devices/pcissss:bb/ssss:bb:dd.f/.../remove
> 	/sys/devices/pcissss:bb/ssss:bb:dd.f/.../rescan
> 	/sys/devices/pcissss:bb/ssss:bb:dd.f/.../pci_bus/ssss:bb/rescan
> 	/sys/bus/pci/rescan
> 2. Sysfs interfaces exported by the PCI hotplug subsystem
> 	/sys/bus/pci/slots/xx/power
> 3. PCI hotplug events triggered by PCI Hotplug Controllers
> 4. ACPI hotplug events for PCI host bridges
> 5. Driver binding/unbinding events
> 	binding/unbinding pci drivers with SR-IOV support

Hi Jiang,
	We're gladly to see the new patch-set, and I tested all the whole 3 parts
patch-set on the latest kernel tree. Because of the limitation of our test environment,
we only use the method *1* to trigger PCI hotplug concurrently, and it works well.

Though I'm not sure whether the way you introduced is the best one to solve these
race condition issues, but it does fix the concurrent PCI hotplug(via sysfs)
issues that disturb us.
Let's wait and see other guys' feedback.:)

Thanks,
Gu

> 
> With current implementation, the PCI core subsystem doesn't support
> concurrent hotplug operations yet. The existing pci_bus_sem lock only
> protects several lists in struct pci_bus, such as children list,
> devices list, but it doesn't protect the pci_bus or pci_dev structure
> themselves.
> 
> Let's take pci_remove_bus_device() as an example, which are used by
> PCI hotplug drivers to hot-remove PCI devices.  Currently all these
> are free running without any protection, so it can't support reentrance.
> pci_remove_bus_device()
>     ->pci_stop_bus_device()
>         ->pci_stop_bus_device()
>             ->pci_stop_bus_devices()
>         ->pci_stop_dev()
> 
> Patch 1-4:
> 1) Introduce PCI bus lock and state machine
> 2) Introduce a state machine for PCI devices to enable reentrance
> 
> Patch 5-8:
> 1) Enhance PCI core to support new PCI bus lock and state machine
> 2) Refine xen-pcifront code with new PCI interfaces
> 
> Patch 9-10:
> 1) Give example about how to enhance PCI hotplug drivers with PCI bus
>    lcok and state machine
> 
> Patch 11:
> 1) Give an example about how to enhance arch specific PCI code with PCI
>    bus lock and state machine
> 
> Jiang Liu (11):
>   PCI: introduce bus lock and state machine to serialize PCI hotplug
>     operations
>   PCI: implement state machine for PCI bus
>   PCI: introduce a state machine to manage PCI device lifecycle
>   PCI: introduce helper function pci_stop_and_remove_device()
>   PCI: enhance PCI core logic to support PCI bus lock
>   PCI, sysfs: use PCI bus lock to serialize hotplug operations triggered
>     by sysfs
>   PCI, xen-pcifront: use new PCI interfaces to simplify implementation
>   PCI, xen-pcifront: use PCI bus lock to protect PCI device hotplug
>   PCI, acpiphp: use PCI bus lock to protect PCI device hotplug
>   PCI, pciehp: use PCI bus lock to protect PCI device hotplug
>   PCI, ACPI, pci_root: use PCI bus lock to protect PCI device hotplug
> 
>  arch/ia64/pci/pci.c                |   2 +-
>  arch/x86/pci/acpi.c                |   3 +-
>  drivers/acpi/pci_root.c            |  14 +-
>  drivers/pci/Kconfig                |   3 +
>  drivers/pci/bus.c                  | 274 ++++++++++++++++++++++++++++++++++++-
>  drivers/pci/hotplug-pci.c          |   4 +
>  drivers/pci/hotplug/acpiphp_glue.c |  16 ++-
>  drivers/pci/hotplug/pciehp_pci.c   |  15 ++
>  drivers/pci/pci-sysfs.c            |  52 ++++---
>  drivers/pci/probe.c                |  33 ++++-
>  drivers/pci/remove.c               | 151 +++++++++++++-------
>  drivers/pci/xen-pcifront.c         |  95 +++++++------
>  include/linux/pci.h                |  63 ++++++++-
>  13 files changed, 588 insertions(+), 137 deletions(-)
> 

Re: [RFC PATCH v2, part3 00/11] Introduce PCI bus lock and state machine

[Yinghai Lu]

On Wed, May 22, 2013 at 2:48 AM, Gu Zheng <guz.fnst@cn.fujitsu.com> wrote:
> On 05/16/2013 11:50 PM, Jiang Liu wrote:
>         We're gladly to see the new patch-set, and I tested all the whole 3 parts
> patch-set on the latest kernel tree. Because of the limitation of our test environment,
> we only use the method *1* to trigger PCI hotplug concurrently, and it works well.
>
> Though I'm not sure whether the way you introduced is the best one to solve these
> race condition issues, but it does fix the concurrent PCI hotplug(via sysfs)
> issues that disturb us.

Not sure what happened, at lease after Jiang's first patch, we should still need
https://patchwork.kernel.org/patch/2562431/
 [1/7] PCI: move back pci_proc_attach_devices calling
https://patchwork.kernel.org/patch/2562461/
 [2/7] PCI: move resources and bus_list releasing to pci_release_dev
https://patchwork.kernel.org/patch/2562451/
 [3/7] PCI: Detach driver in pci_stop_device

they will allow pci_stop_and_remove_bus_device is called two times in the
your case.

this one is not needed.
https://patchwork.kernel.org/patch/2562561/
  [4/7] PCI: Fix racing for pci device removing via sysfs
it should be the same as the two pci_alloc_dev patches.

other three are about sriov one....

Thanks

Yinghai

Re: [RFC PATCH v2, part3 07/11] PCI, xen-pcifront: use new PCI interfaces to simplify implementation

[Konrad Rzeszutek Wilk]

On Thu, May 16, 2013 at 11:50:55PM +0800, Jiang Liu wrote:
> Use new PCI interfaces to simplify xen-pcifront implementation:
> 1) Use pci_create_root_bus() instead of pci_scan_bus_parented()
>    because pci_scan_bus_parented() is marked as __deprecated.This
>    also gets rid of a duplicated call of pci_bus_start_devices().
> 2) Use pci_stop_root_bus() and pci_remove_root_bus() instead of
>    open-coded private implementation.
> 3) Use pci_set_host_bridge_release() to release data structures
>    associated with PCI root buses.
> 4) Use pci_bus_get()/pci_bus_put() to manage PCI root bus reference
>    count.
> 
> This is also a preparation for coming PCI bus lock enhancement.
> 
> Signed-off-by: Jiang Liu <jiang.liu@huawei.com>
> Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> Cc: Jeremy Fitzhardinge <jeremy@goop.org>
> Cc: xen-devel@lists.xensource.com
> Cc: virtualization@lists.linux-foundation.org
> Cc: linux-pci@vger.kernel.org
> Cc: linux-kernel@vger.kernel.org
> ---
>  drivers/pci/xen-pcifront.c | 81 ++++++++++++++++++++++------------------------
>  1 file changed, 39 insertions(+), 42 deletions(-)
> 
> diff --git a/drivers/pci/xen-pcifront.c b/drivers/pci/xen-pcifront.c
> index 816cf94..6aa2c0f 100644
> --- a/drivers/pci/xen-pcifront.c
> +++ b/drivers/pci/xen-pcifront.c
> @@ -25,11 +25,6 @@
>  #define INVALID_GRANT_REF (0)
>  #define INVALID_EVTCHN    (-1)
>  
> -struct pci_bus_entry {
> -	struct list_head list;
> -	struct pci_bus *bus;
> -};
> -
>  #define _PDEVB_op_active		(0)
>  #define PDEVB_op_active			(1 << (_PDEVB_op_active))
>  
> @@ -47,12 +42,12 @@ struct pcifront_device {
>  	struct xen_pci_sharedinfo *sh_info;
>  	struct work_struct op_work;
>  	unsigned long flags;
> -
>  };
>  
>  struct pcifront_sd {
>  	int domain;
>  	struct pcifront_device *pdev;
> +	struct resource busn_res;
>  };
>  
>  static inline struct pcifront_device *
> @@ -67,6 +62,12 @@ static inline void pcifront_init_sd(struct pcifront_sd *sd,
>  {
>  	sd->domain = domain;
>  	sd->pdev = pdev;
> +
> +	/* Xen pci-backend doesn't export P2P bridges */
> +	sd->busn_res.start = bus;
> +	sd->busn_res.end = bus;
> +	sd->busn_res.flags = IORESOURCE_BUS;
> +	sd->busn_res.name = "PCI busn";
>  }
>  
>  static DEFINE_SPINLOCK(pcifront_dev_lock);
> @@ -441,12 +442,19 @@ static int pcifront_scan_bus(struct pcifront_device *pdev,
>  	return 0;
>  }
>  
> +static void pcifront_release_sd(struct pci_host_bridge *bridge)
> +{
> +	struct pcifront_sd *sd = bridge->release_data;
> +
> +	kfree(sd);
> +}
> +
>  static int pcifront_scan_root(struct pcifront_device *pdev,
>  				 unsigned int domain, unsigned int bus)
>  {
>  	struct pci_bus *b;
>  	struct pcifront_sd *sd = NULL;
> -	struct pci_bus_entry *bus_entry = NULL;
> +	LIST_HEAD(resources);
>  	int err = 0;
>  
>  #ifndef CONFIG_PCI_DOMAINS
> @@ -463,16 +471,18 @@ static int pcifront_scan_root(struct pcifront_device *pdev,
>  	dev_info(&pdev->xdev->dev, "Creating PCI Frontend Bus %04x:%02x\n",
>  		 domain, bus);
>  
> -	bus_entry = kmalloc(sizeof(*bus_entry), GFP_KERNEL);
> -	sd = kmalloc(sizeof(*sd), GFP_KERNEL);
> -	if (!bus_entry || !sd) {
> +	sd = kzalloc(sizeof(*sd), GFP_KERNEL);
> +	if (!sd) {
>  		err = -ENOMEM;
>  		goto err_out;
>  	}
>  	pcifront_init_sd(sd, domain, bus, pdev);
>  
> -	b = pci_scan_bus_parented(&pdev->xdev->dev, bus,
> -				  &pcifront_bus_ops, sd);
> +	pci_add_resource(&resources, &ioport_resource);
> +	pci_add_resource(&resources, &iomem_resource);
> +	pci_add_resource(&resources, &sd->busn_res);
> +	b = pci_create_root_bus(&pdev->xdev->dev, bus, &pcifront_bus_ops,
> +				sd, &resources);
>  	if (!b) {
>  		dev_err(&pdev->xdev->dev,
>  			"Error creating PCI Frontend Bus!\n");
> @@ -480,12 +490,14 @@ static int pcifront_scan_root(struct pcifront_device *pdev,
>  		goto err_out;
>  	}
>  
> -	bus_entry->bus = b;
> +	pci_set_host_bridge_release(to_pci_host_bridge(b->bridge),
> +				    pcifront_release_sd, sd);
>  
> -	list_add(&bus_entry->list, &pdev->root_buses);
> -
> -	/* pci_scan_bus_parented skips devices which do not have a have
> -	* devfn==0. The pcifront_scan_bus enumerates all devfn. */
> +	/*
> +	 * Every PCI physical device should have function 0, but that's not
> +	 * true for xen.

That is incorrect. There are two types of backends - one of them will
start at zero, but the other might not.

> +	 * So use pcifront_scan_bus() instead of pci_scan_child_bus().
> +	 */
>  	err = pcifront_scan_bus(pdev, domain, bus, b);
>  
>  	/* Claim resources before going "live" with our devices */
> @@ -497,7 +509,6 @@ static int pcifront_scan_root(struct pcifront_device *pdev,
>  	return err;
>  
>  err_out:
> -	kfree(bus_entry);
>  	kfree(sd);
>  
>  	return err;
> @@ -539,35 +550,19 @@ static int pcifront_rescan_root(struct pcifront_device *pdev,
>  	return err;
>  }
>  
> -static void free_root_bus_devs(struct pci_bus *bus)
> -{
> -	struct pci_dev *dev;
> -
> -	while (!list_empty(&bus->devices)) {
> -		dev = container_of(bus->devices.next, struct pci_dev,
> -				   bus_list);
> -		dev_dbg(&dev->dev, "removing device\n");
> -		pci_stop_and_remove_bus_device(dev);
> -	}
> -}
> -
>  static void pcifront_free_roots(struct pcifront_device *pdev)
>  {
> -	struct pci_bus_entry *bus_entry, *t;
> +	struct pcifront_sd *sd;
> +	struct pci_bus *bus;
>  
>  	dev_dbg(&pdev->xdev->dev, "cleaning up root buses\n");
>  
> -	list_for_each_entry_safe(bus_entry, t, &pdev->root_buses, list) {
> -		list_del(&bus_entry->list);
> -
> -		free_root_bus_devs(bus_entry->bus);
> -
> -		kfree(bus_entry->bus->sysdata);
> -
> -		device_unregister(bus_entry->bus->bridge);
> -		pci_remove_bus(bus_entry->bus);
> -
> -		kfree(bus_entry);
> +	for_each_pci_root_bus(bus) {
> +		sd = bus->sysdata;
> +		if (sd->pdev == pdev) {
> +			pci_stop_root_bus(bus);
> +			pci_remove_root_bus(bus);
> +		}
>  	}
>  }
>  
> @@ -670,6 +665,7 @@ static irqreturn_t pcifront_handler_aer(int irq, void *dev)
>  	schedule_pcifront_aer_op(pdev);
>  	return IRQ_HANDLED;
>  }
> +

Stray
>  static int pcifront_connect_and_init_dma(struct pcifront_device *pdev)
>  {
>  	int err = 0;
> @@ -705,6 +701,7 @@ static void pcifront_disconnect(struct pcifront_device *pdev)
>  
>  	spin_unlock(&pcifront_dev_lock);
>  }
> +

Stray
>  static struct pcifront_device *alloc_pdev(struct xenbus_device *xdev)
>  {
>  	struct pcifront_device *pdev;
> -- 
> 1.8.1.2
> 

Re: [RFC PATCH v2, part3 07/11] PCI, xen-pcifront: use new PCI interfaces to simplify implementation

[Jiang Liu]

On Fri 07 Jun 2013 10:50:24 PM CST, Konrad Rzeszutek Wilk wrote:
> On Thu, May 16, 2013 at 11:50:55PM +0800, Jiang Liu wrote:
>> Use new PCI interfaces to simplify xen-pcifront implementation:
>> 1) Use pci_create_root_bus() instead of pci_scan_bus_parented()
>>    because pci_scan_bus_parented() is marked as __deprecated.This
>>    also gets rid of a duplicated call of pci_bus_start_devices().
>> 2) Use pci_stop_root_bus() and pci_remove_root_bus() instead of
>>    open-coded private implementation.
>> 3) Use pci_set_host_bridge_release() to release data structures
>>    associated with PCI root buses.
>> 4) Use pci_bus_get()/pci_bus_put() to manage PCI root bus reference
>>    count.
>>
>> This is also a preparation for coming PCI bus lock enhancement.
>>
>> Signed-off-by: Jiang Liu <jiang.liu@huawei.com>
>> Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
>> Cc: Jeremy Fitzhardinge <jeremy@goop.org>
>> Cc: xen-devel@lists.xensource.com
>> Cc: virtualization@lists.linux-foundation.org
>> Cc: linux-pci@vger.kernel.org
>> Cc: linux-kernel@vger.kernel.org
>> ---
>>  drivers/pci/xen-pcifront.c | 81 ++++++++++++++++++++++------------------------
>>  1 file changed, 39 insertions(+), 42 deletions(-)
>>
>> diff --git a/drivers/pci/xen-pcifront.c b/drivers/pci/xen-pcifront.c
>> index 816cf94..6aa2c0f 100644
>> --- a/drivers/pci/xen-pcifront.c
>> +++ b/drivers/pci/xen-pcifront.c
......
>> @@ -480,12 +490,14 @@ static int pcifront_scan_root(struct pcifront_device *pdev,
>>  		goto err_out;
>>  	}
>>
>> -	bus_entry->bus = b;
>> +	pci_set_host_bridge_release(to_pci_host_bridge(b->bridge),
>> +				    pcifront_release_sd, sd);
>>
>> -	list_add(&bus_entry->list, &pdev->root_buses);
>> -
>> -	/* pci_scan_bus_parented skips devices which do not have a have
>> -	* devfn==0. The pcifront_scan_bus enumerates all devfn. */
>> +	/*
>> +	 * Every PCI physical device should have function 0, but that's not
>> +	 * true for xen.
>
> That is incorrect. There are two types of backends - one of them will
> start at zero, but the other might not.
Hi Konrad,
      Forgive my poor knowledge about Xen:(, so I will skip this change.
Regards!
Gerry

Re: [RFC PATCH v2, part3 07/11] PCI, xen-pcifront: use new PCI interfaces to simplify implementation

[Konrad Rzeszutek Wilk]

On Fri, Jun 07, 2013 at 10:50:24AM -0400, Konrad Rzeszutek Wilk wrote:
> On Thu, May 16, 2013 at 11:50:55PM +0800, Jiang Liu wrote:
> > Use new PCI interfaces to simplify xen-pcifront implementation:
> > 1) Use pci_create_root_bus() instead of pci_scan_bus_parented()
> >    because pci_scan_bus_parented() is marked as __deprecated.This
> >    also gets rid of a duplicated call of pci_bus_start_devices().
> > 2) Use pci_stop_root_bus() and pci_remove_root_bus() instead of
> >    open-coded private implementation.
> > 3) Use pci_set_host_bridge_release() to release data structures
> >    associated with PCI root buses.
> > 4) Use pci_bus_get()/pci_bus_put() to manage PCI root bus reference
> >    count.
> > 
> > This is also a preparation for coming PCI bus lock enhancement.

With this patch from :

 Merge branch 'pci_lock_v3' of https://github.com/jiangliu/linux into testing


it blows up when detaching the device.

Parsing config from /vm-pv-discard.cfg
Daemon running with PID 4062
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.10.0-rc4upstream-00172-g909fe9e-dirty (konrad@phenom.dumpdata.com) (gcc version 4.4.4 20100503 (Red Hat 4.4.4-2) (GCC) ) #2 SMP Fri Jun 7 10:55:54 EDT 2013
[    0.000000] Command line: console=hvc0 debug 
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] Freeing 20000-40000 pfn range: 131072 pages freed
[    0.000000] 1-1 mapping on 20000->100000
[    0.000000] Released 131072 pages of unused memory
[    0.000000] Set 917504 page(s) to 1-1 mapping
[    0.000000] Populating 100000-120000 pfn range: 131072 pages added
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
[    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[    0.000000] Xen: [mem 0x0000000000100000-0x000000001fffffff] usable
[    0.000000] Xen: [mem 0x0000000020000000-0x00000000201fffff] reserved
[    0.000000] Xen: [mem 0x0000000020200000-0x000000003fffffff] unusable
[    0.000000] Xen: [mem 0x0000000040000000-0x00000000401fffff] reserved
[    0.000000] Xen: [mem 0x0000000040200000-0x00000000c6cd3fff] unusable
[    0.000000] Xen: [mem 0x00000000c6cd4000-0x00000000c6d1cfff] ACPI NVS
[    0.000000] Xen: [mem 0x00000000c6d1d000-0x00000000c6d27fff] ACPI data
[    0.000000] Xen: [mem 0x00000000c6d28000-0x00000000c6d28fff] ACPI NVS
[    0.000000] Xen: [mem 0x00000000c6d29000-0x00000000c6d49fff] reserved
[    0.000000] Xen: [mem 0x00000000c6d4a000-0x00000000c6d4bfff] unusable
[    0.000000] Xen: [mem 0x00000000c6d4c000-0x00000000c6d6cfff] ACPI NVS
[    0.000000] Xen: [mem 0x00000000c6d6d000-0x00000000c6d8ffff] reserved
[    0.000000] Xen: [mem 0x00000000c6d90000-0x00000000c6d9cfff] ACPI NVS
[    0.000000] Xen: [mem 0x00000000c6d9d000-0x00000000c6d9ffff] reserved
[    0.000000] Xen: [mem 0x00000000c6da0000-0x00000000c6db0fff] ACPI NVS
[    0.000000] Xen: [mem 0x00000000c6db1000-0x00000000c6ddcfff] reserved
[    0.000000] Xen: [mem 0x00000000c6ddd000-0x00000000c6e1ffff] ACPI NVS
[    0.000000] Xen: [mem 0x00000000c6e20000-0x00000000c6ffffff] unusable
[    0.000000] Xen: [mem 0x00000000c7800000-0x00000000cf9fffff] reserved
[    0.000000] Xen: [mem 0x00000000fed1c000-0x00000000fed3ffff] reserved
[    0.000000] Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
[    0.000000] Xen: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[    0.000000] Xen: [mem 0x0000000100000000-0x000000011fffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI not present or invalid.
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] No AGP bridge found
[    0.000000] e820: last_pfn = 0x120000 max_arch_pfn = 0x400000000
[    0.000000] e820: last_pfn = 0x20000 max_arch_pfn = 0x400000000
[    0.000000] Scanning 1 areas for low memory corruption
[    0.000000] Base memory trampoline at [ffff88000009a000] 9a000 size 24576
[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[    0.000000]  [mem 0x00000000-0x000fffff] page 4k
[    0.000000] init_memory_mapping: [mem 0x11fe00000-0x11fffffff]
[    0.000000]  [mem 0x11fe00000-0x11fffffff] page 4k
[    0.000000] BRK [0x01d8a000, 0x01d8afff] PGTABLE
[    0.000000] BRK [0x01d8b000, 0x01d8bfff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0x11c000000-0x11fdfffff]
[    0.000000]  [mem 0x11c000000-0x11fdfffff] page 4k
[    0.000000] BRK [0x01d8c000, 0x01d8cfff] PGTABLE
[    0.000000] BRK [0x01d8d000, 0x01d8dfff] PGTABLE
[    0.000000] BRK [0x01d8e000, 0x01d8efff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0x100000000-0x11bffffff]
[    0.000000]  [mem 0x100000000-0x11bffffff] page 4k
[    0.000000] init_memory_mapping: [mem 0x00100000-0x1fffffff]
[    0.000000]  [mem 0x00100000-0x1fffffff] page 4k
[    0.000000] RAMDISK: [mem 0x02197000-0x13f67fff]
[    0.000000] NUMA turned off
[    0.000000] Faking a node at [mem 0x0000000000000000-0x000000011fffffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000-0x11fffffff]
[    0.000000]   NODE_DATA [mem 0x11fea2000-0x11fea5fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
[    0.000000]   Normal   [mem 0x100000000-0x11fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00001000-0x0009ffff]
[    0.000000]   node   0: [mem 0x00100000-0x1fffffff]
[    0.000000]   node   0: [mem 0x100000000-0x11fffffff]
[    0.000000] On node 0 totalpages: 262047
[    0.000000]   DMA zone: 56 pages used for memmap
[    0.000000]   DMA zone: 21 pages reserved
[    0.000000]   DMA zone: 3999 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 1736 pages used for memmap
[    0.000000]   DMA32 zone: 126976 pages, LIFO batch:31
[    0.000000]   Normal zone: 1792 pages used for memmap
[    0.000000]   Normal zone: 131072 pages, LIFO batch:31
[    0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs
[    0.000000] No local APIC present
[    0.000000] APIC: disable apic facility
[    0.000000] APIC: switched to apic NOOP
[    0.000000] nr_irqs_gsi: 16
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
[    0.000000] PM: Registered nosave memory: 0000000020000000 - 0000000020200000
[    0.000000] PM: Registered nosave memory: 0000000020200000 - 0000000040000000
[    0.000000] PM: Registered nosave memory: 0000000040000000 - 0000000040200000
[    0.000000] PM: Registered nosave memory: 0000000040200000 - 00000000c6cd4000
[    0.000000] PM: Registered nosave memory: 00000000c6cd4000 - 00000000c6d1d000
[    0.000000] PM: Registered nosave memory: 00000000c6d1d000 - 00000000c6d28000
[    0.000000] PM: Registered nosave memory: 00000000c6d28000 - 00000000c6d29000
[    0.000000] PM: Registered nosave memory: 00000000c6d29000 - 00000000c6d4a000
[    0.000000] PM: Registered nosave memory: 00000000c6d4a000 - 00000000c6d4c000
[    0.000000] PM: Registered nosave memory: 00000000c6d4c000 - 00000000c6d6d000
[    0.000000] PM: Registered nosave memory: 00000000c6d6d000 - 00000000c6d90000
[    0.000000] PM: Registered nosave memory: 00000000c6d90000 - 00000000c6d9d000
[    0.000000] PM: Registered nosave memory: 00000000c6d9d000 - 00000000c6da0000
[    0.000000] PM: Registered nosave memory: 00000000c6da0000 - 00000000c6db1000
[    0.000000] PM: Registered nosave memory: 00000000c6db1000 - 00000000c6ddd000
[    0.000000] PM: Registered nosave memory: 00000000c6ddd000 - 00000000c6e20000
[    0.000000] PM: Registered nosave memory: 00000000c6e20000 - 00000000c7000000
[    0.000000] PM: Registered nosave memory: 00000000c7000000 - 00000000c7800000
[    0.000000] PM: Registered nosave memory: 00000000c7800000 - 00000000cfa00000
[    0.000000] PM: Registered nosave memory: 00000000cfa00000 - 00000000fed1c000
[    0.000000] PM: Registered nosave memory: 00000000fed1c000 - 00000000fed40000
[    0.000000] PM: Registered nosave memory: 00000000fed40000 - 00000000fee00000
[    0.000000] PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
[    0.000000] PM: Registered nosave memory: 00000000fee01000 - 00000000ff000000
[    0.000000] PM: Registered nosave memory: 00000000ff000000 - 0000000100000000
[    0.000000] e820: [mem 0xcfa00000-0xfed1bfff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 4.3-unstable (preserve-AD)
[    0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:2 nr_node_ids:1
[    0.000000] PERCPU: Embedded 28 pages/cpu @ffff88011f800000 s85632 r8192 d20864 u1048576
[    0.000000] pcpu-alloc: s85632 r8192 d20864 u1048576 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 0 1 
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 258442
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: console=hvc0 debug 
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340
[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Memory: 722480k/4718592k available (6698k kernel code, 3670404k absent, 325708k reserved, 4234k data, 1724k init)
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=2.
[    0.000000] NR_IRQS:33024 nr_irqs:288 16
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [hvc0] enabled
[    0.000000] Xen: using vcpuop timer interface
[    0.000000] installing Xen timer for CPU 0
[    0.000000] tsc: Detected 3092.926 MHz processor
[    0.001000] Calibrating delay loop (skipped), value calculated using timer frequency.. 6185.85 BogoMIPS (lpj=3092926)
[    0.001000] pid_max: default: 32768 minimum: 301
[    0.001000] Security Framework initialized
[    0.001000] SELinux:  Initializing.
[    0.001000] SELinux:  Starting in permissive mode
[    0.001000] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.001000] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.001000] Mount-cache hash table entries: 256
[    0.001582] Initializing cgroup subsys freezer
[    0.001698] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[    0.001698] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
[    0.001708] CPU: Physical Processor ID: 0
[    0.001712] CPU: Processor Core ID: 0
[    0.001718] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0
[    0.001718] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32
[    0.001718] tlb_flushall_shift: 5
[    0.025468] cpu 0 spinlock event irq 17
[    0.025566] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only.
[    0.026081] NMI watchdog: disabled (cpu0): hardware events not enabled
[    0.026680] installing Xen timer for CPU 1
[    0.026723] cpu 1 spinlock event irq 24
[    0.026793] SMP alternatives: switching to SMP code
[    0.048014] Brought up 2 CPUs
[    0.051334] PM: Registering ACPI NVS region [mem 0xc6cd4000-0xc6d1cfff] (299008 bytes)
[    0.051334] PM: Registering ACPI NVS region [mem 0xc6d28000-0xc6d28fff] (4096 bytes)
[    0.051334] PM: Registering ACPI NVS region [mem 0xc6d4c000-0xc6d6cfff] (135168 bytes)
[    0.051334] PM: Registering ACPI NVS region [mem 0xc6d90000-0xc6d9cfff] (53248 bytes)
[    0.051334] PM: Registering ACPI NVS region [mem 0xc6da0000-0xc6db0fff] (69632 bytes)
[    0.051334] PM: Registering ACPI NVS region [mem 0xc6ddd000-0xc6e1ffff] (274432 bytes)
[    0.051399] kworker/u4:0 (19) used greatest stack depth: 6016 bytes left
[    0.052492] Grant tables using version 2 layout.
[    0.052492] Grant table initialized
[    0.071696] RTC time: 165:165:165, date: 165/165/65
[    0.071985] NET: Registered protocol family 16
[    0.072173] kworker/u4:0 (22) used greatest stack depth: 5928 bytes left
[    0.073110] kworker/u4:0 (30) used greatest stack depth: 5464 bytes left
[    0.074030] dca service started, version 1.12.1
[    0.074563] PCI: setting up Xen PCI frontend stub
[    0.074568] PCI: pci_cache_line_size set to 64 bytes
[    0.101148] bio: create slab <bio-0> at 0
[    0.101220] ACPI: Interpreter disabled.
[    0.102003] xen/balloon: Initialising balloon driver.
[    0.103081] xen-balloon: Initialising balloon driver.
[    0.104136] vgaarb: loaded
[    0.105135] usbcore: registered new interface driver usbfs
[    0.105135] usbcore: registered new interface driver hub
[    0.105135] usbcore: registered new device driver usb
[    0.106089] pps_core: LinuxPPS API ver. 1 registered
[    0.106089] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.106089] PTP clock support registered
[    0.106089] PCI: System does not support PCI
[    0.106089] PCI: System does not support PCI
[    0.107154] NetLabel: Initializing
[    0.107154] NetLabel:  domain hash size = 128
[    0.107154] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.107175] NetLabel:  unlabeled traffic allowed by default
[    0.108087] Switching to clocksource xen
[    0.125349] pnp: PnP ACPI: disabled
[    0.138965] NET: Registered protocol family 2
[    0.139578] TCP established hash table entries: 8192 (order: 5, 131072 bytes)
[    0.139647] TCP bind hash table entries: 8192 (order: 5, 131072 bytes)
[    0.139668] TCP: Hash tables configured (established 8192 bind 8192)
[    0.193672] TCP: reno registered
[    0.193693] UDP hash table entries: 512 (order: 2, 16384 bytes)
[    0.193712] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[    0.194030] NET: Registered protocol family 1
[    0.194380] RPC: Registered named UNIX socket transport module.
[    0.194389] RPC: Registered udp transport module.
[    0.194394] RPC: Registered tcp transport module.
[    0.194398] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.194404] PCI: CLS 0 bytes, default 64
[    0.194570] Unpacking initramfs...
[    0.778545] Freeing initrd memory: 292676k freed
[    0.823501] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.823855] Machine check injector initialized
[    0.825040] Scanning for low memory corruption every 60 seconds
[    0.826072] audit: initializing netlink socket (disabled)
[    0.826114] type=2000 audit(1370618465.698:1): initialized
[    0.839204] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.839673] VFS: Disk quotas dquot_6.5.2
[    0.839760] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.840528] NFS: Registering the id_resolver key type
[    0.840553] Key type id_resolver registered
[    0.840558] Key type id_legacy registered
[    0.840587] NTFS driver 2.1.30 [Flags: R/W].
[    0.840919] msgmni has been set to 1982
[    0.841082] SELinux:  Registering netfilter hooks
[    0.842697] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[    0.842706] io scheduler noop registered
[    0.842710] io scheduler deadline registered
[    0.842772] io scheduler cfq registered (default)
[    0.843309] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.846284] intel_idle: does not run on family 6 model 42
[    0.846299] ioatdma: Intel(R) QuickData Technology Driver 4.00
[    0.847400] pcifront pci-0: Installing PCI frontend
[    0.847417] Warning: only able to allocate 4 MB for software IO TLB
[    0.850807] software IO TLB [mem 0x109000000-0x109400000] (4MB) mapped at [ffff880109000000-ffff8801093fffff]
[    0.851175] pcifront pci-0: Creating PCI Frontend Bus 0000:00
[    0.851458] pcifront pci-0: PCI host bridge to bus 0000:00
[    0.851469] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    0.851478] pci_bus 0000:00: root bus resource [mem 0x00000000-0xfffffffff]
[    0.851487] pci_bus 0000:00: root bus resource [bus 00]
[    0.852031] pci 0000:00:00.0: [8086:105e] type 00 class 0x020000
[    0.852419] pci 0000:00:00.0: reg 0x10: [mem 0xfe4a0000-0xfe4bffff]
[    0.852631] pci 0000:00:00.0: reg 0x14: [mem 0xfe480000-0xfe49ffff]
[    0.852743] pci 0000:00:00.0: reg 0x18: [io  0xe020-0xe03f]
[    0.853905] pcifront pci-0: New device on 0000:00:00.0 found.
[    0.857483] pcifront pci-0: claiming resource 0000:00:00.0/0
[    0.857490] pcifront pci-0: claiming resource 0000:00:00.0/1
[    0.857495] pcifront pci-0: claiming resource 0000:00:00.0/2
[    0.939621] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.942227] Non-volatile memory driver v1.3
[    0.942414] Linux agpgart interface v0.103
[    0.943295] [drm] Initialized drm 1.1.0 20060810
[    0.947952] loop: module loaded
[    0.948577] libphy: Fixed MDIO Bus: probed
[    0.948583] tun: Universal TUN/TAP device driver, 1.6
[    0.948588] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    0.948913] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.7.12-k
[    0.948921] ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
[    0.950031] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.950039] ehci_hcd: block sizes: qh 112 qtd 96 itd 192 sitd 96
[    0.950061] ehci-pci: EHCI PCI platform driver
[    0.950201] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.950207] ohci_hcd: block sizes: ed 80 td 96
[    0.950357] uhci_hcd: USB Universal Host Controller Interface driver
[    0.950736] usbcore: registered new interface driver usblp
[    0.951285] i8042: PNP: No PS/2 controller found. Probing ports directly.
[    1.960646] i8042: No controller found
[    1.960872] mousedev: PS/2 mouse device common for all mice
[    2.021775] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    2.021941] rtc_cmos: probe of rtc_cmos failed with error -38
[    2.023233] zram: Created 1 device(s) ...
[    2.023567] Netfilter messages via NETLINK v0.30.
[    2.023612] nf_conntrack version 0.5.0 (7930 buckets, 31720 max)
[    2.023775] ctnetlink v0.93: registering with nfnetlink.
[    2.024045] ip_tables: (C) 2000-2006 Netfilter Core Team
[    2.024170] TCP: cubic registered
[    2.024177] Initializing XFRM netlink socket
[    2.024379] NET: Registered protocol family 10
[    2.024936] ip6_tables: (C) 2000-2006 Netfilter Core Team
[    2.025259] sit: IPv6 over IPv4 tunneling driver
[    2.026039] NET: Registered protocol family 17
[    2.026320] Key type dns_resolver registered
[    2.026874] PM: Hibernation image not present or could not be loaded.
[    2.026922] registered taskstats version 1
[    2.027005] kmemleak: Kernel memory leak detector initialized
[    2.027009] kmemleak: Automatic memory scanning thread started
[    2.027437] XENBUS: Device with no driver: device/vbd/51712
[    2.027537]   Magic number: 1:252:3141
[    2.029169] Freeing unused kernel memory: 1724k freed
[    2.029408] Write protecting the kernel read-only data: 10240k
[    2.033649] Freeing unused kernel memory: 1480k freed
[    2.033908] Freeing unused kernel memory: 72k freed

init started: BusyBox v1.14.3 (2013-06-07 10:58:39 EDT)
[    2.040621] consoletype (1038) used greatest stack depth: 5304 bytes left
Mounting directories  [  OK  ]
[    2.059689] chmod (1044) used greatest stack depth: 4856 bytes left
mount: mount point /proc/bus/usb does not exist
mount: mount point /sys/kernel/config does not exist
[    2.266374] Initialising Xen virtual ethernet driver.
[    2.273190] vbd vbd-51712: blkfront:blkback_changed to state 2.
[    2.283349] vbd vbd-51712: blkfront:blkback_changed to state 4.
[    2.283356] vbd vbd-51712: blkfront_connect:/local/domain/0/backend/vbd/2/51712.
[    2.287707] blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled;
[    2.288337] Entered do_blkif_request
[    2.288344] Entered do_blkif_request
[    2.288821] Entered do_blkif_request
[    2.288829] do_blk_req ffff880108ec9e10: cmd ffff880108ec9f00, sec 0, (4/4) buffer:ffff880108ee6000 [read]
[    2.288839] Entered do_blkif_request
[    2.288859] Entered do_blkif_request
[    2.288864] do_blk_req ffff880108ec9cb8: cmd ffff880108ec9da8, sec 4, (4/4) buffer:ffff880108ee6800 [read]
[    2.288871] Entered do_blkif_request
[    2.292721] Entered do_blkif_request
[    2.292727] Entered do_blkif_request
[    2.292737] Entered do_blkif_request
[    2.292741] Entered do_blkif_request
[    2.292780] Entered do_blkif_request
[    2.292786] do_blk_req ffff880108ec9cb8: cmd ffff880108ec9da8, sec 8, (4/4) buffer:ffff88010f273000 [read]
[    2.292794] Entered do_blkif_request
[    2.292811] Entered do_blkif_request
[    2.292816] do_blk_req ffff880108ec9e10: cmd ffff880108ec9f00, sec c, (4/4) buffer:ffff88010f273800 [read]
[    2.292822] Entered do_blkif_request
[    2.292940] Entered do_blkif_request
[    2.292947] Entered do_blkif_request
[    2.292957] Entered do_blkif_request
[    2.292961] Entered do_blkif_request
[    2.292974]  xvda: unknown partition table
[    2.385804] udevd (1099): /proc/1099/oom_adj is deprecated, please use /proc/1099/oom_score_adj instead.
[    2.444038] e1000e: Intel(R) PRO/1000 Network Driver - 2.3.2-k
[    2.444061] e1000e: Copyright(c) 1999 - 2013 Intel Corporation.
[    2.444110] e1000e 0000:00:00.0: Disabling ASPM  L1
[    2.444232] e1000e 0000:00:00.0: enabling device (0000 -> 0002)
[    2.447244] e1000e 0000:00:00.0: Xen PCI mapped GSI16 to IRQ34
[    2.447790] e1000e 0000:00:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
udevd-work[1106]: error opening ATTR{/sys/devices/system/cpu/cpu0/online} for writing: No such file or directory

[    2.459644] vbd vbd-51712: command: 0x5331, argument: 0x0
[    2.560213] ip (1325) used greatest stack depth: 3760 bytes left
[    2.624581] e1000e 0000:00:00.0 eth0: (PCI Express:2.5GT/s:Width x4) 00:15:17:8f:18:a2
[    2.624595] e1000e 0000:00:00.0 eth0: Intel(R) PRO/1000 Network Connection
[    2.624675] e1000e 0000:00:00.0 eth0: MAC: 0, PHY: 4, PBA No: D50868-003
Waiting for devices [  OK  ]
Waiting for fb [  OK  ]
Starting..[/dev/fb0]
Could not open; err: 2
FATAL: Module agpgart_intel not found.
[    2.814571] [drm] radeon kernel modesetting enabled.
WARNING: Error inserting video (/lib/modules/3.10.0-rc4upstream-00172-g909fe9e-dirty/kernel/drivers/acpi/video.ko): No such device
WARNING: Error inserting mxm_wmi (/lib/modules/3.10.0-rc4upstream-00172-g909fe9e-dirty/kernel/drivers/platform/x86/mxm-wmi.ko): No such device
WARNING: Error inserting drm_kms_helper (/lib/modules/3.10.0-rc4upstream-00172-g909fe9e-dirty/kernel/drivers/gpu/drm/drm_kms_helper.ko): No such device
WARNING: Error inserting ttm (/lib/modules/3.10.0-rc4upstream-00172-g909fe9e-dirty/kernel/drivers/gpu/drm/ttm/ttm.ko): No such device
FATAL: Error inserting nouveau (/lib/modules/3.10.0-rc4upstream-00172-g909fe9e-dirty/kernel/drivers/gpu/drm/nouveau/nouveau.ko): No such device
WARNING: Error inserting drm_kms_helper (/lib/modules/3.10.0-rc4upstream-00172-g909fe9e-dirty/kernel/drivers/gpu/drm/drm_kms_helper.ko): No such device
FATAL: Error inserting i915 (/lib/modules/3.10.0-rc4upstream-00172-g909fe9e-dirty/kernel/drivers/gpu/drm/i915/i915.ko): No such device
Starting..[/dev/fb0]
Could not open; err: 2
VGA: 0000:
Waiting for network [  OK  ]
Bringing up loopback interface:  [  OK  ]
Bringing up interface eth0:  [    3.394987] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    3.397647] device eth0 entered promiscuous mode
[  OK  ]
Bringing up interface switch:  
Determining IP information for switch...[    3.452997] IPv6: ADDRCONF(NETDEV_UP): switch: link is not ready
[    5.533965] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[    5.534117] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    5.534222] switch: port 1(eth0) entered forwarding state
[    5.534239] switch: port 1(eth0) entered forwarding state
[    5.534302] IPv6: ADDRCONF(NETDEV_CHANGE): switch: link becomes ready
 done.
[  OK  ]
Waiting for init.custom [  OK  ]

Starting SSHd ...

    SSH started [2202]


Waiting for SSHd [  OK  ]
WARNING: ssh currently running [2202] ignoring start request
FATAL: Module dump_dma not found.
ERROR: Module dump_dma does not exist in /proc/modules
[    9.129856] SCSI subsystem initialized
[    9.131318] Loading iSCSI transport class v2.0-870.
[    9.134894] iscsi: registered transport (tcp)
hostname: Name or service not known
iscsistart: transport class version 2.0-870. iscsid version 2.0-872
Could not get list of targets from firmware.
Jun  7 15:21:14 (none) syslogd 1.5.0: restart.
Running in PV context on Xen v4.3.
FATAL: Module evtchn not found.
[    9.176036] Event-channel device installed.
00:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
           CPU0       CPU1       
 16:       2600          0  xen-percpu-virq      timer0
 17:          0          0  xen-percpu-ipi       spinlock0
 18:       3473          0  xen-percpu-ipi       resched0
 19:          0          0  xen-percpu-ipi       callfunc0
 20:          0          0  xen-percpu-virq      debug0
 21:        237          0  xen-percpu-ipi       callfuncsingle0
 22:          0          0  xen-percpu-ipi       irqwork0
 23:          0       2560  xen-percpu-virq      timer1
 24:          0          0  xen-percpu-ipi       spinlock1
 25:          0       3833  xen-percpu-ipi       resched1
 26:          0          0  xen-percpu-ipi       callfunc1
 27:          0          0  xen-percpu-virq      debug1
 28:          0        271  xen-percpu-ipi       callfuncsingle1
 29:          0          0  xen-percpu-ipi       irqwork1
 30:        235          0   xen-dyn-event     xenbus
 31:         64          0   xen-dyn-event     pcifront
 32:         89          0   xen-dyn-event     hvc_console
 33:          4          0   xen-dyn-event     blkif
 35:         27          0  xen-pirq-pcifront-msi  eth0
NMI:          0          0   Non-maskable interrupts
LOC:          0          0   Local timer interrupts
SPU:          0          0   Spurious interrupts
PMI:          0          0   Performance monitoring interrupts
IWI:          0          0   IRQ work interrupts
RTR:          0          0   APIC ICR read retries
RES:       3473       3833   Rescheduling interrupts
CAL:        237        271   Function call interrupts
TLB:          0          0   TLB shootdowns
TRM:          0          0   Thermal event interrupts
THR:          0          0   Threshold APIC interrupts
MCE:          0          0   Machine check exceptions
MCP:          0          0   Machine check polls
ERR:          0
MIS:          0
00000000-00000fff : reserved
00001000-0009ffff : System RAM
000a0000-000fffff : reserved
  000f0000-000fffff : System ROM
00100000-1fffffff : System RAM
  01000000-0168a927 : Kernel code
  0168a928-01aad2bf : Kernel data
  01c65000-01d74fff : Kernel bss
20200000-3fffffff : Unusable memory
40200000-c6cd3fff : Unusable memory
c6cd4000-c6d1cfff : ACPI Non-volatile Storage
c6d1d000-c6d27fff : ACPI Tables
c6d28000-c6d28fff : ACPI Non-volatile Storage
c6d4a000-c6d4bfff : Unusable memory
c6d4c000-c6d6cfff : ACPI Non-volatile Storage
c6d90000-c6d9cfff : ACPI Non-volatile Storage
c6da0000-c6db0fff : ACPI Non-volatile Storage
c6ddd000-c6e1ffff : ACPI Non-volatile Storage
c6e20000-c6ffffff : Unusable memory
fe480000-fe49ffff : 0000:00:00.0
  fe480000-fe49ffff : e1000e
fe4a0000-fe4bffff : 0000:00:00.0
  fe4a0000-fe4bffff : e1000e
100000000-11fffffff : System RAM
MemTotal:        1018432 kB
MemFree:          641536 kB
Buffers:               0 kB
Cached:           310180 kB
SwapCached:            0 kB
Active:            19368 kB
Inactive:         290652 kB
Active(anon):      13120 kB
Inactive(anon):    84540 kB
Active(file):       6248 kB
Inactive(file):   206112 kB
Unevictable:        4940 kB
Mlocked:            4940 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:          4684 kB
Mapped:             5152 kB
Shmem:             94284 kB
Slab:              45924 kB
SReclaimable:      12804 kB
SUnreclaim:        33120 kB
KernelStack:         432 kB
PageTables:          692 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      509216 kB
Committed_AS:     102792 kB
VmallocTotal:   34359738367 kB
VmallocUsed:        2332 kB
VmallocChunk:   34359735931 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:     1048576 kB
DirectMap2M:           0 kB
Waiting for init.late [  OK  ]
PING build.dumpdata.com (192.168.102.1) 56(84) bytes of data.

--- build.dumpdata.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.208/0.208/0.208/0.000 ms
[    9.336320] mount.nfs (2304) used greatest stack depth: 3320 bytes left
mount.nfs: access denied by server while mounting build:/srv/results
NFS done
/init.late: line 13: can't create /mnt/results/dmesg/(none)-3.10.0-rc4upstream-00172-g909fe9e-dirty.dmesg: nonexistent directory
 [0x0->0x20000] pfn
 [0x0->0x20000] level entry
 [0x20000->0x100000] identity
 [0x20000->0x100000] level middle
 [0x100000->0x120000] pfn
 [0x100000->0x120000] level entry
 [0x120000->0x140000] level middle
 [0x120000->0x7cfffff] missing
 [0x140000->0x7cfffff] level top
/init.late: line 17: can't create /mnt/results/p2m/(none)-3.10.0-rc4upstream-00172-g909fe9e-dirty.p2m: nonexistent directory
libxl: error: libxl.c:87:libxl_ctx_alloc: Is xenstore daemon running?
failed to stat /var/run/xenstored.pid: No such file or directory
cannot init xl context
[    9.669091] device-mapper: ioctl: 4.24.0-ioctl (2013-01-15) initialised: dm-devel@redhat.com
[    9.670252] device-mapper: multipath: version 1.5.1 loaded
PING 192.168.101.2 (192.168.101.2) 56(84) bytes of data.
kill -1 1

--- 192.168.101.2 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 10000ms

/init.late: line 34: boot_parameter: not found
[   20.576092] switch: port 1(eth0) entered forwarding state
 7 Jun 15:21:27 ntpdate[2323]: adjust time server 17.171.4.15 offset -0.265117 sec
Cannot access the Hardware Clock via any known method.
Use the --debug option to see the details of our search for an access method.
Fri Jun  7 15:21:27 UTC 2013
Jun  7 15:21:27 (none) init: starting pid 2333, tty '/dev/tty0': '/bin/sh'
Jun  7 15:21:27 (none) init: starting pid 2334, tty '/dev/tty1': '/bin/sh'
Jun  7 15:21:27 (none) init: starting pid 2335, tty '/dev/ttyS0': '/bin/sh'
Jun  7 15:21:27 (none) init: starting pid 2336, tty '/dev/hvc0': '/bin/sh'


BusyBox v1.14.3 (2013-06-07 10:58:39 EDT) built-in shell (ash)
Enter 'help' for a list of built-in commands.

# kill -1 1
# Jun  7 15:21:27 (none) init: reloading /etc/inittab

# m\b \bdmesg
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.10.0-rc4upstream-00172-g909fe9e-dirty (konrad@phenom.dumpdata.com) (gcc version 4.4.4 20100503 (Red Hat 4.4.4-2) (GCC) ) #2 SMP Fri Jun 7 10:55:54 EDT 2013
[    0.000000] Command line: console=hvc0 debug 
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] Freeing 20000-40000 pfn range: 131072 pages freed
[    0.000000] 1-1 mapping on 20000->100000
[    0.000000] Released 131072 pages of unused memory
[    0.000000] Set 917504 page(s) to 1-1 mapping
[    0.000000] Populating 100000-120000 pfn range: 131072 pages added
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
[    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[    0.000000] Xen: [mem 0x0000000000100000-0x000000001fffffff] usable
[    0.000000] Xen: [mem 0x0000000020000000-0x00000000201fffff] reserved
[    0.000000] Xen: [mem 0x0000000020200000-0x000000003fffffff] unusable
[    0.000000] Xen: [mem 0x0000000040000000-0x00000000401fffff] reserved
[    0.000000] Xen: [mem 0x0000000040200000-0x00000000c6cd3fff] unusable
[    0.000000] Xen: [mem 0x00000000c6cd4000-0x00000000c6d1cfff] ACPI NVS
[    0.000000] Xen: [mem 0x00000000c6d1d000-0x00000000c6d27fff] ACPI data
[    0.000000] Xen: [mem 0x00000000c6d28000-0x00000000c6d28fff] ACPI NVS
[    0.000000] Xen: [mem 0x00000000c6d29000-0x00000000c6d49fff] reserved
[    0.000000] Xen: [mem 0x00000000c6d4a000-0x00000000c6d4bfff] unusable
[    0.000000] Xen: [mem 0x00000000c6d4c000-0x00000000c6d6cfff] ACPI NVS
[    0.000000] Xen: [mem 0x00000000c6d6d000-0x00000000c6d8ffff] reserved
[    0.000000] Xen: [mem 0x00000000c6d90000-0x00000000c6d9cfff] ACPI NVS
[    0.000000] Xen: [mem 0x00000000c6d9d000-0x00000000c6d9ffff] reserved
[    0.000000] Xen: [mem 0x00000000c6da0000-0x00000000c6db0fff] ACPI NVS
[    0.000000] Xen: [mem 0x00000000c6db1000-0x00000000c6ddcfff] reserved
[    0.000000] Xen: [mem 0x00000000c6ddd000-0x00000000c6e1ffff] ACPI NVS
[    0.000000] Xen: [mem 0x00000000c6e20000-0x00000000c6ffffff] unusable
[    0.000000] Xen: [mem 0x00000000c7800000-0x00000000cf9fffff] reserved
[    0.000000] Xen: [mem 0x00000000fed1c000-0x00000000fed3ffff] reserved
[    0.000000] Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
[    0.000000] Xen: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[    0.000000] Xen: [mem 0x0000000100000000-0x000000011fffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI not present or invalid.
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] No AGP bridge found
[    0.000000] e820: last_pfn = 0x120000 max_arch_pfn = 0x400000000
[    0.000000] e820: last_pfn = 0x20000 max_arch_pfn = 0x400000000
[    0.000000] Scanning 1 areas for low memory corruption
[    0.000000] Base memory trampoline at [ffff88000009a000] 9a000 size 24576
[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[    0.000000]  [mem 0x00000000-0x000fffff] page 4k
[    0.000000] init_memory_mapping: [mem 0x11fe00000-0x11fffffff]
[    0.000000]  [mem 0x11fe00000-0x11fffffff] page 4k
[    0.000000] BRK [0x01d8a000, 0x01d8afff] PGTABLE
[    0.000000] BRK [0x01d8b000, 0x01d8bfff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0x11c000000-0x11fdfffff]
[    0.000000]  [mem 0x11c000000-0x11fdfffff] page 4k
[    0.000000] BRK [0x01d8c000, 0x01d8cfff] PGTABLE
[    0.000000] BRK [0x01d8d000, 0x01d8dfff] PGTABLE
[    0.000000] BRK [0x01d8e000, 0x01d8efff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0x100000000-0x11bffffff]
[    0.000000]  [mem 0x100000000-0x11bffffff] page 4k
[    0.000000] init_memory_mapping: [mem 0x00100000-0x1fffffff]
[    0.000000]  [mem 0x00100000-0x1fffffff] page 4k
[    0.000000] RAMDISK: [mem 0x02197000-0x13f67fff]
[    0.000000] NUMA turned off
[    0.000000] Faking a node at [mem 0x0000000000000000-0x000000011fffffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000-0x11fffffff]
[    0.000000]   NODE_DATA [mem 0x11fea2000-0x11fea5fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
[    0.000000]   Normal   [mem 0x100000000-0x11fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00001000-0x0009ffff]
[    0.000000]   node   0: [mem 0x00100000-0x1fffffff]
[    0.000000]   node   0: [mem 0x100000000-0x11fffffff]
[    0.000000] On node 0 totalpages: 262047
[    0.000000]   DMA zone: 56 pages used for memmap
[    0.000000]   DMA zone: 21 pages reserved
[    0.000000]   DMA zone: 3999 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 1736 pages used for memmap
[    0.000000]   DMA32 zone: 126976 pages, LIFO batch:31
[    0.000000]   Normal zone: 1792 pages used for memmap
[    0.000000]   Normal zone: 131072 pages, LIFO batch:31
[    0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs
[    0.000000] No local APIC present
[    0.000000] APIC: disable apic facility
[    0.000000] APIC: switched to apic NOOP
[    0.000000] nr_irqs_gsi: 16
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
[    0.000000] PM: Registered nosave memory: 0000000020000000 - 0000000020200000
[    0.000000] PM: Registered nosave memory: 0000000020200000 - 0000000040000000
[    0.000000] PM: Registered nosave memory: 0000000040000000 - 0000000040200000
[    0.000000] PM: Registered nosave memory: 0000000040200000 - 00000000c6cd4000
[    0.000000] PM: Registered nosave memory: 00000000c6cd4000 - 00000000c6d1d000
[    0.000000] PM: Registered nosave memory: 00000000c6d1d000 - 00000000c6d28000
[    0.000000] PM: Registered nosave memory: 00000000c6d28000 - 00000000c6d29000
[    0.000000] PM: Registered nosave memory: 00000000c6d29000 - 00000000c6d4a000
[    0.000000] PM: Registered nosave memory: 00000000c6d4a000 - 00000000c6d4c000
[    0.000000] PM: Registered nosave memory: 00000000c6d4c000 - 00000000c6d6d000
[    0.000000] PM: Registered nosave memory: 00000000c6d6d000 - 00000000c6d90000
[    0.000000] PM: Registered nosave memory: 00000000c6d90000 - 00000000c6d9d000
[    0.000000] PM: Registered nosave memory: 00000000c6d9d000 - 00000000c6da0000
[    0.000000] PM: Registered nosave memory: 00000000c6da0000 - 00000000c6db1000
[    0.000000] PM: Registered nosave memory: 00000000c6db1000 - 00000000c6ddd000
[    0.000000] PM: Registered nosave memory: 00000000c6ddd000 - 00000000c6e20000
[    0.000000] PM: Registered nosave memory: 00000000c6e20000 - 00000000c7000000
[    0.000000] PM: Registered nosave memory: 00000000c7000000 - 00000000c7800000
[    0.000000] PM: Registered nosave memory: 00000000c7800000 - 00000000cfa00000
[    0.000000] PM: Registered nosave memory: 00000000cfa00000 - 00000000fed1c000
[    0.000000] PM: Registered nosave memory: 00000000fed1c000 - 00000000fed40000
[    0.000000] PM: Registered nosave memory: 00000000fed40000 - 00000000fee00000
[    0.000000] PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
[    0.000000] PM: Registered nosave memory: 00000000fee01000 - 00000000ff000000
[    0.000000] PM: Registered nosave memory: 00000000ff000000 - 0000000100000000
[    0.000000] e820: [mem 0xcfa00000-0xfed1bfff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 4.3-unstable (preserve-AD)
[    0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:2 nr_node_ids:1
[    0.000000] PERCPU: Embedded 28 pages/cpu @ffff88011f800000 s85632 r8192 d20864 u1048576
[    0.000000] pcpu-alloc: s85632 r8192 d20864 u1048576 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 0 1 
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 258442
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: console=hvc0 debug 
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340
[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Memory: 722480k/4718592k available (6698k kernel code, 3670404k absent, 325708k reserved, 4234k data, 1724k init)
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=2.
[    0.000000] NR_IRQS:33024 nr_irqs:288 16
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [hvc0] enabled
[    0.000000] Xen: using vcpuop timer interface
[    0.000000] installing Xen timer for CPU 0
[    0.000000] tsc: Detected 3092.926 MHz processor
[    0.001000] Calibrating delay loop (skipped), value calculated using timer frequency.. 6185.85 BogoMIPS (lpj=3092926)
[    0.001000] pid_max: default: 32768 minimum: 301
[    0.001000] Security Framework initialized
[    0.001000] SELinux:  Initializing.
[    0.001000] SELinux:  Starting in permissive mode
[    0.001000] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.001000] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.001000] Mount-cache hash table entries: 256
[    0.001582] Initializing cgroup subsys freezer
[    0.001698] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[    0.001698] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
[    0.001708] CPU: Physical Processor ID: 0
[    0.001712] CPU: Processor Core ID: 0
[    0.001718] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0
[    0.001718] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32
[    0.001718] tlb_flushall_shift: 5
[    0.025468] cpu 0 spinlock event irq 17
[    0.025566] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only.
[    0.026081] NMI watchdog: disabled (cpu0): hardware events not enabled
[    0.026680] installing Xen timer for CPU 1
[    0.026723] cpu 1 spinlock event irq 24
[    0.026793] SMP alternatives: switching to SMP code
[    0.048014] Brought up 2 CPUs
[    0.051334] PM: Registering ACPI NVS region [mem 0xc6cd4000-0xc6d1cfff] (299008 bytes)
[    0.051334] PM: Registering ACPI NVS region [mem 0xc6d28000-0xc6d28fff] (4096 bytes)
[    0.051334] PM: Registering ACPI NVS region [mem 0xc6d4c000-0xc6d6cfff] (135168 bytes)
[    0.051334] PM: Registering ACPI NVS region [mem 0xc6d90000-0xc6d9cfff] (53248 bytes)
[    0.051334] PM: Registering ACPI NVS region [mem 0xc6da0000-0xc6db0fff] (69632 bytes)
[    0.051334] PM: Registering ACPI NVS region [mem 0xc6ddd000-0xc6e1ffff] (274432 bytes)
[    0.051399] kworker/u4:0 (19) used greatest stack depth: 6016 bytes left
[    0.052492] Grant tables using version 2 layout.
[    0.052492] Grant table initialized
[    0.071696] RTC time: 165:165:165, date: 165/165/65
[    0.071985] NET: Registered protocol family 16
[    0.072173] kworker/u4:0 (22) used greatest stack depth: 5928 bytes left
[    0.073110] kworker/u4:0 (30) used greatest stack depth: 5464 bytes left
[    0.074030] dca service started, version 1.12.1
[    0.074563] PCI: setting up Xen PCI frontend stub
[    0.074568] PCI: pci_cache_line_size set to 64 bytes
[    0.101148] bio: create slab <bio-0> at 0
[    0.101220] ACPI: Interpreter disabled.
[    0.102003] xen/balloon: Initialising balloon driver.
[    0.103081] xen-balloon: Initialising balloon driver.
[    0.104136] vgaarb: loaded
[    0.105135] usbcore: registered new interface driver usbfs
[    0.105135] usbcore: registered new interface driver hub
[    0.105135] usbcore: registered new device driver usb
[    0.106089] pps_core: LinuxPPS API ver. 1 registered
[    0.106089] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.106089] PTP clock support registered
[    0.106089] PCI: System does not support PCI
[    0.106089] PCI: System does not support PCI
[    0.107154] NetLabel: Initializing
[    0.107154] NetLabel:  domain hash size = 128
[    0.107154] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.107175] NetLabel:  unlabeled traffic allowed by default
[    0.108087] Switching to clocksource xen
[    0.125349] pnp: PnP ACPI: disabled
[    0.138965] NET: Registered protocol family 2
[    0.139578] TCP established hash table entries: 8192 (order: 5, 131072 bytes)
[    0.139647] TCP bind hash table entries: 8192 (order: 5, 131072 bytes)
[    0.139668] TCP: Hash tables configured (established 8192 bind 8192)
[    0.193672] TCP: reno registered
[    0.193693] UDP hash table entries: 512 (order: 2, 16384 bytes)
[    0.193712] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[    0.194030] NET: Registered protocol family 1
[    0.194380] RPC: Registered named UNIX socket transport module.
[    0.194389] RPC: Registered udp transport module.
[    0.194394] RPC: Registered tcp transport module.
[    0.194398] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.194404] PCI: CLS 0 bytes, default 64
[    0.194570] Unpacking initramfs...
[    0.778545] Freeing initrd memory: 292676k freed
[    0.823501] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.823855] Machine check injector initialized
[    0.825040] Scanning for low memory corruption every 60 seconds
[    0.826072] audit: initializing netlink socket (disabled)
[    0.826114] type=2000 audit(1370618465.698:1): initialized
[    0.839204] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.839673] VFS: Disk quotas dquot_6.5.2
[    0.839760] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.840528] NFS: Registering the id_resolver key type
[    0.840553] Key type id_resolver registered
[    0.840558] Key type id_legacy registered
[    0.840587] NTFS driver 2.1.30 [Flags: R/W].
[    0.840919] msgmni has been set to 1982
[    0.841082] SELinux:  Registering netfilter hooks
[    0.842697] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[    0.842706] io scheduler noop registered
[    0.842710] io scheduler deadline registered
[    0.842772] io scheduler cfq registered (default)
[    0.843309] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.846284] intel_idle: does not run on family 6 model 42
[    0.846299] ioatdma: Intel(R) QuickData Technology Driver 4.00
[    0.847400] pcifront pci-0: Installing PCI frontend
[    0.847417] Warning: only able to allocate 4 MB for software IO TLB
[    0.850807] software IO TLB [mem 0x109000000-0x109400000] (4MB) mapped at [ffff880109000000-ffff8801093fffff]
[    0.851175] pcifront pci-0: Creating PCI Frontend Bus 0000:00
[    0.851458] pcifront pci-0: PCI host bridge to bus 0000:00
[    0.851469] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    0.851478] pci_bus 0000:00: root bus resource [mem 0x00000000-0xfffffffff]
[    0.851487] pci_bus 0000:00: root bus resource [bus 00]
[    0.852031] pci 0000:00:00.0: [8086:105e] type 00 class 0x020000
[    0.852419] pci 0000:00:00.0: reg 0x10: [mem 0xfe4a0000-0xfe4bffff]
[    0.852631] pci 0000:00:00.0: reg 0x14: [mem 0xfe480000-0xfe49ffff]
[    0.852743] pci 0000:00:00.0: reg 0x18: [io  0xe020-0xe03f]
[    0.853905] pcifront pci-0: New device on 0000:00:00.0 found.
[    0.857483] pcifront pci-0: claiming resource 0000:00:00.0/0
[    0.857490] pcifront pci-0: claiming resource 0000:00:00.0/1
[    0.857495] pcifront pci-0: claiming resource 0000:00:00.0/2
[    0.939621] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.942227] Non-volatile memory driver v1.3
[    0.942414] Linux agpgart interface v0.103
[    0.943295] [drm] Initialized drm 1.1.0 20060810
[    0.947952] loop: module loaded
[    0.948577] libphy: Fixed MDIO Bus: probed
[    0.948583] tun: Universal TUN/TAP device driver, 1.6
[    0.948588] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    0.948913] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.7.12-k
[    0.948921] ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
[    0.950031] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.950039] ehci_hcd: block sizes: qh 112 qtd 96 itd 192 sitd 96
[    0.950061] ehci-pci: EHCI PCI platform driver
[    0.950201] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.950207] ohci_hcd: block sizes: ed 80 td 96
[    0.950357] uhci_hcd: USB Universal Host Controller Interface driver
[    0.950736] usbcore: registered new interface driver usblp
[    0.951285] i8042: PNP: No PS/2 controller found. Probing ports directly.
[    1.960646] i8042: No controller found
[    1.960872] mousedev: PS/2 mouse device common for all mice
[    2.021775] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    2.021941] rtc_cmos: probe of rtc_cmos failed with error -38
[    2.023233] zram: Created 1 device(s) ...
[    2.023567] Netfilter messages via NETLINK v0.30.
[    2.023612] nf_conntrack version 0.5.0 (7930 buckets, 31720 max)
[    2.023775] ctnetlink v0.93: registering with nfnetlink.
[    2.024045] ip_tables: (C) 2000-2006 Netfilter Core Team
[    2.024170] TCP: cubic registered
[    2.024177] Initializing XFRM netlink socket
[    2.024379] NET: Registered protocol family 10
[    2.024936] ip6_tables: (C) 2000-2006 Netfilter Core Team
[    2.025259] sit: IPv6 over IPv4 tunneling driver
[    2.026039] NET: Registered protocol family 17
[    2.026320] Key type dns_resolver registered
[    2.026874] PM: Hibernation image not present or could not be loaded.
[    2.026922] registered taskstats version 1
[    2.027005] kmemleak: Kernel memory leak detector initialized
[    2.027009] kmemleak: Automatic memory scanning thread started
[    2.027437] XENBUS: Device with no driver: device/vbd/51712
[    2.027537]   Magic number: 1:252:3141
[    2.029169] Freeing unused kernel memory: 1724k freed
[    2.029408] Write protecting the kernel read-only data: 10240k
[    2.033649] Freeing unused kernel memory: 1480k freed
[    2.033908] Freeing unused kernel memory: 72k freed
[    2.040621] consoletype (1038) used greatest stack depth: 5304 bytes left
[    2.059689] chmod (1044) used greatest stack depth: 4856 bytes left
[    2.266374] Initialising Xen virtual ethernet driver.
[    2.273190] vbd vbd-51712: blkfront:blkback_changed to state 2.
[    2.283349] vbd vbd-51712: blkfront:blkback_changed to state 4.
[    2.283356] vbd vbd-51712: blkfront_connect:/local/domain/0/backend/vbd/2/51712.
[    2.287707] blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled;
[    2.288337] Entered do_blkif_request
[    2.288344] Entered do_blkif_request
[    2.288821] Entered do_blkif_request
[    2.288829] do_blk_req ffff880108ec9e10: cmd ffff880108ec9f00, sec 0, (4/4) buffer:ffff880108ee6000 [read]
[    2.288839] Entered do_blkif_request
[    2.288859] Entered do_blkif_request
[    2.288864] do_blk_req ffff880108ec9cb8: cmd ffff880108ec9da8, sec 4, (4/4) buffer:ffff880108ee6800 [read]
[    2.288871] Entered do_blkif_request
[    2.292721] Entered do_blkif_request
[    2.292727] Entered do_blkif_request
[    2.292737] Entered do_blkif_request
[    2.292741] Entered do_blkif_request
[    2.292780] Entered do_blkif_request
[    2.292786] do_blk_req ffff880108ec9cb8: cmd ffff880108ec9da8, sec 8, (4/4) buffer:ffff88010f273000 [read]
[    2.292794] Entered do_blkif_request
[    2.292811] Entered do_blkif_request
[    2.292816] do_blk_req ffff880108ec9e10: cmd ffff880108ec9f00, sec c, (4/4) buffer:ffff88010f273800 [read]
[    2.292822] Entered do_blkif_request
[    2.292940] Entered do_blkif_request
[    2.292947] Entered do_blkif_request
[    2.292957] Entered do_blkif_request
[    2.292961] Entered do_blkif_request
[    2.292974]  xvda: unknown partition table
[    2.385804] udevd (1099): /proc/1099/oom_adj is deprecated, please use /proc/1099/oom_score_adj instead.
[    2.444038] e1000e: Intel(R) PRO/1000 Network Driver - 2.3.2-k
[    2.444061] e1000e: Copyright(c) 1999 - 2013 Intel Corporation.
[    2.444110] e1000e 0000:00:00.0: Disabling ASPM  L1
[    2.444232] e1000e 0000:00:00.0: enabling device (0000 -> 0002)
[    2.447244] e1000e 0000:00:00.0: Xen PCI mapped GSI16 to IRQ34
[    2.447790] e1000e 0000:00:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[    2.459644] vbd vbd-51712: command: 0x5331, argument: 0x0
[    2.560213] ip (1325) used greatest stack depth: 3760 bytes left
[    2.624581] e1000e 0000:00:00.0 eth0: (PCI Express:2.5GT/s:Width x4) 00:15:17:8f:18:a2
[    2.624595] e1000e 0000:00:00.0 eth0: Intel(R) PRO/1000 Network Connection
[    2.624675] e1000e 0000:00:00.0 eth0: MAC: 0, PHY: 4, PBA No: D50868-003
[    2.814571] [drm] radeon kernel modesetting enabled.
[    3.394987] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    3.397647] device eth0 entered promiscuous mode
[    3.452997] IPv6: ADDRCONF(NETDEV_UP): switch: link is not ready
[    5.533965] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[    5.534117] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    5.534222] switch: port 1(eth0) entered forwarding state
[    5.534239] switch: port 1(eth0) entered forwarding state
[    5.534302] IPv6: ADDRCONF(NETDEV_CHANGE): switch: link becomes ready
[    9.129856] SCSI subsystem initialized
[    9.131318] Loading iSCSI transport class v2.0-870.
[    9.134894] iscsi: registered transport (tcp)
[    9.176036] Event-channel device installed.
[    9.336320] mount.nfs (2304) used greatest stack depth: 3320 bytes left
[    9.669091] device-mapper: ioctl: 4.24.0-ioctl (2013-01-15) initialised: dm-devel@redhat.com
[    9.670252] device-mapper: multipath: version 1.5.1 loaded
[   20.576092] switch: port 1(eth0) entered forwarding state
# lspci
00:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:15:17:8F:18:A2  
          inet6 addr: fe80::215:17ff:fe8f:18a2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:116 errors:0 dropped:0 overruns:0 frame:0
          TX packets:141 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:12017 (11.7 KiB)  TX bytes:13404 (13.0 KiB)
          Interrupt:34 Memory:fe4a0000-fe4c0000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:520 (520.0 b)  TX bytes:520 (520.0 b)

switch    Link encap:Ethernet  HWaddr 00:15:17:8F:18:A2  
          inet addr:192.168.102.210  Bcast:192.168.102.255  Mask:255.255.255.0
          inet6 addr: fe80::215:17ff:fe8f:18a2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:116 errors:0 dropped:0 overruns:0 frame:0
          TX packets:134 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:9929 (9.6 KiB)  TX bytes:12244 (11.9 KiB)

# eth\b\b\b
# ethtool ^[[J-i eth0
driver: e1000e
version: 2.3.2-k
firmware-version: 5.11-2
bus-info: 0000:00:00.0
# rmmod e1000e
[   39.032406] switch: port 1(eth0) entered disabled state
[   39.032576] device eth0 left promiscuous mode
[   39.032583] switch: port 1(eth0) entered disabled state
# miod\b \b\b \b\b \bodprobe e1000e
[   42.614015] e1000e: Intel(R) PRO/1000 Network Driver - 2.3.2-k
[   42.614026] e1000e: Copyright(c) 1999 - 2013 Intel Corporation.
[   42.614080] e1000e 0000:00:00.0: Disabling ASPM  L1
[   42.614172] xen_map_pirq_gsi: returning irq 34 for gsi 16
[   42.614181] e1000e 0000:00:00.0: Xen PCI mapped GSI16 to IRQ34
[   42.615024] e1000e 0000:00:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[   42.790351] e1000e 0000:00:00.0 eth0: (PCI Express:2.5GT/s:Width x4) 00:15:17:8f:18:a2
[   42.790363] e1000e 0000:00:00.0 eth0: Intel(R) PRO/1000 Network Connection
[   42.790448] e1000e 0000:00:00.0 eth0: MAC: 0, PHY: 4, PBA No: D50868-003
# [   43.103936] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   43.106574] device eth0 entered promiscuous mode
rmmod e100[   45.190968] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[   45.191117] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   45.191210] switch: port 1(eth0) entered forwarding state
[   45.191227] switch: port 1(eth0) entered forwarding state
0e
[   46.621376] switch: port 1(eth0) entered disabled state
[   46.621538] device eth0 left promiscuous mode
[   46.621547] switch: port 1(eth0) entered disabled state
# 



DETACHING HERE..

# 
# [   53.802739] pcifront pci-0: Rescanning PCI Frontend Bus 0000:00
[   53.887894] pci_bus 0000:00: busn_res: [bus 00] is released
[   53.888042] ------------[ cut here ]------------
[   53.888055] WARNING: at /home/konrad/linux/include/linux/kref.h:47 klist_iter_init_node+0x3e/0x50()
[   53.888073] Modules linked in: dm_multipath dm_mod xen_evtchn iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi scsi_mod libcrc32c crc32c radeon fbcon tileblit font bitblit softcursor ttm drm_kms_helper crc32c_intel xen_blkfront xen_netfront xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xenfs xen_privcmd [last unloaded: e1000e]
[   53.888151] CPU: 1 PID: 26 Comm: xenwatch Not tainted 3.10.0-rc4upstream-00172-g909fe9e-dirty #2
[   53.888158]  ffffffff819392b8 ffff88011e3c1b88 ffffffff8167a27d ffff88011e3c1bc8
[   53.888171]  ffffffff8108d1eb ffff880109536a00 ffff88011e3c1c18 0000000000000000
[   53.888186]  ffffffff8131c410 0000000000000000 ffff88011e3c1e70 ffff88011e3c1bd8
[   53.888197] Call Trace:
[   53.888203]  [<ffffffff8167a27d>] dump_stack+0x19/0x1b
[   53.888212]  [<ffffffff8108d1eb>] warn_slowpath_common+0x6b/0xa0
[   53.888220]  [<ffffffff8131c410>] ? pci_match_next_bus+0x10/0x10
[   53.888227]  [<ffffffff8108d235>] warn_slowpath_null+0x15/0x20
[   53.888234]  [<ffffffff81663b4e>] klist_iter_init_node+0x3e/0x50
[   53.888242]  [<ffffffff8142117d>] class_dev_iter_init+0x3d/0x50
[   53.888248]  [<ffffffff81421348>] class_find_device+0x38/0xb0
[   53.888255]  [<ffffffff8131c8aa>] pci_get_next_root_bus+0x4a/0x80
[   53.888262]  [<ffffffff8132f6a5>] pcifront_free_roots+0x25/0x60
[   53.888268]  [<ffffffff8132f6f1>] free_pdev+0x11/0x80
[   53.888274]  [<ffffffff8132f77a>] pcifront_xenbus_remove+0x1a/0x20
[   53.888283]  [<ffffffff813abae8>] xenbus_dev_remove+0x38/0x70
[   53.888289]  [<ffffffff8141ffc1>] __device_release_driver+0x61/0xd0
[   53.888296]  [<ffffffff81420148>] device_release_driver+0x28/0x40
[   53.888302]  [<ffffffff8141f296>] bus_remove_device+0x106/0x140
[   53.888310]  [<ffffffff8141d120>] device_del+0x110/0x1c0
[   53.888316]  [<ffffffff8141d1e1>] device_unregister+0x11/0x20
[   53.888323]  [<ffffffff813ab946>] xenbus_dev_changed+0x96/0x1d0
[   53.888331]  [<ffffffff81048bd6>] ? xen_spin_lock+0xa6/0x110
[   53.888338]  [<ffffffff813ad2f6>] frontend_changed+0x16/0x20
[   53.888345]  [<ffffffff813a9c5b>] xenwatch_thread+0xcb/0x190
[   53.888352]  [<ffffffff810b49d0>] ? wake_up_bit+0x40/0x40
[   53.888359]  [<ffffffff813a9b90>] ? xs_watch+0x60/0x60
[   53.888365]  [<ffffffff810b42e6>] kthread+0xc6/0xd0
[   53.888371]  [<ffffffff8103a149>] ? xen_end_context_switch+0x19/0x20
[   53.888378]  [<ffffffff810b4220>] ? kthread_freezable_should_stop+0x80/0x80
[   53.888386]  [<ffffffff8168623c>] ret_from_fork+0x7c/0xb0
[   53.888392]  [<ffffffff810b4220>] ? kthread_freezable_should_stop+0x80/0x80
[   53.888397] ---[ end trace 918a23c9bada3aed ]---
[   53.888402] ------------[ cut here ]------------
[   53.888407] WARNING: at /home/konrad/linux/lib/klist.c:189 klist_release+0x112/0x120()
[   53.888412] Modules linked in: dm_multipath dm_mod xen_evtchn iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi scsi_mod libcrc32c crc32c radeon fbcon tileblit font bitblit softcursor ttm drm_kms_helper crc32c_intel xen_blkfront xen_netfront xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xenfs xen_privcmd [last unloaded: e1000e]
[   53.888478] CPU: 1 PID: 26 Comm: xenwatch Tainted: G        W    3.10.0-rc4upstream-00172-g909fe9e-dirty #2
[   53.888484]  ffffffff819a3648 ffff88011e3c1b18 ffffffff8167a27d ffff88011e3c1b58
[   53.888495]  ffffffff8108d1eb 0000000000000010 ffff880109533aa8 ffff880109533a90
[   53.888506]  ffffffff81421460 0000000000000000 dead000000100100 ffff88011e3c1b68
[   53.888517] Call Trace:
[   53.888523]  [<ffffffff8167a27d>] dump_stack+0x19/0x1b
[   53.888529]  [<ffffffff8108d1eb>] warn_slowpath_common+0x6b/0xa0
[   53.888536]  [<ffffffff81421460>] ? class_for_each_device+0xa0/0xa0
[   53.888543]  [<ffffffff8108d235>] warn_slowpath_null+0x15/0x20
[   53.888549]  [<ffffffff81663c82>] klist_release+0x112/0x120
[   53.888556]  [<ffffffff81421460>] ? class_for_each_device+0xa0/0xa0
[   53.888563]  [<ffffffff81663cb8>] klist_dec_and_del+0x28/0x30
[   53.888601]  [<ffffffff81663eb5>] klist_next+0x45/0x140
[   53.888608]  [<ffffffff8131c410>] ? pci_match_next_bus+0x10/0x10
[   53.888615]  [<ffffffff81421108>] class_dev_iter_next+0x18/0x50
[   53.888621]  [<ffffffff81421358>] class_find_device+0x48/0xb0
[   53.888628]  [<ffffffff8131c8aa>] pci_get_next_root_bus+0x4a/0x80
[   53.888634]  [<ffffffff8132f6a5>] pcifront_free_roots+0x25/0x60
[   53.888640]  [<ffffffff8132f6f1>] free_pdev+0x11/0x80
[   53.888646]  [<ffffffff8132f77a>] pcifront_xenbus_remove+0x1a/0x20
[   53.888653]  [<ffffffff813abae8>] xenbus_dev_remove+0x38/0x70
[   53.888660]  [<ffffffff8141ffc1>] __device_release_driver+0x61/0xd0
[   53.888667]  [<ffffffff81420148>] device_release_driver+0x28/0x40
[   53.888673]  [<ffffffff8141f296>] bus_remove_device+0x106/0x140
[   53.888680]  [<ffffffff8141d120>] device_del+0x110/0x1c0
[   53.888688]  [<ffffffff8141d1e1>] device_unregister+0x11/0x20
[   53.888694]  [<ffffffff813ab946>] xenbus_dev_changed+0x96/0x1d0
[   53.888701]  [<ffffffff81048bd6>] ? xen_spin_lock+0xa6/0x110
[   53.888708]  [<ffffffff813ad2f6>] frontend_changed+0x16/0x20
[   53.888714]  [<ffffffff813a9c5b>] xenwatch_thread+0xcb/0x190
[   53.888721]  [<ffffffff810b49d0>] ? wake_up_bit+0x40/0x40
[   53.888727]  [<ffffffff813a9b90>] ? xs_watch+0x60/0x60
[   53.888733]  [<ffffffff810b42e6>] kthread+0xc6/0xd0
[   53.888739]  [<ffffffff8103a149>] ? xen_end_context_switch+0x19/0x20
[   53.888746]  [<ffffffff810b4220>] ? kthread_freezable_should_stop+0x80/0x80
[   53.888755]  [<ffffffff8168623c>] ret_from_fork+0x7c/0xb0
[   53.888761]  [<ffffffff810b4220>] ? kthread_freezable_should_stop+0x80/0x80
[   53.888766] ---[ end trace 918a23c9bada3aee ]---
[   53.888776] general protection fault: 0000 [#1] SMP 
[   53.888783] Modules linked in: dm_multipath dm_mod xen_evtchn iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi scsi_mod libcrc32c crc32c radeon fbcon tileblit font bitblit softcursor ttm drm_kms_helper crc32c_intel xen_blkfront xen_netfront xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xenfs xen_privcmd [last unloaded: e1000e]
[   53.888847] CPU: 1 PID: 26 Comm: xenwatch Tainted: G        W    3.10.0-rc4upstream-00172-g909fe9e-dirty #2
[   53.888853] task: ffff88011e2eb800 ti: ffff88011e3c0000 task.ti: ffff88011e3c0000
[   53.888859] RIP: e030:[<ffffffff81663baa>]  [<ffffffff81663baa>] klist_release+0x3a/0x120
[   53.888867] RSP: e02b:ffff88011e3c1b78  EFLAGS: 00010292
[   53.888872] RAX: dead000000200200 RBX: ffff880109533aa8 RCX: 0000006c3739ee67
[   53.888877] RDX: dead000000100100 RSI: dead000000200200 RDI: dead000000100100
[   53.888882] RBP: ffff88011e3c1b98 R08: 00000000a589f9bc R09: 0720072007200720
[   53.888887] R10: 0720072007200720 R11: 0720072007200720 R12: ffff880109533a90
[   53.888892] R13: ffffffff81421460 R14: 0000000000000000 R15: dead000000100100
[   53.888900] FS:  00007fe6ef1d97a0(0000) GS:ffff88011f900000(0000) knlGS:0000000000000000
[   53.888907] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[   53.888912] CR2: 00007f1c4090ddd0 CR3: 000000010f0cd000 CR4: 0000000000042660
[   53.888917] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   53.888923] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   53.888928] Stack:
[   53.888932]  ffff880109533a90 ffff88011e3c1c18 ffff880109533a90 ffffffff81421460
[   53.888943]  ffff88011e3c1ba8 ffffffff81663cb8 ffff88011e3c1be8 ffffffff81663eb5
[   53.888954]  0000000000000000 ffff88011e3c1c18 0000000000000000 ffffffff8131c410
[   53.888965] Call Trace:
[   53.888971]  [<ffffffff81421460>] ? class_for_each_device+0xa0/0xa0
[   53.888978]  [<ffffffff81663cb8>] klist_dec_and_del+0x28/0x30
[   53.888985]  [<ffffffff81663eb5>] klist_next+0x45/0x140
[   53.888992]  [<ffffffff8131c410>] ? pci_match_next_bus+0x10/0x10
[   53.888999]  [<ffffffff81421108>] class_dev_iter_next+0x18/0x50
[   53.889006]  [<ffffffff81421358>] class_find_device+0x48/0xb0
[   53.889012]  [<ffffffff8131c8aa>] pci_get_next_root_bus+0x4a/0x80
[   53.889020]  [<ffffffff8132f6a5>] pcifront_free_roots+0x25/0x60
[   53.889026]  [<ffffffff8132f6f1>] free_pdev+0x11/0x80
[   53.889032]  [<ffffffff8132f77a>] pcifront_xenbus_remove+0x1a/0x20
[   53.889039]  [<ffffffff813abae8>] xenbus_dev_remove+0x38/0x70
[   53.889046]  [<ffffffff8141ffc1>] __device_release_driver+0x61/0xd0
[   53.889054]  [<ffffffff81420148>] device_release_driver+0x28/0x40
[   53.889062]  [<ffffffff8141f296>] bus_remove_device+0x106/0x140
[   53.889064]  [<ffffffff8141d120>] device_del+0x110/0x1c0
[   53.889064]  [<ffffffff8141d1e1>] device_unregister+0x11/0x20
[   53.889064]  [<ffffffff813ab946>] xenbus_dev_changed+0x96/0x1d0
[   53.889064]  [<ffffffff81048bd6>] ? xen_spin_lock+0xa6/0x110
[   53.889064]  [<ffffffff813ad2f6>] frontend_changed+0x16/0x20
[   53.889064]  [<ffffffff813a9c5b>] xenwatch_thread+0xcb/0x190
[   53.889064]  [<ffffffff810b49d0>] ? wake_up_bit+0x40/0x40
[   53.889064]  [<ffffffff813a9b90>] ? xs_watch+0x60/0x60
[   53.889064]  [<ffffffff810b42e6>] kthread+0xc6/0xd0
[   53.889064]  [<ffffffff8103a149>] ? xen_end_context_switch+0x19/0x20
[   53.889064]  [<ffffffff810b4220>] ? kthread_freezable_should_stop+0x80/0x80
[   53.889064]  [<ffffffff8168623c>] ret_from_fork+0x7c/0xb0
[   53.889064]  [<ffffffff810b4220>] ? kthread_freezable_should_stop+0x80/0x80
[   53.889064] Code: fb 48 83 ec 08 f6 47 e8 01 0f 84 e3 00 00 00 48 8b 43 f8 48 8b 53 f0 48 bf 00 01 10 00 00 00 ad de 48 be 00 02 20 00 00 00 ad de <48> 89 42 08 48 89 10 48 89 7b f0 48 89 73 f8 48 c7 c7 54 44 d7 
[   53.889064] RIP  [<ffffffff81663baa>] klist_release+0x3a/0x120
[   53.889064]  RSP <ffff88011e3c1b78>
[   53.889285] ---[ end trace 918a23c9bada3aef ]---
[   53.889290] BUG: sleeping function called from invalid context at /home/konrad/linux/kernel/rwsem.c:20
[   53.889296] in_atomic(): 1, irqs_disabled(): 0, pid: 26, name: xenwatch
[   53.889302] CPU: 1 PID: 26 Comm: xenwatch Tainted: G      D W    3.10.0-rc4upstream-00172-g909fe9e-dirty #2
[   53.889307]  000000000000000b ffff88011e3c1928 ffffffff8167a27d ffff88011e3c1938
[   53.889319]  ffffffff810bf5a8 ffff88011e3c1958 ffffffff8167b56f 0720072007200720
[   53.889331]  ffff88011e2eb800 ffff88011e3c1998 ffffffff810a279a 0000000000000000
[   53.889342] Call Trace:
[   53.889347]  [<ffffffff8167a27d>] dump_stack+0x19/0x1b
[   53.889354]  [<ffffffff810bf5a8>] __might_sleep+0xd8/0x100
[   53.889360]  [<ffffffff8167b56f>] down_read+0x1f/0x40
[   53.889368]  [<ffffffff810a279a>] exit_signals+0x2a/0x170
[   53.889374]  [<ffffffff81091ddf>] do_exit+0xaf/0xbe0
[   53.889380]  [<ffffffff8167a12e>] ? printk+0x48/0x4a
[   53.889387]  [<ffffffff810423f2>] ? check_events+0x12/0x20
[   53.889394]  [<ffffffff8167f220>] oops_end+0xb0/0xf0
[   53.889401]  [<ffffffff8104da06>] die+0x56/0x90
[   53.889407]  [<ffffffff8167effc>] do_general_protection+0xdc/0x160
[   53.889414]  [<ffffffff81421460>] ? class_for_each_device+0xa0/0xa0
[   53.889420]  [<ffffffff8167e668>] general_protection+0x28/0x30
[   53.889427]  [<ffffffff81421460>] ? class_for_each_device+0xa0/0xa0
[   53.889434]  [<ffffffff81663baa>] ? klist_release+0x3a/0x120
[   53.889441]  [<ffffffff81663c82>] ? klist_release+0x112/0x120
[   53.889447]  [<ffffffff81421460>] ? class_for_each_device+0xa0/0xa0
[   53.889454]  [<ffffffff81663cb8>] klist_dec_and_del+0x28/0x30
[   53.889461]  [<ffffffff81663eb5>] klist_next+0x45/0x140
[   53.889467]  [<ffffffff8131c410>] ? pci_match_next_bus+0x10/0x10
[   53.889473]  [<ffffffff81421108>] class_dev_iter_next+0x18/0x50
[   53.889480]  [<ffffffff81421358>] class_find_device+0x48/0xb0
[   53.889486]  [<ffffffff8131c8aa>] pci_get_next_root_bus+0x4a/0x80
[   53.889493]  [<ffffffff8132f6a5>] pcifront_free_roots+0x25/0x60
[   54.002497]  [<ffffffff8132f6f1>] free_pdev+0x11/0x80
[   54.002505]  [<ffffffff8132f77a>] pcifront_xenbus_remove+0x1a/0x20
[   54.002516]  [<ffffffff813abae8>] xenbus_dev_remove+0x38/0x70
[   54.002524]  [<ffffffff8141ffc1>] __device_release_driver+0x61/0xd0
[   54.002530]  [<ffffffff81420148>] device_release_driver+0x28/0x40
[   54.002537]  [<ffffffff8141f296>] bus_remove_device+0x106/0x140
[   54.002544]  [<ffffffff8141d120>] device_del+0x110/0x1c0
[   54.002551]  [<ffffffff8141d1e1>] device_unregister+0x11/0x20
[   54.002558]  [<ffffffff813ab946>] xenbus_dev_changed+0x96/0x1d0
[   54.002567]  [<ffffffff81048bd6>] ? xen_spin_lock+0xa6/0x110
[   54.002574]  [<ffffffff813ad2f6>] frontend_changed+0x16/0x20
[   54.002581]  [<ffffffff813a9c5b>] xenwatch_thread+0xcb/0x190
[   54.002588]  [<ffffffff810b49d0>] ? wake_up_bit+0x40/0x40
[   54.002595]  [<ffffffff813a9b90>] ? xs_watch+0x60/0x60
[   54.002601]  [<ffffffff810b42e6>] kthread+0xc6/0xd0
[   54.002607]  [<ffffffff8103a149>] ? xen_end_context_switch+0x19/0x20
[   54.002615]  [<ffffffff810b4220>] ? kthread_freezable_should_stop+0x80/0x80
[   54.002624]  [<ffffffff8168623c>] ret_from_fork+0x7c/0xb0
[   54.002630]  [<ffffffff810b4220>] ? kthread_freezable_should_stop+0x80/0x80
[   54.002637] note: xenwatch[26] exited with preempt_count 1

# 
# 

Re: [RFC PATCH v2, part3 07/11] PCI, xen-pcifront: use new PCI interfaces to simplify implementation

[Jiang Liu]

On Fri 07 Jun 2013 10:50:24 PM CST, Konrad Rzeszutek Wilk wrote:
> On Thu, May 16, 2013 at 11:50:55PM +0800, Jiang Liu wrote:
>> Use new PCI interfaces to simplify xen-pcifront implementation:
>> 1) Use pci_create_root_bus() instead of pci_scan_bus_parented()
>>    because pci_scan_bus_parented() is marked as __deprecated.This
>>    also gets rid of a duplicated call of pci_bus_start_devices().
>> 2) Use pci_stop_root_bus() and pci_remove_root_bus() instead of
>>    open-coded private implementation.
>> 3) Use pci_set_host_bridge_release() to release data structures
>>    associated with PCI root buses.
>> 4) Use pci_bus_get()/pci_bus_put() to manage PCI root bus reference
>>    count.
>>
>> This is also a preparation for coming PCI bus lock enhancement.
>>
>> Signed-off-by: Jiang Liu <jiang.liu@huawei.com>
>> Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
>> Cc: Jeremy Fitzhardinge <jeremy@goop.org>
>> Cc: xen-devel@lists.xensource.com
>> Cc: virtualization@lists.linux-foundation.org
>> Cc: linux-pci@vger.kernel.org
>> Cc: linux-kernel@vger.kernel.org
>> ---
>>  drivers/pci/xen-pcifront.c | 81 ++++++++++++++++++++++------------------------
>>  1 file changed, 39 insertions(+), 42 deletions(-)
>>
>> diff --git a/drivers/pci/xen-pcifront.c b/drivers/pci/xen-pcifront.c
>> index 816cf94..6aa2c0f 100644
>> --- a/drivers/pci/xen-pcifront.c
>> +++ b/drivers/pci/xen-pcifront.c
......
>> @@ -480,12 +490,14 @@ static int pcifront_scan_root(struct pcifront_device *pdev,
>>  		goto err_out;
>>  	}
>>
>> -	bus_entry->bus = b;
>> +	pci_set_host_bridge_release(to_pci_host_bridge(b->bridge),
>> +				    pcifront_release_sd, sd);
>>
>> -	list_add(&bus_entry->list, &pdev->root_buses);
>> -
>> -	/* pci_scan_bus_parented skips devices which do not have a have
>> -	* devfn==0. The pcifront_scan_bus enumerates all devfn. */
>> +	/*
>> +	 * Every PCI physical device should have function 0, but that's not
>> +	 * true for xen.
>
> That is incorrect. There are two types of backends - one of them will
> start at zero, but the other might not.
Hi Konrad,
      Forgive my poor knowledge about Xen:(, so I will skip this change.
Regards!
Gerry

Re: [RFC PATCH v2, part3 07/11] PCI, xen-pcifront: use new PCI interfaces to simplify implementation

[Jiang Liu]

On 06/07/2013 11:38 PM, Konrad Rzeszutek Wilk wrote:
> On Fri, Jun 07, 2013 at 10:50:24AM -0400, Konrad Rzeszutek Wilk wrote:
>> On Thu, May 16, 2013 at 11:50:55PM +0800, Jiang Liu wrote:
>>> Use new PCI interfaces to simplify xen-pcifront implementation:
>>> 1) Use pci_create_root_bus() instead of pci_scan_bus_parented()
>>>    because pci_scan_bus_parented() is marked as __deprecated.This
>>>    also gets rid of a duplicated call of pci_bus_start_devices().
>>> 2) Use pci_stop_root_bus() and pci_remove_root_bus() instead of
>>>    open-coded private implementation.
>>> 3) Use pci_set_host_bridge_release() to release data structures
>>>    associated with PCI root buses.
>>> 4) Use pci_bus_get()/pci_bus_put() to manage PCI root bus reference
>>>    count.
>>>
>>> This is also a preparation for coming PCI bus lock enhancement.
> 
> With this patch from :
> 
>  Merge branch 'pci_lock_v3' of https://github.com/jiangliu/linux into testing
> 
> 
> it blows up when detaching the device.
Hi Konrad,
   Thanks for testing! According to the log messages, this issue should
be related to pci bus reference counter management. Seems we have done
an extra(unbalanced) release of pci bus device.
   Will investigate it tomorrow!
Regards!
Gerry

> 
> Parsing config from /vm-pv-discard.cfg
> Daemon running with PID 4062
> [    0.000000] Initializing cgroup subsys cpuset
> [    0.000000] Initializing cgroup subsys cpu
> [    0.000000] Initializing cgroup subsys cpuacct
> [    0.000000] Linux version 3.10.0-rc4upstream-00172-g909fe9e-dirty (konrad@phenom.dumpdata.com) (gcc version 4.4.4 20100503 (Red Hat 4.4.4-2) (GCC) ) #2 SMP Fri Jun 7 10:55:54 EDT 2013
> [    0.000000] Command line: console=hvc0 debug 
> [    0.000000] ACPI in unprivileged domain disabled
> [    0.000000] Freeing 20000-40000 pfn range: 131072 pages freed
> [    0.000000] 1-1 mapping on 20000->100000
> [    0.000000] Released 131072 pages of unused memory
> [    0.000000] Set 917504 page(s) to 1-1 mapping
> [    0.000000] Populating 100000-120000 pfn range: 131072 pages added
> [    0.000000] e820: BIOS-provided physical RAM map:
> [    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
> [    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
> [    0.000000] Xen: [mem 0x0000000000100000-0x000000001fffffff] usable
> [    0.000000] Xen: [mem 0x0000000020000000-0x00000000201fffff] reserved
> [    0.000000] Xen: [mem 0x0000000020200000-0x000000003fffffff] unusable
> [    0.000000] Xen: [mem 0x0000000040000000-0x00000000401fffff] reserved
> [    0.000000] Xen: [mem 0x0000000040200000-0x00000000c6cd3fff] unusable
> [    0.000000] Xen: [mem 0x00000000c6cd4000-0x00000000c6d1cfff] ACPI NVS
> [    0.000000] Xen: [mem 0x00000000c6d1d000-0x00000000c6d27fff] ACPI data
> [    0.000000] Xen: [mem 0x00000000c6d28000-0x00000000c6d28fff] ACPI NVS
> [    0.000000] Xen: [mem 0x00000000c6d29000-0x00000000c6d49fff] reserved
> [    0.000000] Xen: [mem 0x00000000c6d4a000-0x00000000c6d4bfff] unusable
> [    0.000000] Xen: [mem 0x00000000c6d4c000-0x00000000c6d6cfff] ACPI NVS
> [    0.000000] Xen: [mem 0x00000000c6d6d000-0x00000000c6d8ffff] reserved
> [    0.000000] Xen: [mem 0x00000000c6d90000-0x00000000c6d9cfff] ACPI NVS
> [    0.000000] Xen: [mem 0x00000000c6d9d000-0x00000000c6d9ffff] reserved
> [    0.000000] Xen: [mem 0x00000000c6da0000-0x00000000c6db0fff] ACPI NVS
> [    0.000000] Xen: [mem 0x00000000c6db1000-0x00000000c6ddcfff] reserved
> [    0.000000] Xen: [mem 0x00000000c6ddd000-0x00000000c6e1ffff] ACPI NVS
> [    0.000000] Xen: [mem 0x00000000c6e20000-0x00000000c6ffffff] unusable
> [    0.000000] Xen: [mem 0x00000000c7800000-0x00000000cf9fffff] reserved
> [    0.000000] Xen: [mem 0x00000000fed1c000-0x00000000fed3ffff] reserved
> [    0.000000] Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
> [    0.000000] Xen: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
> [    0.000000] Xen: [mem 0x0000000100000000-0x000000011fffffff] usable
> [    0.000000] NX (Execute Disable) protection: active
> [    0.000000] DMI not present or invalid.
> [    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
> [    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
> [    0.000000] No AGP bridge found
> [    0.000000] e820: last_pfn = 0x120000 max_arch_pfn = 0x400000000
> [    0.000000] e820: last_pfn = 0x20000 max_arch_pfn = 0x400000000
> [    0.000000] Scanning 1 areas for low memory corruption
> [    0.000000] Base memory trampoline at [ffff88000009a000] 9a000 size 24576
> [    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
> [    0.000000]  [mem 0x00000000-0x000fffff] page 4k
> [    0.000000] init_memory_mapping: [mem 0x11fe00000-0x11fffffff]
> [    0.000000]  [mem 0x11fe00000-0x11fffffff] page 4k
> [    0.000000] BRK [0x01d8a000, 0x01d8afff] PGTABLE
> [    0.000000] BRK [0x01d8b000, 0x01d8bfff] PGTABLE
> [    0.000000] init_memory_mapping: [mem 0x11c000000-0x11fdfffff]
> [    0.000000]  [mem 0x11c000000-0x11fdfffff] page 4k
> [    0.000000] BRK [0x01d8c000, 0x01d8cfff] PGTABLE
> [    0.000000] BRK [0x01d8d000, 0x01d8dfff] PGTABLE
> [    0.000000] BRK [0x01d8e000, 0x01d8efff] PGTABLE
> [    0.000000] init_memory_mapping: [mem 0x100000000-0x11bffffff]
> [    0.000000]  [mem 0x100000000-0x11bffffff] page 4k
> [    0.000000] init_memory_mapping: [mem 0x00100000-0x1fffffff]
> [    0.000000]  [mem 0x00100000-0x1fffffff] page 4k
> [    0.000000] RAMDISK: [mem 0x02197000-0x13f67fff]
> [    0.000000] NUMA turned off
> [    0.000000] Faking a node at [mem 0x0000000000000000-0x000000011fffffff]
> [    0.000000] Initmem setup node 0 [mem 0x00000000-0x11fffffff]
> [    0.000000]   NODE_DATA [mem 0x11fea2000-0x11fea5fff]
> [    0.000000] Zone ranges:
> [    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
> [    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
> [    0.000000]   Normal   [mem 0x100000000-0x11fffffff]
> [    0.000000] Movable zone start for each node
> [    0.000000] Early memory node ranges
> [    0.000000]   node   0: [mem 0x00001000-0x0009ffff]
> [    0.000000]   node   0: [mem 0x00100000-0x1fffffff]
> [    0.000000]   node   0: [mem 0x100000000-0x11fffffff]
> [    0.000000] On node 0 totalpages: 262047
> [    0.000000]   DMA zone: 56 pages used for memmap
> [    0.000000]   DMA zone: 21 pages reserved
> [    0.000000]   DMA zone: 3999 pages, LIFO batch:0
> [    0.000000]   DMA32 zone: 1736 pages used for memmap
> [    0.000000]   DMA32 zone: 126976 pages, LIFO batch:31
> [    0.000000]   Normal zone: 1792 pages used for memmap
> [    0.000000]   Normal zone: 131072 pages, LIFO batch:31
> [    0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs
> [    0.000000] No local APIC present
> [    0.000000] APIC: disable apic facility
> [    0.000000] APIC: switched to apic NOOP
> [    0.000000] nr_irqs_gsi: 16
> [    0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
> [    0.000000] PM: Registered nosave memory: 0000000020000000 - 0000000020200000
> [    0.000000] PM: Registered nosave memory: 0000000020200000 - 0000000040000000
> [    0.000000] PM: Registered nosave memory: 0000000040000000 - 0000000040200000
> [    0.000000] PM: Registered nosave memory: 0000000040200000 - 00000000c6cd4000
> [    0.000000] PM: Registered nosave memory: 00000000c6cd4000 - 00000000c6d1d000
> [    0.000000] PM: Registered nosave memory: 00000000c6d1d000 - 00000000c6d28000
> [    0.000000] PM: Registered nosave memory: 00000000c6d28000 - 00000000c6d29000
> [    0.000000] PM: Registered nosave memory: 00000000c6d29000 - 00000000c6d4a000
> [    0.000000] PM: Registered nosave memory: 00000000c6d4a000 - 00000000c6d4c000
> [    0.000000] PM: Registered nosave memory: 00000000c6d4c000 - 00000000c6d6d000
> [    0.000000] PM: Registered nosave memory: 00000000c6d6d000 - 00000000c6d90000
> [    0.000000] PM: Registered nosave memory: 00000000c6d90000 - 00000000c6d9d000
> [    0.000000] PM: Registered nosave memory: 00000000c6d9d000 - 00000000c6da0000
> [    0.000000] PM: Registered nosave memory: 00000000c6da0000 - 00000000c6db1000
> [    0.000000] PM: Registered nosave memory: 00000000c6db1000 - 00000000c6ddd000
> [    0.000000] PM: Registered nosave memory: 00000000c6ddd000 - 00000000c6e20000
> [    0.000000] PM: Registered nosave memory: 00000000c6e20000 - 00000000c7000000
> [    0.000000] PM: Registered nosave memory: 00000000c7000000 - 00000000c7800000
> [    0.000000] PM: Registered nosave memory: 00000000c7800000 - 00000000cfa00000
> [    0.000000] PM: Registered nosave memory: 00000000cfa00000 - 00000000fed1c000
> [    0.000000] PM: Registered nosave memory: 00000000fed1c000 - 00000000fed40000
> [    0.000000] PM: Registered nosave memory: 00000000fed40000 - 00000000fee00000
> [    0.000000] PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
> [    0.000000] PM: Registered nosave memory: 00000000fee01000 - 00000000ff000000
> [    0.000000] PM: Registered nosave memory: 00000000ff000000 - 0000000100000000
> [    0.000000] e820: [mem 0xcfa00000-0xfed1bfff] available for PCI devices
> [    0.000000] Booting paravirtualized kernel on Xen
> [    0.000000] Xen version: 4.3-unstable (preserve-AD)
> [    0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:2 nr_node_ids:1
> [    0.000000] PERCPU: Embedded 28 pages/cpu @ffff88011f800000 s85632 r8192 d20864 u1048576
> [    0.000000] pcpu-alloc: s85632 r8192 d20864 u1048576 alloc=1*2097152
> [    0.000000] pcpu-alloc: [0] 0 1 
> [    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 258442
> [    0.000000] Policy zone: Normal
> [    0.000000] Kernel command line: console=hvc0 debug 
> [    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
> [    0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340
> [    0.000000] Checking aperture...
> [    0.000000] No AGP bridge found
> [    0.000000] Memory: 722480k/4718592k available (6698k kernel code, 3670404k absent, 325708k reserved, 4234k data, 1724k init)
> [    0.000000] Hierarchical RCU implementation.
> [    0.000000] 	RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=2.
> [    0.000000] NR_IRQS:33024 nr_irqs:288 16
> [    0.000000] Console: colour dummy device 80x25
> [    0.000000] console [tty0] enabled
> [    0.000000] console [hvc0] enabled
> [    0.000000] Xen: using vcpuop timer interface
> [    0.000000] installing Xen timer for CPU 0
> [    0.000000] tsc: Detected 3092.926 MHz processor
> [    0.001000] Calibrating delay loop (skipped), value calculated using timer frequency.. 6185.85 BogoMIPS (lpj=3092926)
> [    0.001000] pid_max: default: 32768 minimum: 301
> [    0.001000] Security Framework initialized
> [    0.001000] SELinux:  Initializing.
> [    0.001000] SELinux:  Starting in permissive mode
> [    0.001000] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
> [    0.001000] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
> [    0.001000] Mount-cache hash table entries: 256
> [    0.001582] Initializing cgroup subsys freezer
> [    0.001698] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
> [    0.001698] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
> [    0.001708] CPU: Physical Processor ID: 0
> [    0.001712] CPU: Processor Core ID: 0
> [    0.001718] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0
> [    0.001718] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32
> [    0.001718] tlb_flushall_shift: 5
> [    0.025468] cpu 0 spinlock event irq 17
> [    0.025566] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only.
> [    0.026081] NMI watchdog: disabled (cpu0): hardware events not enabled
> [    0.026680] installing Xen timer for CPU 1
> [    0.026723] cpu 1 spinlock event irq 24
> [    0.026793] SMP alternatives: switching to SMP code
> [    0.048014] Brought up 2 CPUs
> [    0.051334] PM: Registering ACPI NVS region [mem 0xc6cd4000-0xc6d1cfff] (299008 bytes)
> [    0.051334] PM: Registering ACPI NVS region [mem 0xc6d28000-0xc6d28fff] (4096 bytes)
> [    0.051334] PM: Registering ACPI NVS region [mem 0xc6d4c000-0xc6d6cfff] (135168 bytes)
> [    0.051334] PM: Registering ACPI NVS region [mem 0xc6d90000-0xc6d9cfff] (53248 bytes)
> [    0.051334] PM: Registering ACPI NVS region [mem 0xc6da0000-0xc6db0fff] (69632 bytes)
> [    0.051334] PM: Registering ACPI NVS region [mem 0xc6ddd000-0xc6e1ffff] (274432 bytes)
> [    0.051399] kworker/u4:0 (19) used greatest stack depth: 6016 bytes left
> [    0.052492] Grant tables using version 2 layout.
> [    0.052492] Grant table initialized
> [    0.071696] RTC time: 165:165:165, date: 165/165/65
> [    0.071985] NET: Registered protocol family 16
> [    0.072173] kworker/u4:0 (22) used greatest stack depth: 5928 bytes left
> [    0.073110] kworker/u4:0 (30) used greatest stack depth: 5464 bytes left
> [    0.074030] dca service started, version 1.12.1
> [    0.074563] PCI: setting up Xen PCI frontend stub
> [    0.074568] PCI: pci_cache_line_size set to 64 bytes
> [    0.101148] bio: create slab <bio-0> at 0
> [    0.101220] ACPI: Interpreter disabled.
> [    0.102003] xen/balloon: Initialising balloon driver.
> [    0.103081] xen-balloon: Initialising balloon driver.
> [    0.104136] vgaarb: loaded
> [    0.105135] usbcore: registered new interface driver usbfs
> [    0.105135] usbcore: registered new interface driver hub
> [    0.105135] usbcore: registered new device driver usb
> [    0.106089] pps_core: LinuxPPS API ver. 1 registered
> [    0.106089] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
> [    0.106089] PTP clock support registered
> [    0.106089] PCI: System does not support PCI
> [    0.106089] PCI: System does not support PCI
> [    0.107154] NetLabel: Initializing
> [    0.107154] NetLabel:  domain hash size = 128
> [    0.107154] NetLabel:  protocols = UNLABELED CIPSOv4
> [    0.107175] NetLabel:  unlabeled traffic allowed by default
> [    0.108087] Switching to clocksource xen
> [    0.125349] pnp: PnP ACPI: disabled
> [    0.138965] NET: Registered protocol family 2
> [    0.139578] TCP established hash table entries: 8192 (order: 5, 131072 bytes)
> [    0.139647] TCP bind hash table entries: 8192 (order: 5, 131072 bytes)
> [    0.139668] TCP: Hash tables configured (established 8192 bind 8192)
> [    0.193672] TCP: reno registered
> [    0.193693] UDP hash table entries: 512 (order: 2, 16384 bytes)
> [    0.193712] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
> [    0.194030] NET: Registered protocol family 1
> [    0.194380] RPC: Registered named UNIX socket transport module.
> [    0.194389] RPC: Registered udp transport module.
> [    0.194394] RPC: Registered tcp transport module.
> [    0.194398] RPC: Registered tcp NFSv4.1 backchannel transport module.
> [    0.194404] PCI: CLS 0 bytes, default 64
> [    0.194570] Unpacking initramfs...
> [    0.778545] Freeing initrd memory: 292676k freed
> [    0.823501] platform rtc_cmos: registered platform RTC device (no PNP device found)
> [    0.823855] Machine check injector initialized
> [    0.825040] Scanning for low memory corruption every 60 seconds
> [    0.826072] audit: initializing netlink socket (disabled)
> [    0.826114] type=2000 audit(1370618465.698:1): initialized
> [    0.839204] HugeTLB registered 2 MB page size, pre-allocated 0 pages
> [    0.839673] VFS: Disk quotas dquot_6.5.2
> [    0.839760] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
> [    0.840528] NFS: Registering the id_resolver key type
> [    0.840553] Key type id_resolver registered
> [    0.840558] Key type id_legacy registered
> [    0.840587] NTFS driver 2.1.30 [Flags: R/W].
> [    0.840919] msgmni has been set to 1982
> [    0.841082] SELinux:  Registering netfilter hooks
> [    0.842697] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
> [    0.842706] io scheduler noop registered
> [    0.842710] io scheduler deadline registered
> [    0.842772] io scheduler cfq registered (default)
> [    0.843309] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
> [    0.846284] intel_idle: does not run on family 6 model 42
> [    0.846299] ioatdma: Intel(R) QuickData Technology Driver 4.00
> [    0.847400] pcifront pci-0: Installing PCI frontend
> [    0.847417] Warning: only able to allocate 4 MB for software IO TLB
> [    0.850807] software IO TLB [mem 0x109000000-0x109400000] (4MB) mapped at [ffff880109000000-ffff8801093fffff]
> [    0.851175] pcifront pci-0: Creating PCI Frontend Bus 0000:00
> [    0.851458] pcifront pci-0: PCI host bridge to bus 0000:00
> [    0.851469] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
> [    0.851478] pci_bus 0000:00: root bus resource [mem 0x00000000-0xfffffffff]
> [    0.851487] pci_bus 0000:00: root bus resource [bus 00]
> [    0.852031] pci 0000:00:00.0: [8086:105e] type 00 class 0x020000
> [    0.852419] pci 0000:00:00.0: reg 0x10: [mem 0xfe4a0000-0xfe4bffff]
> [    0.852631] pci 0000:00:00.0: reg 0x14: [mem 0xfe480000-0xfe49ffff]
> [    0.852743] pci 0000:00:00.0: reg 0x18: [io  0xe020-0xe03f]
> [    0.853905] pcifront pci-0: New device on 0000:00:00.0 found.
> [    0.857483] pcifront pci-0: claiming resource 0000:00:00.0/0
> [    0.857490] pcifront pci-0: claiming resource 0000:00:00.0/1
> [    0.857495] pcifront pci-0: claiming resource 0000:00:00.0/2
> [    0.939621] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
> [    0.942227] Non-volatile memory driver v1.3
> [    0.942414] Linux agpgart interface v0.103
> [    0.943295] [drm] Initialized drm 1.1.0 20060810
> [    0.947952] loop: module loaded
> [    0.948577] libphy: Fixed MDIO Bus: probed
> [    0.948583] tun: Universal TUN/TAP device driver, 1.6
> [    0.948588] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
> [    0.948913] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.7.12-k
> [    0.948921] ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
> [    0.950031] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
> [    0.950039] ehci_hcd: block sizes: qh 112 qtd 96 itd 192 sitd 96
> [    0.950061] ehci-pci: EHCI PCI platform driver
> [    0.950201] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
> [    0.950207] ohci_hcd: block sizes: ed 80 td 96
> [    0.950357] uhci_hcd: USB Universal Host Controller Interface driver
> [    0.950736] usbcore: registered new interface driver usblp
> [    0.951285] i8042: PNP: No PS/2 controller found. Probing ports directly.
> [    1.960646] i8042: No controller found
> [    1.960872] mousedev: PS/2 mouse device common for all mice
> [    2.021775] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
> [    2.021941] rtc_cmos: probe of rtc_cmos failed with error -38
> [    2.023233] zram: Created 1 device(s) ...
> [    2.023567] Netfilter messages via NETLINK v0.30.
> [    2.023612] nf_conntrack version 0.5.0 (7930 buckets, 31720 max)
> [    2.023775] ctnetlink v0.93: registering with nfnetlink.
> [    2.024045] ip_tables: (C) 2000-2006 Netfilter Core Team
> [    2.024170] TCP: cubic registered
> [    2.024177] Initializing XFRM netlink socket
> [    2.024379] NET: Registered protocol family 10
> [    2.024936] ip6_tables: (C) 2000-2006 Netfilter Core Team
> [    2.025259] sit: IPv6 over IPv4 tunneling driver
> [    2.026039] NET: Registered protocol family 17
> [    2.026320] Key type dns_resolver registered
> [    2.026874] PM: Hibernation image not present or could not be loaded.
> [    2.026922] registered taskstats version 1
> [    2.027005] kmemleak: Kernel memory leak detector initialized
> [    2.027009] kmemleak: Automatic memory scanning thread started
> [    2.027437] XENBUS: Device with no driver: device/vbd/51712
> [    2.027537]   Magic number: 1:252:3141
> [    2.029169] Freeing unused kernel memory: 1724k freed
> [    2.029408] Write protecting the kernel read-only data: 10240k
> [    2.033649] Freeing unused kernel memory: 1480k freed
> [    2.033908] Freeing unused kernel memory: 72k freed
> 
> init started: BusyBox v1.14.3 (2013-06-07 10:58:39 EDT)
> [    2.040621] consoletype (1038) used greatest stack depth: 5304 bytes left
> Mounting directories  [  OK  ]
> [    2.059689] chmod (1044) used greatest stack depth: 4856 bytes left
> mount: mount point /proc/bus/usb does not exist
> mount: mount point /sys/kernel/config does not exist
> [    2.266374] Initialising Xen virtual ethernet driver.
> [    2.273190] vbd vbd-51712: blkfront:blkback_changed to state 2.
> [    2.283349] vbd vbd-51712: blkfront:blkback_changed to state 4.
> [    2.283356] vbd vbd-51712: blkfront_connect:/local/domain/0/backend/vbd/2/51712.
> [    2.287707] blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled;
> [    2.288337] Entered do_blkif_request
> [    2.288344] Entered do_blkif_request
> [    2.288821] Entered do_blkif_request
> [    2.288829] do_blk_req ffff880108ec9e10: cmd ffff880108ec9f00, sec 0, (4/4) buffer:ffff880108ee6000 [read]
> [    2.288839] Entered do_blkif_request
> [    2.288859] Entered do_blkif_request
> [    2.288864] do_blk_req ffff880108ec9cb8: cmd ffff880108ec9da8, sec 4, (4/4) buffer:ffff880108ee6800 [read]
> [    2.288871] Entered do_blkif_request
> [    2.292721] Entered do_blkif_request
> [    2.292727] Entered do_blkif_request
> [    2.292737] Entered do_blkif_request
> [    2.292741] Entered do_blkif_request
> [    2.292780] Entered do_blkif_request
> [    2.292786] do_blk_req ffff880108ec9cb8: cmd ffff880108ec9da8, sec 8, (4/4) buffer:ffff88010f273000 [read]
> [    2.292794] Entered do_blkif_request
> [    2.292811] Entered do_blkif_request
> [    2.292816] do_blk_req ffff880108ec9e10: cmd ffff880108ec9f00, sec c, (4/4) buffer:ffff88010f273800 [read]
> [    2.292822] Entered do_blkif_request
> [    2.292940] Entered do_blkif_request
> [    2.292947] Entered do_blkif_request
> [    2.292957] Entered do_blkif_request
> [    2.292961] Entered do_blkif_request
> [    2.292974]  xvda: unknown partition table
> [    2.385804] udevd (1099): /proc/1099/oom_adj is deprecated, please use /proc/1099/oom_score_adj instead.
> [    2.444038] e1000e: Intel(R) PRO/1000 Network Driver - 2.3.2-k
> [    2.444061] e1000e: Copyright(c) 1999 - 2013 Intel Corporation.
> [    2.444110] e1000e 0000:00:00.0: Disabling ASPM  L1
> [    2.444232] e1000e 0000:00:00.0: enabling device (0000 -> 0002)
> [    2.447244] e1000e 0000:00:00.0: Xen PCI mapped GSI16 to IRQ34
> [    2.447790] e1000e 0000:00:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
> udevd-work[1106]: error opening ATTR{/sys/devices/system/cpu/cpu0/online} for writing: No such file or directory
> 
> [    2.459644] vbd vbd-51712: command: 0x5331, argument: 0x0
> [    2.560213] ip (1325) used greatest stack depth: 3760 bytes left
> [    2.624581] e1000e 0000:00:00.0 eth0: (PCI Express:2.5GT/s:Width x4) 00:15:17:8f:18:a2
> [    2.624595] e1000e 0000:00:00.0 eth0: Intel(R) PRO/1000 Network Connection
> [    2.624675] e1000e 0000:00:00.0 eth0: MAC: 0, PHY: 4, PBA No: D50868-003
> Waiting for devices [  OK  ]
> Waiting for fb [  OK  ]
> Starting..[/dev/fb0]
> Could not open; err: 2
> FATAL: Module agpgart_intel not found.
> [    2.814571] [drm] radeon kernel modesetting enabled.
> WARNING: Error inserting video (/lib/modules/3.10.0-rc4upstream-00172-g909fe9e-dirty/kernel/drivers/acpi/video.ko): No such device
> WARNING: Error inserting mxm_wmi (/lib/modules/3.10.0-rc4upstream-00172-g909fe9e-dirty/kernel/drivers/platform/x86/mxm-wmi.ko): No such device
> WARNING: Error inserting drm_kms_helper (/lib/modules/3.10.0-rc4upstream-00172-g909fe9e-dirty/kernel/drivers/gpu/drm/drm_kms_helper.ko): No such device
> WARNING: Error inserting ttm (/lib/modules/3.10.0-rc4upstream-00172-g909fe9e-dirty/kernel/drivers/gpu/drm/ttm/ttm.ko): No such device
> FATAL: Error inserting nouveau (/lib/modules/3.10.0-rc4upstream-00172-g909fe9e-dirty/kernel/drivers/gpu/drm/nouveau/nouveau.ko): No such device
> WARNING: Error inserting drm_kms_helper (/lib/modules/3.10.0-rc4upstream-00172-g909fe9e-dirty/kernel/drivers/gpu/drm/drm_kms_helper.ko): No such device
> FATAL: Error inserting i915 (/lib/modules/3.10.0-rc4upstream-00172-g909fe9e-dirty/kernel/drivers/gpu/drm/i915/i915.ko): No such device
> Starting..[/dev/fb0]
> Could not open; err: 2
> VGA: 0000:
> Waiting for network [  OK  ]
> Bringing up loopback interface:  [  OK  ]
> Bringing up interface eth0:  [    3.394987] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
> [    3.397647] device eth0 entered promiscuous mode
> [  OK  ]
> Bringing up interface switch:  
> Determining IP information for switch...[    3.452997] IPv6: ADDRCONF(NETDEV_UP): switch: link is not ready
> [    5.533965] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> [    5.534117] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [    5.534222] switch: port 1(eth0) entered forwarding state
> [    5.534239] switch: port 1(eth0) entered forwarding state
> [    5.534302] IPv6: ADDRCONF(NETDEV_CHANGE): switch: link becomes ready
>  done.
> [  OK  ]
> Waiting for init.custom [  OK  ]
> 
> Starting SSHd ...
> 
>     SSH started [2202]
> 
> 
> Waiting for SSHd [  OK  ]
> WARNING: ssh currently running [2202] ignoring start request
> FATAL: Module dump_dma not found.
> ERROR: Module dump_dma does not exist in /proc/modules
> [    9.129856] SCSI subsystem initialized
> [    9.131318] Loading iSCSI transport class v2.0-870.
> [    9.134894] iscsi: registered transport (tcp)
> hostname: Name or service not known
> iscsistart: transport class version 2.0-870. iscsid version 2.0-872
> Could not get list of targets from firmware.
> Jun  7 15:21:14 (none) syslogd 1.5.0: restart.
> Running in PV context on Xen v4.3.
> FATAL: Module evtchn not found.
> [    9.176036] Event-channel device installed.
> 00:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
>            CPU0       CPU1       
>  16:       2600          0  xen-percpu-virq      timer0
>  17:          0          0  xen-percpu-ipi       spinlock0
>  18:       3473          0  xen-percpu-ipi       resched0
>  19:          0          0  xen-percpu-ipi       callfunc0
>  20:          0          0  xen-percpu-virq      debug0
>  21:        237          0  xen-percpu-ipi       callfuncsingle0
>  22:          0          0  xen-percpu-ipi       irqwork0
>  23:          0       2560  xen-percpu-virq      timer1
>  24:          0          0  xen-percpu-ipi       spinlock1
>  25:          0       3833  xen-percpu-ipi       resched1
>  26:          0          0  xen-percpu-ipi       callfunc1
>  27:          0          0  xen-percpu-virq      debug1
>  28:          0        271  xen-percpu-ipi       callfuncsingle1
>  29:          0          0  xen-percpu-ipi       irqwork1
>  30:        235          0   xen-dyn-event     xenbus
>  31:         64          0   xen-dyn-event     pcifront
>  32:         89          0   xen-dyn-event     hvc_console
>  33:          4          0   xen-dyn-event     blkif
>  35:         27          0  xen-pirq-pcifront-msi  eth0
> NMI:          0          0   Non-maskable interrupts
> LOC:          0          0   Local timer interrupts
> SPU:          0          0   Spurious interrupts
> PMI:          0          0   Performance monitoring interrupts
> IWI:          0          0   IRQ work interrupts
> RTR:          0          0   APIC ICR read retries
> RES:       3473       3833   Rescheduling interrupts
> CAL:        237        271   Function call interrupts
> TLB:          0          0   TLB shootdowns
> TRM:          0          0   Thermal event interrupts
> THR:          0          0   Threshold APIC interrupts
> MCE:          0          0   Machine check exceptions
> MCP:          0          0   Machine check polls
> ERR:          0
> MIS:          0
> 00000000-00000fff : reserved
> 00001000-0009ffff : System RAM
> 000a0000-000fffff : reserved
>   000f0000-000fffff : System ROM
> 00100000-1fffffff : System RAM
>   01000000-0168a927 : Kernel code
>   0168a928-01aad2bf : Kernel data
>   01c65000-01d74fff : Kernel bss
> 20200000-3fffffff : Unusable memory
> 40200000-c6cd3fff : Unusable memory
> c6cd4000-c6d1cfff : ACPI Non-volatile Storage
> c6d1d000-c6d27fff : ACPI Tables
> c6d28000-c6d28fff : ACPI Non-volatile Storage
> c6d4a000-c6d4bfff : Unusable memory
> c6d4c000-c6d6cfff : ACPI Non-volatile Storage
> c6d90000-c6d9cfff : ACPI Non-volatile Storage
> c6da0000-c6db0fff : ACPI Non-volatile Storage
> c6ddd000-c6e1ffff : ACPI Non-volatile Storage
> c6e20000-c6ffffff : Unusable memory
> fe480000-fe49ffff : 0000:00:00.0
>   fe480000-fe49ffff : e1000e
> fe4a0000-fe4bffff : 0000:00:00.0
>   fe4a0000-fe4bffff : e1000e
> 100000000-11fffffff : System RAM
> MemTotal:        1018432 kB
> MemFree:          641536 kB
> Buffers:               0 kB
> Cached:           310180 kB
> SwapCached:            0 kB
> Active:            19368 kB
> Inactive:         290652 kB
> Active(anon):      13120 kB
> Inactive(anon):    84540 kB
> Active(file):       6248 kB
> Inactive(file):   206112 kB
> Unevictable:        4940 kB
> Mlocked:            4940 kB
> SwapTotal:             0 kB
> SwapFree:              0 kB
> Dirty:                 0 kB
> Writeback:             0 kB
> AnonPages:          4684 kB
> Mapped:             5152 kB
> Shmem:             94284 kB
> Slab:              45924 kB
> SReclaimable:      12804 kB
> SUnreclaim:        33120 kB
> KernelStack:         432 kB
> PageTables:          692 kB
> NFS_Unstable:          0 kB
> Bounce:                0 kB
> WritebackTmp:          0 kB
> CommitLimit:      509216 kB
> Committed_AS:     102792 kB
> VmallocTotal:   34359738367 kB
> VmallocUsed:        2332 kB
> VmallocChunk:   34359735931 kB
> AnonHugePages:         0 kB
> HugePages_Total:       0
> HugePages_Free:        0
> HugePages_Rsvd:        0
> HugePages_Surp:        0
> Hugepagesize:       2048 kB
> DirectMap4k:     1048576 kB
> DirectMap2M:           0 kB
> Waiting for init.late [  OK  ]
> PING build.dumpdata.com (192.168.102.1) 56(84) bytes of data.
> 
> --- build.dumpdata.com ping statistics ---
> 1 packets transmitted, 1 received, 0% packet loss, time 0ms
> rtt min/avg/max/mdev = 0.208/0.208/0.208/0.000 ms
> [    9.336320] mount.nfs (2304) used greatest stack depth: 3320 bytes left
> mount.nfs: access denied by server while mounting build:/srv/results
> NFS done
> /init.late: line 13: can't create /mnt/results/dmesg/(none)-3.10.0-rc4upstream-00172-g909fe9e-dirty.dmesg: nonexistent directory
>  [0x0->0x20000] pfn
>  [0x0->0x20000] level entry
>  [0x20000->0x100000] identity
>  [0x20000->0x100000] level middle
>  [0x100000->0x120000] pfn
>  [0x100000->0x120000] level entry
>  [0x120000->0x140000] level middle
>  [0x120000->0x7cfffff] missing
>  [0x140000->0x7cfffff] level top
> /init.late: line 17: can't create /mnt/results/p2m/(none)-3.10.0-rc4upstream-00172-g909fe9e-dirty.p2m: nonexistent directory
> libxl: error: libxl.c:87:libxl_ctx_alloc: Is xenstore daemon running?
> failed to stat /var/run/xenstored.pid: No such file or directory
> cannot init xl context
> [    9.669091] device-mapper: ioctl: 4.24.0-ioctl (2013-01-15) initialised: dm-devel@redhat.com
> [    9.670252] device-mapper: multipath: version 1.5.1 loaded
> PING 192.168.101.2 (192.168.101.2) 56(84) bytes of data.
> kill -1 1
> 
> --- 192.168.101.2 ping statistics ---
> 1 packets transmitted, 0 received, 100% packet loss, time 10000ms
> 
> /init.late: line 34: boot_parameter: not found
> [   20.576092] switch: port 1(eth0) entered forwarding state
>  7 Jun 15:21:27 ntpdate[2323]: adjust time server 17.171.4.15 offset -0.265117 sec
> Cannot access the Hardware Clock via any known method.
> Use the --debug option to see the details of our search for an access method.
> Fri Jun  7 15:21:27 UTC 2013
> Jun  7 15:21:27 (none) init: starting pid 2333, tty '/dev/tty0': '/bin/sh'
> Jun  7 15:21:27 (none) init: starting pid 2334, tty '/dev/tty1': '/bin/sh'
> Jun  7 15:21:27 (none) init: starting pid 2335, tty '/dev/ttyS0': '/bin/sh'
> Jun  7 15:21:27 (none) init: starting pid 2336, tty '/dev/hvc0': '/bin/sh'
> 
> 
> BusyBox v1.14.3 (2013-06-07 10:58:39 EDT) built-in shell (ash)
> Enter 'help' for a list of built-in commands.
> 
> # kill -1 1
> # Jun  7 15:21:27 (none) init: reloading /etc/inittab
> 
> # m\b \bdmesg
> [    0.000000] Initializing cgroup subsys cpuset
> [    0.000000] Initializing cgroup subsys cpu
> [    0.000000] Initializing cgroup subsys cpuacct
> [    0.000000] Linux version 3.10.0-rc4upstream-00172-g909fe9e-dirty (konrad@phenom.dumpdata.com) (gcc version 4.4.4 20100503 (Red Hat 4.4.4-2) (GCC) ) #2 SMP Fri Jun 7 10:55:54 EDT 2013
> [    0.000000] Command line: console=hvc0 debug 
> [    0.000000] ACPI in unprivileged domain disabled
> [    0.000000] Freeing 20000-40000 pfn range: 131072 pages freed
> [    0.000000] 1-1 mapping on 20000->100000
> [    0.000000] Released 131072 pages of unused memory
> [    0.000000] Set 917504 page(s) to 1-1 mapping
> [    0.000000] Populating 100000-120000 pfn range: 131072 pages added
> [    0.000000] e820: BIOS-provided physical RAM map:
> [    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
> [    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
> [    0.000000] Xen: [mem 0x0000000000100000-0x000000001fffffff] usable
> [    0.000000] Xen: [mem 0x0000000020000000-0x00000000201fffff] reserved
> [    0.000000] Xen: [mem 0x0000000020200000-0x000000003fffffff] unusable
> [    0.000000] Xen: [mem 0x0000000040000000-0x00000000401fffff] reserved
> [    0.000000] Xen: [mem 0x0000000040200000-0x00000000c6cd3fff] unusable
> [    0.000000] Xen: [mem 0x00000000c6cd4000-0x00000000c6d1cfff] ACPI NVS
> [    0.000000] Xen: [mem 0x00000000c6d1d000-0x00000000c6d27fff] ACPI data
> [    0.000000] Xen: [mem 0x00000000c6d28000-0x00000000c6d28fff] ACPI NVS
> [    0.000000] Xen: [mem 0x00000000c6d29000-0x00000000c6d49fff] reserved
> [    0.000000] Xen: [mem 0x00000000c6d4a000-0x00000000c6d4bfff] unusable
> [    0.000000] Xen: [mem 0x00000000c6d4c000-0x00000000c6d6cfff] ACPI NVS
> [    0.000000] Xen: [mem 0x00000000c6d6d000-0x00000000c6d8ffff] reserved
> [    0.000000] Xen: [mem 0x00000000c6d90000-0x00000000c6d9cfff] ACPI NVS
> [    0.000000] Xen: [mem 0x00000000c6d9d000-0x00000000c6d9ffff] reserved
> [    0.000000] Xen: [mem 0x00000000c6da0000-0x00000000c6db0fff] ACPI NVS
> [    0.000000] Xen: [mem 0x00000000c6db1000-0x00000000c6ddcfff] reserved
> [    0.000000] Xen: [mem 0x00000000c6ddd000-0x00000000c6e1ffff] ACPI NVS
> [    0.000000] Xen: [mem 0x00000000c6e20000-0x00000000c6ffffff] unusable
> [    0.000000] Xen: [mem 0x00000000c7800000-0x00000000cf9fffff] reserved
> [    0.000000] Xen: [mem 0x00000000fed1c000-0x00000000fed3ffff] reserved
> [    0.000000] Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
> [    0.000000] Xen: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
> [    0.000000] Xen: [mem 0x0000000100000000-0x000000011fffffff] usable
> [    0.000000] NX (Execute Disable) protection: active
> [    0.000000] DMI not present or invalid.
> [    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
> [    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
> [    0.000000] No AGP bridge found
> [    0.000000] e820: last_pfn = 0x120000 max_arch_pfn = 0x400000000
> [    0.000000] e820: last_pfn = 0x20000 max_arch_pfn = 0x400000000
> [    0.000000] Scanning 1 areas for low memory corruption
> [    0.000000] Base memory trampoline at [ffff88000009a000] 9a000 size 24576
> [    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
> [    0.000000]  [mem 0x00000000-0x000fffff] page 4k
> [    0.000000] init_memory_mapping: [mem 0x11fe00000-0x11fffffff]
> [    0.000000]  [mem 0x11fe00000-0x11fffffff] page 4k
> [    0.000000] BRK [0x01d8a000, 0x01d8afff] PGTABLE
> [    0.000000] BRK [0x01d8b000, 0x01d8bfff] PGTABLE
> [    0.000000] init_memory_mapping: [mem 0x11c000000-0x11fdfffff]
> [    0.000000]  [mem 0x11c000000-0x11fdfffff] page 4k
> [    0.000000] BRK [0x01d8c000, 0x01d8cfff] PGTABLE
> [    0.000000] BRK [0x01d8d000, 0x01d8dfff] PGTABLE
> [    0.000000] BRK [0x01d8e000, 0x01d8efff] PGTABLE
> [    0.000000] init_memory_mapping: [mem 0x100000000-0x11bffffff]
> [    0.000000]  [mem 0x100000000-0x11bffffff] page 4k
> [    0.000000] init_memory_mapping: [mem 0x00100000-0x1fffffff]
> [    0.000000]  [mem 0x00100000-0x1fffffff] page 4k
> [    0.000000] RAMDISK: [mem 0x02197000-0x13f67fff]
> [    0.000000] NUMA turned off
> [    0.000000] Faking a node at [mem 0x0000000000000000-0x000000011fffffff]
> [    0.000000] Initmem setup node 0 [mem 0x00000000-0x11fffffff]
> [    0.000000]   NODE_DATA [mem 0x11fea2000-0x11fea5fff]
> [    0.000000] Zone ranges:
> [    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
> [    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
> [    0.000000]   Normal   [mem 0x100000000-0x11fffffff]
> [    0.000000] Movable zone start for each node
> [    0.000000] Early memory node ranges
> [    0.000000]   node   0: [mem 0x00001000-0x0009ffff]
> [    0.000000]   node   0: [mem 0x00100000-0x1fffffff]
> [    0.000000]   node   0: [mem 0x100000000-0x11fffffff]
> [    0.000000] On node 0 totalpages: 262047
> [    0.000000]   DMA zone: 56 pages used for memmap
> [    0.000000]   DMA zone: 21 pages reserved
> [    0.000000]   DMA zone: 3999 pages, LIFO batch:0
> [    0.000000]   DMA32 zone: 1736 pages used for memmap
> [    0.000000]   DMA32 zone: 126976 pages, LIFO batch:31
> [    0.000000]   Normal zone: 1792 pages used for memmap
> [    0.000000]   Normal zone: 131072 pages, LIFO batch:31
> [    0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs
> [    0.000000] No local APIC present
> [    0.000000] APIC: disable apic facility
> [    0.000000] APIC: switched to apic NOOP
> [    0.000000] nr_irqs_gsi: 16
> [    0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
> [    0.000000] PM: Registered nosave memory: 0000000020000000 - 0000000020200000
> [    0.000000] PM: Registered nosave memory: 0000000020200000 - 0000000040000000
> [    0.000000] PM: Registered nosave memory: 0000000040000000 - 0000000040200000
> [    0.000000] PM: Registered nosave memory: 0000000040200000 - 00000000c6cd4000
> [    0.000000] PM: Registered nosave memory: 00000000c6cd4000 - 00000000c6d1d000
> [    0.000000] PM: Registered nosave memory: 00000000c6d1d000 - 00000000c6d28000
> [    0.000000] PM: Registered nosave memory: 00000000c6d28000 - 00000000c6d29000
> [    0.000000] PM: Registered nosave memory: 00000000c6d29000 - 00000000c6d4a000
> [    0.000000] PM: Registered nosave memory: 00000000c6d4a000 - 00000000c6d4c000
> [    0.000000] PM: Registered nosave memory: 00000000c6d4c000 - 00000000c6d6d000
> [    0.000000] PM: Registered nosave memory: 00000000c6d6d000 - 00000000c6d90000
> [    0.000000] PM: Registered nosave memory: 00000000c6d90000 - 00000000c6d9d000
> [    0.000000] PM: Registered nosave memory: 00000000c6d9d000 - 00000000c6da0000
> [    0.000000] PM: Registered nosave memory: 00000000c6da0000 - 00000000c6db1000
> [    0.000000] PM: Registered nosave memory: 00000000c6db1000 - 00000000c6ddd000
> [    0.000000] PM: Registered nosave memory: 00000000c6ddd000 - 00000000c6e20000
> [    0.000000] PM: Registered nosave memory: 00000000c6e20000 - 00000000c7000000
> [    0.000000] PM: Registered nosave memory: 00000000c7000000 - 00000000c7800000
> [    0.000000] PM: Registered nosave memory: 00000000c7800000 - 00000000cfa00000
> [    0.000000] PM: Registered nosave memory: 00000000cfa00000 - 00000000fed1c000
> [    0.000000] PM: Registered nosave memory: 00000000fed1c000 - 00000000fed40000
> [    0.000000] PM: Registered nosave memory: 00000000fed40000 - 00000000fee00000
> [    0.000000] PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
> [    0.000000] PM: Registered nosave memory: 00000000fee01000 - 00000000ff000000
> [    0.000000] PM: Registered nosave memory: 00000000ff000000 - 0000000100000000
> [    0.000000] e820: [mem 0xcfa00000-0xfed1bfff] available for PCI devices
> [    0.000000] Booting paravirtualized kernel on Xen
> [    0.000000] Xen version: 4.3-unstable (preserve-AD)
> [    0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:2 nr_node_ids:1
> [    0.000000] PERCPU: Embedded 28 pages/cpu @ffff88011f800000 s85632 r8192 d20864 u1048576
> [    0.000000] pcpu-alloc: s85632 r8192 d20864 u1048576 alloc=1*2097152
> [    0.000000] pcpu-alloc: [0] 0 1 
> [    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 258442
> [    0.000000] Policy zone: Normal
> [    0.000000] Kernel command line: console=hvc0 debug 
> [    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
> [    0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340
> [    0.000000] Checking aperture...
> [    0.000000] No AGP bridge found
> [    0.000000] Memory: 722480k/4718592k available (6698k kernel code, 3670404k absent, 325708k reserved, 4234k data, 1724k init)
> [    0.000000] Hierarchical RCU implementation.
> [    0.000000] 	RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=2.
> [    0.000000] NR_IRQS:33024 nr_irqs:288 16
> [    0.000000] Console: colour dummy device 80x25
> [    0.000000] console [tty0] enabled
> [    0.000000] console [hvc0] enabled
> [    0.000000] Xen: using vcpuop timer interface
> [    0.000000] installing Xen timer for CPU 0
> [    0.000000] tsc: Detected 3092.926 MHz processor
> [    0.001000] Calibrating delay loop (skipped), value calculated using timer frequency.. 6185.85 BogoMIPS (lpj=3092926)
> [    0.001000] pid_max: default: 32768 minimum: 301
> [    0.001000] Security Framework initialized
> [    0.001000] SELinux:  Initializing.
> [    0.001000] SELinux:  Starting in permissive mode
> [    0.001000] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
> [    0.001000] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
> [    0.001000] Mount-cache hash table entries: 256
> [    0.001582] Initializing cgroup subsys freezer
> [    0.001698] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
> [    0.001698] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
> [    0.001708] CPU: Physical Processor ID: 0
> [    0.001712] CPU: Processor Core ID: 0
> [    0.001718] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0
> [    0.001718] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32
> [    0.001718] tlb_flushall_shift: 5
> [    0.025468] cpu 0 spinlock event irq 17
> [    0.025566] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only.
> [    0.026081] NMI watchdog: disabled (cpu0): hardware events not enabled
> [    0.026680] installing Xen timer for CPU 1
> [    0.026723] cpu 1 spinlock event irq 24
> [    0.026793] SMP alternatives: switching to SMP code
> [    0.048014] Brought up 2 CPUs
> [    0.051334] PM: Registering ACPI NVS region [mem 0xc6cd4000-0xc6d1cfff] (299008 bytes)
> [    0.051334] PM: Registering ACPI NVS region [mem 0xc6d28000-0xc6d28fff] (4096 bytes)
> [    0.051334] PM: Registering ACPI NVS region [mem 0xc6d4c000-0xc6d6cfff] (135168 bytes)
> [    0.051334] PM: Registering ACPI NVS region [mem 0xc6d90000-0xc6d9cfff] (53248 bytes)
> [    0.051334] PM: Registering ACPI NVS region [mem 0xc6da0000-0xc6db0fff] (69632 bytes)
> [    0.051334] PM: Registering ACPI NVS region [mem 0xc6ddd000-0xc6e1ffff] (274432 bytes)
> [    0.051399] kworker/u4:0 (19) used greatest stack depth: 6016 bytes left
> [    0.052492] Grant tables using version 2 layout.
> [    0.052492] Grant table initialized
> [    0.071696] RTC time: 165:165:165, date: 165/165/65
> [    0.071985] NET: Registered protocol family 16
> [    0.072173] kworker/u4:0 (22) used greatest stack depth: 5928 bytes left
> [    0.073110] kworker/u4:0 (30) used greatest stack depth: 5464 bytes left
> [    0.074030] dca service started, version 1.12.1
> [    0.074563] PCI: setting up Xen PCI frontend stub
> [    0.074568] PCI: pci_cache_line_size set to 64 bytes
> [    0.101148] bio: create slab <bio-0> at 0
> [    0.101220] ACPI: Interpreter disabled.
> [    0.102003] xen/balloon: Initialising balloon driver.
> [    0.103081] xen-balloon: Initialising balloon driver.
> [    0.104136] vgaarb: loaded
> [    0.105135] usbcore: registered new interface driver usbfs
> [    0.105135] usbcore: registered new interface driver hub
> [    0.105135] usbcore: registered new device driver usb
> [    0.106089] pps_core: LinuxPPS API ver. 1 registered
> [    0.106089] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
> [    0.106089] PTP clock support registered
> [    0.106089] PCI: System does not support PCI
> [    0.106089] PCI: System does not support PCI
> [    0.107154] NetLabel: Initializing
> [    0.107154] NetLabel:  domain hash size = 128
> [    0.107154] NetLabel:  protocols = UNLABELED CIPSOv4
> [    0.107175] NetLabel:  unlabeled traffic allowed by default
> [    0.108087] Switching to clocksource xen
> [    0.125349] pnp: PnP ACPI: disabled
> [    0.138965] NET: Registered protocol family 2
> [    0.139578] TCP established hash table entries: 8192 (order: 5, 131072 bytes)
> [    0.139647] TCP bind hash table entries: 8192 (order: 5, 131072 bytes)
> [    0.139668] TCP: Hash tables configured (established 8192 bind 8192)
> [    0.193672] TCP: reno registered
> [    0.193693] UDP hash table entries: 512 (order: 2, 16384 bytes)
> [    0.193712] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
> [    0.194030] NET: Registered protocol family 1
> [    0.194380] RPC: Registered named UNIX socket transport module.
> [    0.194389] RPC: Registered udp transport module.
> [    0.194394] RPC: Registered tcp transport module.
> [    0.194398] RPC: Registered tcp NFSv4.1 backchannel transport module.
> [    0.194404] PCI: CLS 0 bytes, default 64
> [    0.194570] Unpacking initramfs...
> [    0.778545] Freeing initrd memory: 292676k freed
> [    0.823501] platform rtc_cmos: registered platform RTC device (no PNP device found)
> [    0.823855] Machine check injector initialized
> [    0.825040] Scanning for low memory corruption every 60 seconds
> [    0.826072] audit: initializing netlink socket (disabled)
> [    0.826114] type=2000 audit(1370618465.698:1): initialized
> [    0.839204] HugeTLB registered 2 MB page size, pre-allocated 0 pages
> [    0.839673] VFS: Disk quotas dquot_6.5.2
> [    0.839760] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
> [    0.840528] NFS: Registering the id_resolver key type
> [    0.840553] Key type id_resolver registered
> [    0.840558] Key type id_legacy registered
> [    0.840587] NTFS driver 2.1.30 [Flags: R/W].
> [    0.840919] msgmni has been set to 1982
> [    0.841082] SELinux:  Registering netfilter hooks
> [    0.842697] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
> [    0.842706] io scheduler noop registered
> [    0.842710] io scheduler deadline registered
> [    0.842772] io scheduler cfq registered (default)
> [    0.843309] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
> [    0.846284] intel_idle: does not run on family 6 model 42
> [    0.846299] ioatdma: Intel(R) QuickData Technology Driver 4.00
> [    0.847400] pcifront pci-0: Installing PCI frontend
> [    0.847417] Warning: only able to allocate 4 MB for software IO TLB
> [    0.850807] software IO TLB [mem 0x109000000-0x109400000] (4MB) mapped at [ffff880109000000-ffff8801093fffff]
> [    0.851175] pcifront pci-0: Creating PCI Frontend Bus 0000:00
> [    0.851458] pcifront pci-0: PCI host bridge to bus 0000:00
> [    0.851469] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
> [    0.851478] pci_bus 0000:00: root bus resource [mem 0x00000000-0xfffffffff]
> [    0.851487] pci_bus 0000:00: root bus resource [bus 00]
> [    0.852031] pci 0000:00:00.0: [8086:105e] type 00 class 0x020000
> [    0.852419] pci 0000:00:00.0: reg 0x10: [mem 0xfe4a0000-0xfe4bffff]
> [    0.852631] pci 0000:00:00.0: reg 0x14: [mem 0xfe480000-0xfe49ffff]
> [    0.852743] pci 0000:00:00.0: reg 0x18: [io  0xe020-0xe03f]
> [    0.853905] pcifront pci-0: New device on 0000:00:00.0 found.
> [    0.857483] pcifront pci-0: claiming resource 0000:00:00.0/0
> [    0.857490] pcifront pci-0: claiming resource 0000:00:00.0/1
> [    0.857495] pcifront pci-0: claiming resource 0000:00:00.0/2
> [    0.939621] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
> [    0.942227] Non-volatile memory driver v1.3
> [    0.942414] Linux agpgart interface v0.103
> [    0.943295] [drm] Initialized drm 1.1.0 20060810
> [    0.947952] loop: module loaded
> [    0.948577] libphy: Fixed MDIO Bus: probed
> [    0.948583] tun: Universal TUN/TAP device driver, 1.6
> [    0.948588] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
> [    0.948913] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.7.12-k
> [    0.948921] ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
> [    0.950031] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
> [    0.950039] ehci_hcd: block sizes: qh 112 qtd 96 itd 192 sitd 96
> [    0.950061] ehci-pci: EHCI PCI platform driver
> [    0.950201] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
> [    0.950207] ohci_hcd: block sizes: ed 80 td 96
> [    0.950357] uhci_hcd: USB Universal Host Controller Interface driver
> [    0.950736] usbcore: registered new interface driver usblp
> [    0.951285] i8042: PNP: No PS/2 controller found. Probing ports directly.
> [    1.960646] i8042: No controller found
> [    1.960872] mousedev: PS/2 mouse device common for all mice
> [    2.021775] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
> [    2.021941] rtc_cmos: probe of rtc_cmos failed with error -38
> [    2.023233] zram: Created 1 device(s) ...
> [    2.023567] Netfilter messages via NETLINK v0.30.
> [    2.023612] nf_conntrack version 0.5.0 (7930 buckets, 31720 max)
> [    2.023775] ctnetlink v0.93: registering with nfnetlink.
> [    2.024045] ip_tables: (C) 2000-2006 Netfilter Core Team
> [    2.024170] TCP: cubic registered
> [    2.024177] Initializing XFRM netlink socket
> [    2.024379] NET: Registered protocol family 10
> [    2.024936] ip6_tables: (C) 2000-2006 Netfilter Core Team
> [    2.025259] sit: IPv6 over IPv4 tunneling driver
> [    2.026039] NET: Registered protocol family 17
> [    2.026320] Key type dns_resolver registered
> [    2.026874] PM: Hibernation image not present or could not be loaded.
> [    2.026922] registered taskstats version 1
> [    2.027005] kmemleak: Kernel memory leak detector initialized
> [    2.027009] kmemleak: Automatic memory scanning thread started
> [    2.027437] XENBUS: Device with no driver: device/vbd/51712
> [    2.027537]   Magic number: 1:252:3141
> [    2.029169] Freeing unused kernel memory: 1724k freed
> [    2.029408] Write protecting the kernel read-only data: 10240k
> [    2.033649] Freeing unused kernel memory: 1480k freed
> [    2.033908] Freeing unused kernel memory: 72k freed
> [    2.040621] consoletype (1038) used greatest stack depth: 5304 bytes left
> [    2.059689] chmod (1044) used greatest stack depth: 4856 bytes left
> [    2.266374] Initialising Xen virtual ethernet driver.
> [    2.273190] vbd vbd-51712: blkfront:blkback_changed to state 2.
> [    2.283349] vbd vbd-51712: blkfront:blkback_changed to state 4.
> [    2.283356] vbd vbd-51712: blkfront_connect:/local/domain/0/backend/vbd/2/51712.
> [    2.287707] blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled;
> [    2.288337] Entered do_blkif_request
> [    2.288344] Entered do_blkif_request
> [    2.288821] Entered do_blkif_request
> [    2.288829] do_blk_req ffff880108ec9e10: cmd ffff880108ec9f00, sec 0, (4/4) buffer:ffff880108ee6000 [read]
> [    2.288839] Entered do_blkif_request
> [    2.288859] Entered do_blkif_request
> [    2.288864] do_blk_req ffff880108ec9cb8: cmd ffff880108ec9da8, sec 4, (4/4) buffer:ffff880108ee6800 [read]
> [    2.288871] Entered do_blkif_request
> [    2.292721] Entered do_blkif_request
> [    2.292727] Entered do_blkif_request
> [    2.292737] Entered do_blkif_request
> [    2.292741] Entered do_blkif_request
> [    2.292780] Entered do_blkif_request
> [    2.292786] do_blk_req ffff880108ec9cb8: cmd ffff880108ec9da8, sec 8, (4/4) buffer:ffff88010f273000 [read]
> [    2.292794] Entered do_blkif_request
> [    2.292811] Entered do_blkif_request
> [    2.292816] do_blk_req ffff880108ec9e10: cmd ffff880108ec9f00, sec c, (4/4) buffer:ffff88010f273800 [read]
> [    2.292822] Entered do_blkif_request
> [    2.292940] Entered do_blkif_request
> [    2.292947] Entered do_blkif_request
> [    2.292957] Entered do_blkif_request
> [    2.292961] Entered do_blkif_request
> [    2.292974]  xvda: unknown partition table
> [    2.385804] udevd (1099): /proc/1099/oom_adj is deprecated, please use /proc/1099/oom_score_adj instead.
> [    2.444038] e1000e: Intel(R) PRO/1000 Network Driver - 2.3.2-k
> [    2.444061] e1000e: Copyright(c) 1999 - 2013 Intel Corporation.
> [    2.444110] e1000e 0000:00:00.0: Disabling ASPM  L1
> [    2.444232] e1000e 0000:00:00.0: enabling device (0000 -> 0002)
> [    2.447244] e1000e 0000:00:00.0: Xen PCI mapped GSI16 to IRQ34
> [    2.447790] e1000e 0000:00:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
> [    2.459644] vbd vbd-51712: command: 0x5331, argument: 0x0
> [    2.560213] ip (1325) used greatest stack depth: 3760 bytes left
> [    2.624581] e1000e 0000:00:00.0 eth0: (PCI Express:2.5GT/s:Width x4) 00:15:17:8f:18:a2
> [    2.624595] e1000e 0000:00:00.0 eth0: Intel(R) PRO/1000 Network Connection
> [    2.624675] e1000e 0000:00:00.0 eth0: MAC: 0, PHY: 4, PBA No: D50868-003
> [    2.814571] [drm] radeon kernel modesetting enabled.
> [    3.394987] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
> [    3.397647] device eth0 entered promiscuous mode
> [    3.452997] IPv6: ADDRCONF(NETDEV_UP): switch: link is not ready
> [    5.533965] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> [    5.534117] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [    5.534222] switch: port 1(eth0) entered forwarding state
> [    5.534239] switch: port 1(eth0) entered forwarding state
> [    5.534302] IPv6: ADDRCONF(NETDEV_CHANGE): switch: link becomes ready
> [    9.129856] SCSI subsystem initialized
> [    9.131318] Loading iSCSI transport class v2.0-870.
> [    9.134894] iscsi: registered transport (tcp)
> [    9.176036] Event-channel device installed.
> [    9.336320] mount.nfs (2304) used greatest stack depth: 3320 bytes left
> [    9.669091] device-mapper: ioctl: 4.24.0-ioctl (2013-01-15) initialised: dm-devel@redhat.com
> [    9.670252] device-mapper: multipath: version 1.5.1 loaded
> [   20.576092] switch: port 1(eth0) entered forwarding state
> # lspci
> 00:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
> # ifconfig
> eth0      Link encap:Ethernet  HWaddr 00:15:17:8F:18:A2  
>           inet6 addr: fe80::215:17ff:fe8f:18a2/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:116 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:141 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000 
>           RX bytes:12017 (11.7 KiB)  TX bytes:13404 (13.0 KiB)
>           Interrupt:34 Memory:fe4a0000-fe4c0000 
> 
> lo        Link encap:Local Loopback  
>           inet addr:127.0.0.1  Mask:255.0.0.0
>           inet6 addr: ::1/128 Scope:Host
>           UP LOOPBACK RUNNING  MTU:65536  Metric:1
>           RX packets:8 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0 
>           RX bytes:520 (520.0 b)  TX bytes:520 (520.0 b)
> 
> switch    Link encap:Ethernet  HWaddr 00:15:17:8F:18:A2  
>           inet addr:192.168.102.210  Bcast:192.168.102.255  Mask:255.255.255.0
>           inet6 addr: fe80::215:17ff:fe8f:18a2/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:116 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:134 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0 
>           RX bytes:9929 (9.6 KiB)  TX bytes:12244 (11.9 KiB)
> 
> # eth\b\b\b
> # ethtool ^[[J-i eth0
> driver: e1000e
> version: 2.3.2-k
> firmware-version: 5.11-2
> bus-info: 0000:00:00.0
> # rmmod e1000e
> [   39.032406] switch: port 1(eth0) entered disabled state
> [   39.032576] device eth0 left promiscuous mode
> [   39.032583] switch: port 1(eth0) entered disabled state
> # miod\b \b\b \b\b \bodprobe e1000e
> [   42.614015] e1000e: Intel(R) PRO/1000 Network Driver - 2.3.2-k
> [   42.614026] e1000e: Copyright(c) 1999 - 2013 Intel Corporation.
> [   42.614080] e1000e 0000:00:00.0: Disabling ASPM  L1
> [   42.614172] xen_map_pirq_gsi: returning irq 34 for gsi 16
> [   42.614181] e1000e 0000:00:00.0: Xen PCI mapped GSI16 to IRQ34
> [   42.615024] e1000e 0000:00:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
> [   42.790351] e1000e 0000:00:00.0 eth0: (PCI Express:2.5GT/s:Width x4) 00:15:17:8f:18:a2
> [   42.790363] e1000e 0000:00:00.0 eth0: Intel(R) PRO/1000 Network Connection
> [   42.790448] e1000e 0000:00:00.0 eth0: MAC: 0, PHY: 4, PBA No: D50868-003
> # [   43.103936] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
> [   43.106574] device eth0 entered promiscuous mode
> rmmod e100[   45.190968] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> [   45.191117] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [   45.191210] switch: port 1(eth0) entered forwarding state
> [   45.191227] switch: port 1(eth0) entered forwarding state
> 0e
> [   46.621376] switch: port 1(eth0) entered disabled state
> [   46.621538] device eth0 left promiscuous mode
> [   46.621547] switch: port 1(eth0) entered disabled state
> # 
> 
> 
> 
> DETACHING HERE..
> 
> # 
> # [   53.802739] pcifront pci-0: Rescanning PCI Frontend Bus 0000:00
> [   53.887894] pci_bus 0000:00: busn_res: [bus 00] is released
> [   53.888042] ------------[ cut here ]------------
> [   53.888055] WARNING: at /home/konrad/linux/include/linux/kref.h:47 klist_iter_init_node+0x3e/0x50()
> [   53.888073] Modules linked in: dm_multipath dm_mod xen_evtchn iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi scsi_mod libcrc32c crc32c radeon fbcon tileblit font bitblit softcursor ttm drm_kms_helper crc32c_intel xen_blkfront xen_netfront xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xenfs xen_privcmd [last unloaded: e1000e]
> [   53.888151] CPU: 1 PID: 26 Comm: xenwatch Not tainted 3.10.0-rc4upstream-00172-g909fe9e-dirty #2
> [   53.888158]  ffffffff819392b8 ffff88011e3c1b88 ffffffff8167a27d ffff88011e3c1bc8
> [   53.888171]  ffffffff8108d1eb ffff880109536a00 ffff88011e3c1c18 0000000000000000
> [   53.888186]  ffffffff8131c410 0000000000000000 ffff88011e3c1e70 ffff88011e3c1bd8
> [   53.888197] Call Trace:
> [   53.888203]  [<ffffffff8167a27d>] dump_stack+0x19/0x1b
> [   53.888212]  [<ffffffff8108d1eb>] warn_slowpath_common+0x6b/0xa0
> [   53.888220]  [<ffffffff8131c410>] ? pci_match_next_bus+0x10/0x10
> [   53.888227]  [<ffffffff8108d235>] warn_slowpath_null+0x15/0x20
> [   53.888234]  [<ffffffff81663b4e>] klist_iter_init_node+0x3e/0x50
> [   53.888242]  [<ffffffff8142117d>] class_dev_iter_init+0x3d/0x50
> [   53.888248]  [<ffffffff81421348>] class_find_device+0x38/0xb0
> [   53.888255]  [<ffffffff8131c8aa>] pci_get_next_root_bus+0x4a/0x80
> [   53.888262]  [<ffffffff8132f6a5>] pcifront_free_roots+0x25/0x60
> [   53.888268]  [<ffffffff8132f6f1>] free_pdev+0x11/0x80
> [   53.888274]  [<ffffffff8132f77a>] pcifront_xenbus_remove+0x1a/0x20
> [   53.888283]  [<ffffffff813abae8>] xenbus_dev_remove+0x38/0x70
> [   53.888289]  [<ffffffff8141ffc1>] __device_release_driver+0x61/0xd0
> [   53.888296]  [<ffffffff81420148>] device_release_driver+0x28/0x40
> [   53.888302]  [<ffffffff8141f296>] bus_remove_device+0x106/0x140
> [   53.888310]  [<ffffffff8141d120>] device_del+0x110/0x1c0
> [   53.888316]  [<ffffffff8141d1e1>] device_unregister+0x11/0x20
> [   53.888323]  [<ffffffff813ab946>] xenbus_dev_changed+0x96/0x1d0
> [   53.888331]  [<ffffffff81048bd6>] ? xen_spin_lock+0xa6/0x110
> [   53.888338]  [<ffffffff813ad2f6>] frontend_changed+0x16/0x20
> [   53.888345]  [<ffffffff813a9c5b>] xenwatch_thread+0xcb/0x190
> [   53.888352]  [<ffffffff810b49d0>] ? wake_up_bit+0x40/0x40
> [   53.888359]  [<ffffffff813a9b90>] ? xs_watch+0x60/0x60
> [   53.888365]  [<ffffffff810b42e6>] kthread+0xc6/0xd0
> [   53.888371]  [<ffffffff8103a149>] ? xen_end_context_switch+0x19/0x20
> [   53.888378]  [<ffffffff810b4220>] ? kthread_freezable_should_stop+0x80/0x80
> [   53.888386]  [<ffffffff8168623c>] ret_from_fork+0x7c/0xb0
> [   53.888392]  [<ffffffff810b4220>] ? kthread_freezable_should_stop+0x80/0x80
> [   53.888397] ---[ end trace 918a23c9bada3aed ]---
> [   53.888402] ------------[ cut here ]------------
> [   53.888407] WARNING: at /home/konrad/linux/lib/klist.c:189 klist_release+0x112/0x120()
> [   53.888412] Modules linked in: dm_multipath dm_mod xen_evtchn iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi scsi_mod libcrc32c crc32c radeon fbcon tileblit font bitblit softcursor ttm drm_kms_helper crc32c_intel xen_blkfront xen_netfront xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xenfs xen_privcmd [last unloaded: e1000e]
> [   53.888478] CPU: 1 PID: 26 Comm: xenwatch Tainted: G        W    3.10.0-rc4upstream-00172-g909fe9e-dirty #2
> [   53.888484]  ffffffff819a3648 ffff88011e3c1b18 ffffffff8167a27d ffff88011e3c1b58
> [   53.888495]  ffffffff8108d1eb 0000000000000010 ffff880109533aa8 ffff880109533a90
> [   53.888506]  ffffffff81421460 0000000000000000 dead000000100100 ffff88011e3c1b68
> [   53.888517] Call Trace:
> [   53.888523]  [<ffffffff8167a27d>] dump_stack+0x19/0x1b
> [   53.888529]  [<ffffffff8108d1eb>] warn_slowpath_common+0x6b/0xa0
> [   53.888536]  [<ffffffff81421460>] ? class_for_each_device+0xa0/0xa0
> [   53.888543]  [<ffffffff8108d235>] warn_slowpath_null+0x15/0x20
> [   53.888549]  [<ffffffff81663c82>] klist_release+0x112/0x120
> [   53.888556]  [<ffffffff81421460>] ? class_for_each_device+0xa0/0xa0
> [   53.888563]  [<ffffffff81663cb8>] klist_dec_and_del+0x28/0x30
> [   53.888601]  [<ffffffff81663eb5>] klist_next+0x45/0x140
> [   53.888608]  [<ffffffff8131c410>] ? pci_match_next_bus+0x10/0x10
> [   53.888615]  [<ffffffff81421108>] class_dev_iter_next+0x18/0x50
> [   53.888621]  [<ffffffff81421358>] class_find_device+0x48/0xb0
> [   53.888628]  [<ffffffff8131c8aa>] pci_get_next_root_bus+0x4a/0x80
> [   53.888634]  [<ffffffff8132f6a5>] pcifront_free_roots+0x25/0x60
> [   53.888640]  [<ffffffff8132f6f1>] free_pdev+0x11/0x80
> [   53.888646]  [<ffffffff8132f77a>] pcifront_xenbus_remove+0x1a/0x20
> [   53.888653]  [<ffffffff813abae8>] xenbus_dev_remove+0x38/0x70
> [   53.888660]  [<ffffffff8141ffc1>] __device_release_driver+0x61/0xd0
> [   53.888667]  [<ffffffff81420148>] device_release_driver+0x28/0x40
> [   53.888673]  [<ffffffff8141f296>] bus_remove_device+0x106/0x140
> [   53.888680]  [<ffffffff8141d120>] device_del+0x110/0x1c0
> [   53.888688]  [<ffffffff8141d1e1>] device_unregister+0x11/0x20
> [   53.888694]  [<ffffffff813ab946>] xenbus_dev_changed+0x96/0x1d0
> [   53.888701]  [<ffffffff81048bd6>] ? xen_spin_lock+0xa6/0x110
> [   53.888708]  [<ffffffff813ad2f6>] frontend_changed+0x16/0x20
> [   53.888714]  [<ffffffff813a9c5b>] xenwatch_thread+0xcb/0x190
> [   53.888721]  [<ffffffff810b49d0>] ? wake_up_bit+0x40/0x40
> [   53.888727]  [<ffffffff813a9b90>] ? xs_watch+0x60/0x60
> [   53.888733]  [<ffffffff810b42e6>] kthread+0xc6/0xd0
> [   53.888739]  [<ffffffff8103a149>] ? xen_end_context_switch+0x19/0x20
> [   53.888746]  [<ffffffff810b4220>] ? kthread_freezable_should_stop+0x80/0x80
> [   53.888755]  [<ffffffff8168623c>] ret_from_fork+0x7c/0xb0
> [   53.888761]  [<ffffffff810b4220>] ? kthread_freezable_should_stop+0x80/0x80
> [   53.888766] ---[ end trace 918a23c9bada3aee ]---
> [   53.888776] general protection fault: 0000 [#1] SMP 
> [   53.888783] Modules linked in: dm_multipath dm_mod xen_evtchn iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi scsi_mod libcrc32c crc32c radeon fbcon tileblit font bitblit softcursor ttm drm_kms_helper crc32c_intel xen_blkfront xen_netfront xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xenfs xen_privcmd [last unloaded: e1000e]
> [   53.888847] CPU: 1 PID: 26 Comm: xenwatch Tainted: G        W    3.10.0-rc4upstream-00172-g909fe9e-dirty #2
> [   53.888853] task: ffff88011e2eb800 ti: ffff88011e3c0000 task.ti: ffff88011e3c0000
> [   53.888859] RIP: e030:[<ffffffff81663baa>]  [<ffffffff81663baa>] klist_release+0x3a/0x120
> [   53.888867] RSP: e02b:ffff88011e3c1b78  EFLAGS: 00010292
> [   53.888872] RAX: dead000000200200 RBX: ffff880109533aa8 RCX: 0000006c3739ee67
> [   53.888877] RDX: dead000000100100 RSI: dead000000200200 RDI: dead000000100100
> [   53.888882] RBP: ffff88011e3c1b98 R08: 00000000a589f9bc R09: 0720072007200720
> [   53.888887] R10: 0720072007200720 R11: 0720072007200720 R12: ffff880109533a90
> [   53.888892] R13: ffffffff81421460 R14: 0000000000000000 R15: dead000000100100
> [   53.888900] FS:  00007fe6ef1d97a0(0000) GS:ffff88011f900000(0000) knlGS:0000000000000000
> [   53.888907] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   53.888912] CR2: 00007f1c4090ddd0 CR3: 000000010f0cd000 CR4: 0000000000042660
> [   53.888917] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [   53.888923] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [   53.888928] Stack:
> [   53.888932]  ffff880109533a90 ffff88011e3c1c18 ffff880109533a90 ffffffff81421460
> [   53.888943]  ffff88011e3c1ba8 ffffffff81663cb8 ffff88011e3c1be8 ffffffff81663eb5
> [   53.888954]  0000000000000000 ffff88011e3c1c18 0000000000000000 ffffffff8131c410
> [   53.888965] Call Trace:
> [   53.888971]  [<ffffffff81421460>] ? class_for_each_device+0xa0/0xa0
> [   53.888978]  [<ffffffff81663cb8>] klist_dec_and_del+0x28/0x30
> [   53.888985]  [<ffffffff81663eb5>] klist_next+0x45/0x140
> [   53.888992]  [<ffffffff8131c410>] ? pci_match_next_bus+0x10/0x10
> [   53.888999]  [<ffffffff81421108>] class_dev_iter_next+0x18/0x50
> [   53.889006]  [<ffffffff81421358>] class_find_device+0x48/0xb0
> [   53.889012]  [<ffffffff8131c8aa>] pci_get_next_root_bus+0x4a/0x80
> [   53.889020]  [<ffffffff8132f6a5>] pcifront_free_roots+0x25/0x60
> [   53.889026]  [<ffffffff8132f6f1>] free_pdev+0x11/0x80
> [   53.889032]  [<ffffffff8132f77a>] pcifront_xenbus_remove+0x1a/0x20
> [   53.889039]  [<ffffffff813abae8>] xenbus_dev_remove+0x38/0x70
> [   53.889046]  [<ffffffff8141ffc1>] __device_release_driver+0x61/0xd0
> [   53.889054]  [<ffffffff81420148>] device_release_driver+0x28/0x40
> [   53.889062]  [<ffffffff8141f296>] bus_remove_device+0x106/0x140
> [   53.889064]  [<ffffffff8141d120>] device_del+0x110/0x1c0
> [   53.889064]  [<ffffffff8141d1e1>] device_unregister+0x11/0x20
> [   53.889064]  [<ffffffff813ab946>] xenbus_dev_changed+0x96/0x1d0
> [   53.889064]  [<ffffffff81048bd6>] ? xen_spin_lock+0xa6/0x110
> [   53.889064]  [<ffffffff813ad2f6>] frontend_changed+0x16/0x20
> [   53.889064]  [<ffffffff813a9c5b>] xenwatch_thread+0xcb/0x190
> [   53.889064]  [<ffffffff810b49d0>] ? wake_up_bit+0x40/0x40
> [   53.889064]  [<ffffffff813a9b90>] ? xs_watch+0x60/0x60
> [   53.889064]  [<ffffffff810b42e6>] kthread+0xc6/0xd0
> [   53.889064]  [<ffffffff8103a149>] ? xen_end_context_switch+0x19/0x20
> [   53.889064]  [<ffffffff810b4220>] ? kthread_freezable_should_stop+0x80/0x80
> [   53.889064]  [<ffffffff8168623c>] ret_from_fork+0x7c/0xb0
> [   53.889064]  [<ffffffff810b4220>] ? kthread_freezable_should_stop+0x80/0x80
> [   53.889064] Code: fb 48 83 ec 08 f6 47 e8 01 0f 84 e3 00 00 00 48 8b 43 f8 48 8b 53 f0 48 bf 00 01 10 00 00 00 ad de 48 be 00 02 20 00 00 00 ad de <48> 89 42 08 48 89 10 48 89 7b f0 48 89 73 f8 48 c7 c7 54 44 d7 
> [   53.889064] RIP  [<ffffffff81663baa>] klist_release+0x3a/0x120
> [   53.889064]  RSP <ffff88011e3c1b78>
> [   53.889285] ---[ end trace 918a23c9bada3aef ]---
> [   53.889290] BUG: sleeping function called from invalid context at /home/konrad/linux/kernel/rwsem.c:20
> [   53.889296] in_atomic(): 1, irqs_disabled(): 0, pid: 26, name: xenwatch
> [   53.889302] CPU: 1 PID: 26 Comm: xenwatch Tainted: G      D W    3.10.0-rc4upstream-00172-g909fe9e-dirty #2
> [   53.889307]  000000000000000b ffff88011e3c1928 ffffffff8167a27d ffff88011e3c1938
> [   53.889319]  ffffffff810bf5a8 ffff88011e3c1958 ffffffff8167b56f 0720072007200720
> [   53.889331]  ffff88011e2eb800 ffff88011e3c1998 ffffffff810a279a 0000000000000000
> [   53.889342] Call Trace:
> [   53.889347]  [<ffffffff8167a27d>] dump_stack+0x19/0x1b
> [   53.889354]  [<ffffffff810bf5a8>] __might_sleep+0xd8/0x100
> [   53.889360]  [<ffffffff8167b56f>] down_read+0x1f/0x40
> [   53.889368]  [<ffffffff810a279a>] exit_signals+0x2a/0x170
> [   53.889374]  [<ffffffff81091ddf>] do_exit+0xaf/0xbe0
> [   53.889380]  [<ffffffff8167a12e>] ? printk+0x48/0x4a
> [   53.889387]  [<ffffffff810423f2>] ? check_events+0x12/0x20
> [   53.889394]  [<ffffffff8167f220>] oops_end+0xb0/0xf0
> [   53.889401]  [<ffffffff8104da06>] die+0x56/0x90
> [   53.889407]  [<ffffffff8167effc>] do_general_protection+0xdc/0x160
> [   53.889414]  [<ffffffff81421460>] ? class_for_each_device+0xa0/0xa0
> [   53.889420]  [<ffffffff8167e668>] general_protection+0x28/0x30
> [   53.889427]  [<ffffffff81421460>] ? class_for_each_device+0xa0/0xa0
> [   53.889434]  [<ffffffff81663baa>] ? klist_release+0x3a/0x120
> [   53.889441]  [<ffffffff81663c82>] ? klist_release+0x112/0x120
> [   53.889447]  [<ffffffff81421460>] ? class_for_each_device+0xa0/0xa0
> [   53.889454]  [<ffffffff81663cb8>] klist_dec_and_del+0x28/0x30
> [   53.889461]  [<ffffffff81663eb5>] klist_next+0x45/0x140
> [   53.889467]  [<ffffffff8131c410>] ? pci_match_next_bus+0x10/0x10
> [   53.889473]  [<ffffffff81421108>] class_dev_iter_next+0x18/0x50
> [   53.889480]  [<ffffffff81421358>] class_find_device+0x48/0xb0
> [   53.889486]  [<ffffffff8131c8aa>] pci_get_next_root_bus+0x4a/0x80
> [   53.889493]  [<ffffffff8132f6a5>] pcifront_free_roots+0x25/0x60
> [   54.002497]  [<ffffffff8132f6f1>] free_pdev+0x11/0x80
> [   54.002505]  [<ffffffff8132f77a>] pcifront_xenbus_remove+0x1a/0x20
> [   54.002516]  [<ffffffff813abae8>] xenbus_dev_remove+0x38/0x70
> [   54.002524]  [<ffffffff8141ffc1>] __device_release_driver+0x61/0xd0
> [   54.002530]  [<ffffffff81420148>] device_release_driver+0x28/0x40
> [   54.002537]  [<ffffffff8141f296>] bus_remove_device+0x106/0x140
> [   54.002544]  [<ffffffff8141d120>] device_del+0x110/0x1c0
> [   54.002551]  [<ffffffff8141d1e1>] device_unregister+0x11/0x20
> [   54.002558]  [<ffffffff813ab946>] xenbus_dev_changed+0x96/0x1d0
> [   54.002567]  [<ffffffff81048bd6>] ? xen_spin_lock+0xa6/0x110
> [   54.002574]  [<ffffffff813ad2f6>] frontend_changed+0x16/0x20
> [   54.002581]  [<ffffffff813a9c5b>] xenwatch_thread+0xcb/0x190
> [   54.002588]  [<ffffffff810b49d0>] ? wake_up_bit+0x40/0x40
> [   54.002595]  [<ffffffff813a9b90>] ? xs_watch+0x60/0x60
> [   54.002601]  [<ffffffff810b42e6>] kthread+0xc6/0xd0
> [   54.002607]  [<ffffffff8103a149>] ? xen_end_context_switch+0x19/0x20
> [   54.002615]  [<ffffffff810b4220>] ? kthread_freezable_should_stop+0x80/0x80
> [   54.002624]  [<ffffffff8168623c>] ret_from_fork+0x7c/0xb0
> [   54.002630]  [<ffffffff810b4220>] ? kthread_freezable_should_stop+0x80/0x80
> [   54.002637] note: xenwatch[26] exited with preempt_count 1
> 
> # 
> # 
> 

Re: [RFC PATCH v2, part3 07/11] PCI, xen-pcifront: use new PCI interfaces to simplify implementation

[Konrad Rzeszutek Wilk]

On Sat, Jun 08, 2013 at 12:50:31AM +0800, Jiang Liu wrote:
> On 06/07/2013 11:38 PM, Konrad Rzeszutek Wilk wrote:
> > On Fri, Jun 07, 2013 at 10:50:24AM -0400, Konrad Rzeszutek Wilk wrote:
> >> On Thu, May 16, 2013 at 11:50:55PM +0800, Jiang Liu wrote:
> >>> Use new PCI interfaces to simplify xen-pcifront implementation:
> >>> 1) Use pci_create_root_bus() instead of pci_scan_bus_parented()
> >>>    because pci_scan_bus_parented() is marked as __deprecated.This
> >>>    also gets rid of a duplicated call of pci_bus_start_devices().
> >>> 2) Use pci_stop_root_bus() and pci_remove_root_bus() instead of
> >>>    open-coded private implementation.
> >>> 3) Use pci_set_host_bridge_release() to release data structures
> >>>    associated with PCI root buses.
> >>> 4) Use pci_bus_get()/pci_bus_put() to manage PCI root bus reference
> >>>    count.
> >>>
> >>> This is also a preparation for coming PCI bus lock enhancement.
> > 
> > With this patch from :
> > 
> >  Merge branch 'pci_lock_v3' of https://github.com/jiangliu/linux into testing
> > 
> > 
> > it blows up when detaching the device.
> Hi Konrad,
>    Thanks for testing! According to the log messages, this issue should
> be related to pci bus reference counter management. Seems we have done
> an extra(unbalanced) release of pci bus device.
>    Will investigate it tomorrow!

That is quite commendable that you are willing to look over this on
the weekend but I am not going to be able to rerun this test until
some time in the week. You could enjoy the weekend and just look at
this during the week.

Re: [RFC PATCH v2, part3 07/11] PCI, xen-pcifront: use new PCI interfaces to simplify implementation

[Jiang Liu]

On Sat 08 Jun 2013 01:07:06 AM CST, Konrad Rzeszutek Wilk wrote:
> On Sat, Jun 08, 2013 at 12:50:31AM +0800, Jiang Liu wrote:
>> On 06/07/2013 11:38 PM, Konrad Rzeszutek Wilk wrote:
>>> On Fri, Jun 07, 2013 at 10:50:24AM -0400, Konrad Rzeszutek Wilk wrote:
>>>> On Thu, May 16, 2013 at 11:50:55PM +0800, Jiang Liu wrote:
>>>>> Use new PCI interfaces to simplify xen-pcifront implementation:
>>>>> 1) Use pci_create_root_bus() instead of pci_scan_bus_parented()
>>>>>    because pci_scan_bus_parented() is marked as __deprecated.This
>>>>>    also gets rid of a duplicated call of pci_bus_start_devices().
>>>>> 2) Use pci_stop_root_bus() and pci_remove_root_bus() instead of
>>>>>    open-coded private implementation.
>>>>> 3) Use pci_set_host_bridge_release() to release data structures
>>>>>    associated with PCI root buses.
>>>>> 4) Use pci_bus_get()/pci_bus_put() to manage PCI root bus reference
>>>>>    count.
>>>>>
>>>>> This is also a preparation for coming PCI bus lock enhancement.
>>>
>>> With this patch from :
>>>
>>>  Merge branch 'pci_lock_v3' of https://github.com/jiangliu/linux into testing
>>>
>>>
>>> it blows up when detaching the device.
>> Hi Konrad,
>>    Thanks for testing! According to the log messages, this issue should
>> be related to pci bus reference counter management. Seems we have done
>> an extra(unbalanced) release of pci bus device.
>>    Will investigate it tomorrow!
>
> That is quite commendable that you are willing to look over this on
> the weekend but I am not going to be able to rerun this test until
> some time in the week. You could enjoy the weekend and just look at
> this during the week.

Hi Konrad,
     We should have root-caused this bug, which is caused by 
for_each_pci_root_bus().
Current implementation doesn't support root bus deletion when walking 
PCI root
buses by for_each_pci_root_bus(). The reference counter 
(pci_bus->dev.knode_class.n_ref)
becomes zero after returning from pci_remove_root_bus(), so it triggers 
kref warnings
and double-free of klist_node object when we call 
pci_get_next_root_bus() to get the
next PCI root bus.
     So we will first revert to  list_for_each_entry_safe(bus, temp, 
&pci_root_buses, node)
and solve this issue in next version of for_each_pci_root_bus().

Regards!
Gerry

---
diff --git a/drivers/pci/xen-pcifront.c b/drivers/pci/xen-pcifront.c
index 6aa2c0f..6e577db 100644
--- a/drivers/pci/xen-pcifront.c
+++ b/drivers/pci/xen-pcifront.c
@@ -553,11 +553,11 @@ static int pcifront_rescan_root(struct 
pcifront_device *pdev,
 static void pcifront_free_roots(struct pcifront_device *pdev)
 {
        struct pcifront_sd *sd;
-       struct pci_bus *bus;
+       struct pci_bus *bus, *temp;

        dev_dbg(&pdev->xdev->dev, "cleaning up root buses\n");

-       for_each_pci_root_bus(bus) {
+       list_for_each_entry_safe(bus, temp, &pci_root_buses, node) {
                sd = bus->sysdata;
                if (sd->pdev == pdev) {
                        pci_stop_root_bus(bus);
---

Re: [RFC PATCH v2, part3 07/11] PCI, xen-pcifront: use new PCI interfaces to simplify implementation

[Konrad Rzeszutek Wilk]

On Mon, Jun 10, 2013 at 12:50:46AM +0800, Jiang Liu wrote:
> On Sat 08 Jun 2013 01:07:06 AM CST, Konrad Rzeszutek Wilk wrote:
> > On Sat, Jun 08, 2013 at 12:50:31AM +0800, Jiang Liu wrote:
> >> On 06/07/2013 11:38 PM, Konrad Rzeszutek Wilk wrote:
> >>> On Fri, Jun 07, 2013 at 10:50:24AM -0400, Konrad Rzeszutek Wilk wrote:
> >>>> On Thu, May 16, 2013 at 11:50:55PM +0800, Jiang Liu wrote:
> >>>>> Use new PCI interfaces to simplify xen-pcifront implementation:
> >>>>> 1) Use pci_create_root_bus() instead of pci_scan_bus_parented()
> >>>>>    because pci_scan_bus_parented() is marked as __deprecated.This
> >>>>>    also gets rid of a duplicated call of pci_bus_start_devices().
> >>>>> 2) Use pci_stop_root_bus() and pci_remove_root_bus() instead of
> >>>>>    open-coded private implementation.
> >>>>> 3) Use pci_set_host_bridge_release() to release data structures
> >>>>>    associated with PCI root buses.
> >>>>> 4) Use pci_bus_get()/pci_bus_put() to manage PCI root bus reference
> >>>>>    count.
> >>>>>
> >>>>> This is also a preparation for coming PCI bus lock enhancement.
> >>>
> >>> With this patch from :
> >>>
> >>>  Merge branch 'pci_lock_v3' of https://github.com/jiangliu/linux into testing
> >>>
> >>>
> >>> it blows up when detaching the device.
> >> Hi Konrad,
> >>    Thanks for testing! According to the log messages, this issue should
> >> be related to pci bus reference counter management. Seems we have done
> >> an extra(unbalanced) release of pci bus device.
> >>    Will investigate it tomorrow!
> >
> > That is quite commendable that you are willing to look over this on
> > the weekend but I am not going to be able to rerun this test until
> > some time in the week. You could enjoy the weekend and just look at
> > this during the week.
> 
> Hi Konrad,
>      We should have root-caused this bug, which is caused by 
> for_each_pci_root_bus().
> Current implementation doesn't support root bus deletion when walking 
> PCI root
> buses by for_each_pci_root_bus(). The reference counter 
> (pci_bus->dev.knode_class.n_ref)
> becomes zero after returning from pci_remove_root_bus(), so it triggers 
> kref warnings
> and double-free of klist_node object when we call 
> pci_get_next_root_bus() to get the
> next PCI root bus.
>      So we will first revert to  list_for_each_entry_safe(bus, temp, 
> &pci_root_buses, node)
> and solve this issue in next version of for_each_pci_root_bus().

That definitly solves the issue. Thanks!

I used pci_lock_v3 of https://github.com/jiangliu/linux and the fix
below.

Is there a new patchset you are going to be posting or a git branch
that I can look over?

Thanks.

Re: [RFC PATCH v2, part3 07/11] PCI, xen-pcifront: use new PCI interfaces to simplify implementation

[Jiang Liu]

On Tue 11 Jun 2013 12:58:01 AM CST, Konrad Rzeszutek Wilk wrote:
> On Mon, Jun 10, 2013 at 12:50:46AM +0800, Jiang Liu wrote:
>> On Sat 08 Jun 2013 01:07:06 AM CST, Konrad Rzeszutek Wilk wrote:
>>> On Sat, Jun 08, 2013 at 12:50:31AM +0800, Jiang Liu wrote:
>>>> On 06/07/2013 11:38 PM, Konrad Rzeszutek Wilk wrote:
>>>>> On Fri, Jun 07, 2013 at 10:50:24AM -0400, Konrad Rzeszutek Wilk wrote:
>>>>>> On Thu, May 16, 2013 at 11:50:55PM +0800, Jiang Liu wrote:
>>>>>>> Use new PCI interfaces to simplify xen-pcifront implementation:
>>>>>>> 1) Use pci_create_root_bus() instead of pci_scan_bus_parented()
>>>>>>>    because pci_scan_bus_parented() is marked as __deprecated.This
>>>>>>>    also gets rid of a duplicated call of pci_bus_start_devices().
>>>>>>> 2) Use pci_stop_root_bus() and pci_remove_root_bus() instead of
>>>>>>>    open-coded private implementation.
>>>>>>> 3) Use pci_set_host_bridge_release() to release data structures
>>>>>>>    associated with PCI root buses.
>>>>>>> 4) Use pci_bus_get()/pci_bus_put() to manage PCI root bus reference
>>>>>>>    count.
>>>>>>>
>>>>>>> This is also a preparation for coming PCI bus lock enhancement.
>>>>>
>>>>> With this patch from :
>>>>>
>>>>>  Merge branch 'pci_lock_v3' of https://github.com/jiangliu/linux into testing
>>>>>
>>>>>
>>>>> it blows up when detaching the device.
>>>> Hi Konrad,
>>>>    Thanks for testing! According to the log messages, this issue should
>>>> be related to pci bus reference counter management. Seems we have done
>>>> an extra(unbalanced) release of pci bus device.
>>>>    Will investigate it tomorrow!
>>>
>>> That is quite commendable that you are willing to look over this on
>>> the weekend but I am not going to be able to rerun this test until
>>> some time in the week. You could enjoy the weekend and just look at
>>> this during the week.
>>
>> Hi Konrad,
>>      We should have root-caused this bug, which is caused by
>> for_each_pci_root_bus().
>> Current implementation doesn't support root bus deletion when walking
>> PCI root
>> buses by for_each_pci_root_bus(). The reference counter
>> (pci_bus->dev.knode_class.n_ref)
>> becomes zero after returning from pci_remove_root_bus(), so it triggers
>> kref warnings
>> and double-free of klist_node object when we call
>> pci_get_next_root_bus() to get the
>> next PCI root bus.
>>      So we will first revert to  list_for_each_entry_safe(bus, temp,
>> &pci_root_buses, node)
>> and solve this issue in next version of for_each_pci_root_bus().
>
> That definitly solves the issue. Thanks!
>
> I used pci_lock_v3 of https://github.com/jiangliu/linux and the fix
> below.
>
> Is there a new patchset you are going to be posting or a git branch
> that I can look over?
>
> Thanks.
Hi Konrad,
     Really appreciate your support! I will try to post a new version 
tomorrow,
but I think there won't be big changes except addressing some review
comments from you.
    Thanks!
    Gerry

Re: [RFC PATCH v2, part3 07/11] PCI, xen-pcifront: use new PCI interfaces to simplify implementation

[Konrad Rzeszutek Wilk]

On Tue, Jun 11, 2013 at 01:08:14AM +0800, Jiang Liu wrote:
> On Tue 11 Jun 2013 12:58:01 AM CST, Konrad Rzeszutek Wilk wrote:
> > On Mon, Jun 10, 2013 at 12:50:46AM +0800, Jiang Liu wrote:
> >> On Sat 08 Jun 2013 01:07:06 AM CST, Konrad Rzeszutek Wilk wrote:
> >>> On Sat, Jun 08, 2013 at 12:50:31AM +0800, Jiang Liu wrote:
> >>>> On 06/07/2013 11:38 PM, Konrad Rzeszutek Wilk wrote:
> >>>>> On Fri, Jun 07, 2013 at 10:50:24AM -0400, Konrad Rzeszutek Wilk wrote:
> >>>>>> On Thu, May 16, 2013 at 11:50:55PM +0800, Jiang Liu wrote:
> >>>>>>> Use new PCI interfaces to simplify xen-pcifront implementation:
> >>>>>>> 1) Use pci_create_root_bus() instead of pci_scan_bus_parented()
> >>>>>>>    because pci_scan_bus_parented() is marked as __deprecated.This
> >>>>>>>    also gets rid of a duplicated call of pci_bus_start_devices().
> >>>>>>> 2) Use pci_stop_root_bus() and pci_remove_root_bus() instead of
> >>>>>>>    open-coded private implementation.
> >>>>>>> 3) Use pci_set_host_bridge_release() to release data structures
> >>>>>>>    associated with PCI root buses.
> >>>>>>> 4) Use pci_bus_get()/pci_bus_put() to manage PCI root bus reference
> >>>>>>>    count.
> >>>>>>>
> >>>>>>> This is also a preparation for coming PCI bus lock enhancement.
> >>>>>
> >>>>> With this patch from :
> >>>>>
> >>>>>  Merge branch 'pci_lock_v3' of https://github.com/jiangliu/linux into testing
> >>>>>
> >>>>>
> >>>>> it blows up when detaching the device.
> >>>> Hi Konrad,
> >>>>    Thanks for testing! According to the log messages, this issue should
> >>>> be related to pci bus reference counter management. Seems we have done
> >>>> an extra(unbalanced) release of pci bus device.
> >>>>    Will investigate it tomorrow!
> >>>
> >>> That is quite commendable that you are willing to look over this on
> >>> the weekend but I am not going to be able to rerun this test until
> >>> some time in the week. You could enjoy the weekend and just look at
> >>> this during the week.
> >>
> >> Hi Konrad,
> >>      We should have root-caused this bug, which is caused by
> >> for_each_pci_root_bus().
> >> Current implementation doesn't support root bus deletion when walking
> >> PCI root
> >> buses by for_each_pci_root_bus(). The reference counter
> >> (pci_bus->dev.knode_class.n_ref)
> >> becomes zero after returning from pci_remove_root_bus(), so it triggers
> >> kref warnings
> >> and double-free of klist_node object when we call
> >> pci_get_next_root_bus() to get the
> >> next PCI root bus.
> >>      So we will first revert to  list_for_each_entry_safe(bus, temp,
> >> &pci_root_buses, node)
> >> and solve this issue in next version of for_each_pci_root_bus().
> >
> > That definitly solves the issue. Thanks!
> >
> > I used pci_lock_v3 of https://github.com/jiangliu/linux and the fix
> > below.
> >
> > Is there a new patchset you are going to be posting or a git branch
> > that I can look over?
> >
> > Thanks.
> Hi Konrad,
>      Really appreciate your support! I will try to post a new version 
> tomorrow,
> but I think there won't be big changes except addressing some review
> comments from you.

I also get this:

ERROR: "pci_set_host_bridge_release" [drivers/pci/xen-pcifront.ko] undefined!
ERROR: "pci_create_root_bus" [drivers/pci/xen-pcifront.ko] undefined!
ERROR: "pci_remove_root_bus" [drivers/pci/xen-pcifront.ko] undefined!
ERROR: "pci_stop_root_bus" [drivers/pci/xen-pcifront.ko] undefined!

when building it as a module.