[PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

Hi Tejun,

This is a patch I sent to Jeff few months ago. As you asked, I am
resending it on top of 3.10-rc2 Linus tree. Jeff has said he
applied this patch, but I am not sure what exactly it means ;)

Also, I am not sure about my reading of the statistics and the
trade-off I identified (below), so this patch is a RFC.

The numbers are taken by running 'if=/dev/sd{a,b,c} of=/dev/null'
All time values is in us.

Before this update host lock average holdtime was 3.266532061 and
average waittime was 0.009832679 [1]. After the update average
holdtime (slightly) rose up to 0.335267418 while average waittime
decreased to 0.000320469 [2]. Which means host lock with local
interrupt disabled is held roughly the same while the average
waittime dropped 30 times.

After this update port events are handled with local interrupts
enabled and compete on individual per-port locks with average
holdtime 1.540987475 and average waittime 0.000714864 [3].
Comparing to [1], ata_scsi_queuecmd() holds port locks 2 times
less and waits for locks 13 times less.

The downside of this change is introduction of a kernel thread
and (supposedly) increased total average time of handling a
AHCI interrupt - at most 1.5 times.

The upside is better access times from ata_scsi_queuecmd() to
port locks and moving port interrupt handling out of the
hardware interrupt context.

Thanks!

Lock usage statistics.

	1. ahci_interrupt vs ata_scsi_queuecmd (host->lock)

Test	holdtime-total	waittime-total	acquisitions	holdtime-avg	waittime-avg
#
01.	22732497.77	93399.89	06393367	3.555637862	0.014608874
02.	20358052.08	52869.72	06454133	3.154265969	0.008191607
03.	20322516.57	54981.40	06459318	3.146232554	0.008511951
04.	18558686.89	39178.05	06469468	2.868657344	0.006055838
05.	19069799.90	31961.00	06455953	2.953831897	0.004950625
06.	23783542.56	97159.79	06387322	3.723554654	0.015211350
07.	23889266.74	102625.45	06386666	3.740491007	0.016068705
08.	19284522.61	32655.91	06450568	2.989585198	0.005062486
							-----------	-----------
avg							3.266532061	0.009832679

	2. ahci_single_irq_intr vs ahci_port_thread_fn (host->lock)


Alexander Gordeev (1):
  AHCI: Optimize interrupt processing

 drivers/ata/acard-ahci.c    |    8 ++---
 drivers/ata/ahci.c          |   54 ++++++++++++++++++-------------
 drivers/ata/ahci.h          |   10 +++--
 drivers/ata/ahci_platform.c |    3 +-
 drivers/ata/libahci.c       |   74 +++++++++++++++++++++++++------------------
 5 files changed, 85 insertions(+), 64 deletions(-)

-- 
1.7.7.6


-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

[PATCH RESEND 1/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

Split interrupt service routine into hardware context handler and
threaded context handler. That allows to protect ports with individual
locks rather than with a single host-wide lock, which results in better
parallelism.

Signed-off-by: Alexander Gordeev <agordeev@redhat.com>
---
 drivers/ata/acard-ahci.c    |    8 ++---
 drivers/ata/ahci.c          |   54 ++++++++++++++++++-------------
 drivers/ata/ahci.h          |   10 +++--
 drivers/ata/ahci_platform.c |    3 +-
 drivers/ata/libahci.c       |   74 +++++++++++++++++++++++++------------------
 5 files changed, 85 insertions(+), 64 deletions(-)

diff --git a/drivers/ata/acard-ahci.c b/drivers/ata/acard-ahci.c
index 4e94ba2..e429225 100644
--- a/drivers/ata/acard-ahci.c
+++ b/drivers/ata/acard-ahci.c
@@ -409,7 +409,7 @@ static int acard_ahci_init_one(struct pci_dev *pdev, const struct pci_device_id
 	struct device *dev = &pdev->dev;
 	struct ahci_host_priv *hpriv;
 	struct ata_host *host;
-	int n_ports, i, rc;
+	int n_ports, n_msis, i, rc;
 
 	VPRINTK("ENTER\n");
 
@@ -436,8 +436,7 @@ static int acard_ahci_init_one(struct pci_dev *pdev, const struct pci_device_id
 		return -ENOMEM;
 	hpriv->flags |= (unsigned long)pi.private_data;
 
-	if (!(hpriv->flags & AHCI_HFLAG_NO_MSI))
-		pci_enable_msi(pdev);
+	n_msis = ahci_init_interrupts(pdev, hpriv);
 
 	hpriv->mmio = pcim_iomap_table(pdev)[AHCI_PCI_BAR];
 
@@ -499,8 +498,7 @@ static int acard_ahci_init_one(struct pci_dev *pdev, const struct pci_device_id
 	acard_ahci_pci_print_info(host);
 
 	pci_set_master(pdev);
-	return ata_host_activate(host, pdev->irq, ahci_interrupt, IRQF_SHARED,
-				 &acard_ahci_sht);
+	return ahci_host_activate(host, pdev->irq, n_msis, &acard_ahci_sht);
 }
 
 module_pci_driver(acard_ahci_pci_driver);
diff --git a/drivers/ata/ahci.c b/drivers/ata/ahci.c
index 251e57d..e4d915f 100644
--- a/drivers/ata/ahci.c
+++ b/drivers/ata/ahci.c
@@ -1110,14 +1110,14 @@ int ahci_init_interrupts(struct pci_dev *pdev, struct ahci_host_priv *hpriv)
 	pci_intx(pdev, 1);
 	return 0;
 }
+EXPORT_SYMBOL_GPL(ahci_init_interrupts);
 
 /**
  *	ahci_host_activate - start AHCI host, request IRQs and register it
  *	@host: target ATA host
  *	@irq: base IRQ number to request
  *	@n_msis: number of MSIs allocated for this host
- *	@irq_handler: irq_handler used when requesting IRQs
- *	@irq_flags: irq_flags used when requesting IRQs
+ *	@sht: scsi_host_template to use when registering the host
  *
  *	Similar to ata_host_activate, but requests IRQs according to AHCI-1.1
  *	when multiple MSIs were allocated. That is one MSI per port, starting
@@ -1129,43 +1129,59 @@ int ahci_init_interrupts(struct pci_dev *pdev, struct ahci_host_priv *hpriv)
  *	RETURNS:
  *	0 on success, -errno otherwise.
  */
-int ahci_host_activate(struct ata_host *host, int irq, unsigned int n_msis)
+int ahci_host_activate(struct ata_host *host, int irq, unsigned int n_msis,
+		       struct scsi_host_template *sht)
 {
 	int i, rc;
-
-	/* Sharing Last Message among several ports is not supported */
-	if (n_msis < host->n_ports)
-		return -EINVAL;
+	unsigned int n_irqs;
 
 	rc = ata_host_start(host);
 	if (rc)
 		return rc;
 
-	for (i = 0; i < host->n_ports; i++) {
-		rc = devm_request_threaded_irq(host->dev,
-			irq + i, ahci_hw_interrupt, ahci_thread_fn, IRQF_SHARED,
-			dev_driver_string(host->dev), host->ports[i]);
+	n_irqs = min(host->n_ports, n_msis);
+	n_irqs = max(n_irqs, 1u);
+
+	if (n_irqs > 1) {
+		/* Sharing Last Message among several ports is not supported */
+		if (n_irqs < host->n_ports)
+			return -EINVAL;
+
+		for (i = 0; i < n_irqs; i++) {
+			rc = devm_request_threaded_irq(host->dev, irq + i,
+				ahci_multi_irqs_intr, ahci_port_thread_fn,
+				IRQF_SHARED, dev_driver_string(host->dev),
+				host->ports[i]);
+			if (rc)
+				goto out_free_irqs;
+		}
+	} else {
+		rc = devm_request_threaded_irq(host->dev, irq,
+			ahci_single_irq_intr, ahci_thread_fn, IRQF_SHARED,
+			dev_driver_string(host->dev), host);
 		if (rc)
-			goto out_free_irqs;
+			goto out;
 	}
 
-	for (i = 0; i < host->n_ports; i++)
+	for (i = 0; i < n_irqs; i++)
 		ata_port_desc(host->ports[i], "irq %d", irq + i);
 
-	rc = ata_host_register(host, &ahci_sht);
+	rc = ata_host_register(host, sht);
 	if (rc)
 		goto out_free_all_irqs;
 
 	return 0;
 
 out_free_all_irqs:
-	i = host->n_ports;
+	i = n_irqs;
 out_free_irqs:
 	for (i--; i >= 0; i--)
 		devm_free_irq(host->dev, irq + i, host->ports[i]);
+out:
 
 	return rc;
 }
+EXPORT_SYMBOL_GPL(ahci_host_activate);
 
 static int ahci_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 {
@@ -1265,8 +1281,6 @@ static int ahci_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 	hpriv->mmio = pcim_iomap_table(pdev)[ahci_pci_bar];
 
 	n_msis = ahci_init_interrupts(pdev, hpriv);
-	if (n_msis > 1)
-		hpriv->flags |= AHCI_HFLAG_MULTI_MSI;
 
 	/* save initial config */
 	ahci_pci_save_initial_config(pdev, hpriv);
@@ -1364,11 +1378,7 @@ static int ahci_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 
 	pci_set_master(pdev);
 
-	if (hpriv->flags & AHCI_HFLAG_MULTI_MSI)
-		return ahci_host_activate(host, pdev->irq, n_msis);
-
-	return ata_host_activate(host, pdev->irq, ahci_interrupt, IRQF_SHARED,
-				 &ahci_sht);
+	return ahci_host_activate(host, pdev->irq, n_msis, &ahci_sht);
 }
 
 module_pci_driver(ahci_pci_driver);
diff --git a/drivers/ata/ahci.h b/drivers/ata/ahci.h
index b830e6c..ed1fbc8 100644
--- a/drivers/ata/ahci.h
+++ b/drivers/ata/ahci.h
@@ -231,7 +231,6 @@ enum {
 	AHCI_HFLAG_DELAY_ENGINE		= (1 << 15), /* do not start engine on
 						        port start (wait until
 						        error-handling stage) */
-	AHCI_HFLAG_MULTI_MSI		= (1 << 16), /* multiple PCI MSIs */
 
 	/* ap->flags bits */
 
@@ -361,11 +360,14 @@ int ahci_port_resume(struct ata_port *ap);
 void ahci_set_em_messages(struct ahci_host_priv *hpriv,
 			  struct ata_port_info *pi);
 int ahci_reset_em(struct ata_host *host);
-irqreturn_t ahci_interrupt(int irq, void *dev_instance);
-irqreturn_t ahci_hw_interrupt(int irq, void *dev_instance);
+irqreturn_t ahci_single_irq_intr(int irq, void *dev_instance);
+irqreturn_t ahci_multi_irqs_intr(int irq, void *dev_instance);
 irqreturn_t ahci_thread_fn(int irq, void *dev_instance);
+irqreturn_t ahci_port_thread_fn(int irq, void *dev_instance);
 void ahci_print_info(struct ata_host *host, const char *scc_s);
-int ahci_host_activate(struct ata_host *host, int irq, unsigned int n_msis);
+int ahci_host_activate(struct ata_host *host, int irq, unsigned int n_msis,
+		       struct scsi_host_template *sht);
+int ahci_init_interrupts(struct pci_dev *pdev, struct ahci_host_priv *hpriv);
 
 static inline void __iomem *__ahci_port_base(struct ata_host *host,
 					     unsigned int port_no)
diff --git a/drivers/ata/ahci_platform.c b/drivers/ata/ahci_platform.c
index 7a8a284..b10cd87 100644
--- a/drivers/ata/ahci_platform.c
+++ b/drivers/ata/ahci_platform.c
@@ -211,8 +211,7 @@ static int ahci_probe(struct platform_device *pdev)
 	ahci_init_controller(host);
 	ahci_print_info(host, "platform");
 
-	rc = ata_host_activate(host, irq, ahci_interrupt, IRQF_SHARED,
-			       &ahci_platform_sht);
+	rc = ahci_host_activate(host, irq, 0, &ahci_platform_sht);
 	if (rc)
 		goto pdata_exit;
 
diff --git a/drivers/ata/libahci.c b/drivers/ata/libahci.c
index 34c8216..68b3bdd 100644
--- a/drivers/ata/libahci.c
+++ b/drivers/ata/libahci.c
@@ -1655,9 +1655,9 @@ static void ahci_error_intr(struct ata_port *ap, u32 irq_stat)
 		ata_port_abort(ap);
 }
 
-static void ahci_handle_port_interrupt(struct ata_port *ap,
-				       void __iomem *port_mmio, u32 status)
+static void ahci_handle_port_interrupt(struct ata_port *ap, u32 status)
 {
+	void __iomem *port_mmio = ahci_port_base(ap);
 	struct ata_eh_info *ehi = &ap->link.eh_info;
 	struct ahci_port_priv *pp = ap->private_data;
 	struct ahci_host_priv *hpriv = ap->host->private_data;
@@ -1740,22 +1740,10 @@ static void ahci_handle_port_interrupt(struct ata_port *ap,
 	}
 }
 
-void ahci_port_intr(struct ata_port *ap)
-{
-	void __iomem *port_mmio = ahci_port_base(ap);
-	u32 status;
-
-	status = readl(port_mmio + PORT_IRQ_STAT);
-	writel(status, port_mmio + PORT_IRQ_STAT);
-
-	ahci_handle_port_interrupt(ap, port_mmio, status);
-}
-
-irqreturn_t ahci_thread_fn(int irq, void *dev_instance)
+irqreturn_t ahci_port_thread_fn(int irq, void *dev_instance)
 {
 	struct ata_port *ap = dev_instance;
 	struct ahci_port_priv *pp = ap->private_data;
-	void __iomem *port_mmio = ahci_port_base(ap);
 	unsigned long flags;
 	u32 status;
 
@@ -1766,14 +1754,43 @@ irqreturn_t ahci_thread_fn(int irq, void *dev_instance)
 	spin_unlock_irqrestore(&ap->host->lock, flags);
 
 	spin_lock_bh(ap->lock);
-	ahci_handle_port_interrupt(ap, port_mmio, status);
+	ahci_handle_port_interrupt(ap, status);
 	spin_unlock_bh(ap->lock);
 
 	return IRQ_HANDLED;
 }
+EXPORT_SYMBOL_GPL(ahci_port_thread_fn);
+
+irqreturn_t ahci_thread_fn(int irq, void *dev_instance)
+{
+	struct ata_host *host = dev_instance;
+	struct ahci_host_priv *hpriv = host->private_data;
+	u32 irq_masked = hpriv->port_map;
+	unsigned int i;
+
+	for (i = 0; i < host->n_ports; i++) {
+		struct ata_port *ap;
+
+		if (!(irq_masked & (1 << i)))
+			continue;
+
+		ap = host->ports[i];
+		if (ap) {
+			ahci_port_thread_fn(irq, ap);
+			VPRINTK("port %u\n", i);
+		} else {
+			VPRINTK("port %u (no irq)\n", i);
+			if (ata_ratelimit())
+				dev_warn(host->dev,
+					 "interrupt on disabled port %u\n", i);
+		}
+	}
+
+	return IRQ_HANDLED;
+}
 EXPORT_SYMBOL_GPL(ahci_thread_fn);
 
-void ahci_hw_port_interrupt(struct ata_port *ap)
+void ahci_update_intr_status(struct ata_port *ap)
 {
 	void __iomem *port_mmio = ahci_port_base(ap);
 	struct ahci_port_priv *pp = ap->private_data;
@@ -1785,7 +1802,7 @@ void ahci_hw_port_interrupt(struct ata_port *ap)
 	pp->intr_status |= status;
 }
 
-irqreturn_t ahci_hw_interrupt(int irq, void *dev_instance)
+irqreturn_t ahci_multi_irqs_intr(int irq, void *dev_instance)
 {
 	struct ata_port *ap_this = dev_instance;
 	struct ahci_port_priv *pp = ap_this->private_data;
@@ -1821,7 +1838,7 @@ irqreturn_t ahci_hw_interrupt(int irq, void *dev_instance)
 
 		ap = host->ports[i];
 		if (ap) {
-			ahci_hw_port_interrupt(ap);
+			ahci_update_intr_status(ap);
 			VPRINTK("port %u\n", i);
 		} else {
 			VPRINTK("port %u (no irq)\n", i);
@@ -1839,9 +1856,9 @@ irqreturn_t ahci_hw_interrupt(int irq, void *dev_instance)
 
 	return IRQ_WAKE_THREAD;
 }
-EXPORT_SYMBOL_GPL(ahci_hw_interrupt);
+EXPORT_SYMBOL_GPL(ahci_multi_irqs_intr);
 
-irqreturn_t ahci_interrupt(int irq, void *dev_instance)
+irqreturn_t ahci_single_irq_intr(int irq, void *dev_instance)
 {
 	struct ata_host *host = dev_instance;
 	struct ahci_host_priv *hpriv;
@@ -1871,7 +1888,7 @@ irqreturn_t ahci_interrupt(int irq, void *dev_instance)
 
 		ap = host->ports[i];
 		if (ap) {
-			ahci_port_intr(ap);
+			ahci_update_intr_status(ap);
 			VPRINTK("port %u\n", i);
 		} else {
 			VPRINTK("port %u (no irq)\n", i);
@@ -1898,9 +1915,9 @@ irqreturn_t ahci_interrupt(int irq, void *dev_instance)
 
 	VPRINTK("EXIT\n");
 
-	return IRQ_RETVAL(handled);
+	return handled ? IRQ_WAKE_THREAD : IRQ_NONE;
 }
-EXPORT_SYMBOL_GPL(ahci_interrupt);
+EXPORT_SYMBOL_GPL(ahci_single_irq_intr);
 
 static unsigned int ahci_qc_issue(struct ata_queued_cmd *qc)
 {
@@ -2294,13 +2311,8 @@ static int ahci_port_start(struct ata_port *ap)
 	 */
 	pp->intr_mask = DEF_PORT_IRQ;
 
-	/*
-	 * Switch to per-port locking in case each port has its own MSI vector.
-	 */
-	if ((hpriv->flags & AHCI_HFLAG_MULTI_MSI)) {
-		spin_lock_init(&pp->lock);
-		ap->lock = &pp->lock;
-	}
+	spin_lock_init(&pp->lock);
+	ap->lock = &pp->lock;
 
 	ap->private_data = pp;
 
-- 
1.7.7.6


-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Tejun Heo]

Hello, Alexander.

(cc'ing Jens and Nicholas, hey guys)

On Tue, May 21, 2013 at 09:00:27PM +0200, Alexander Gordeev wrote:
> Before this update host lock average holdtime was 3.266532061 and
> average waittime was 0.009832679 [1]. After the update average
> holdtime (slightly) rose up to 0.335267418 while average waittime
> decreased to 0.000320469 [2]. Which means host lock with local
> interrupt disabled is held roughly the same while the average
> waittime dropped 30 times.
> 
> After this update port events are handled with local interrupts
> enabled and compete on individual per-port locks with average
> holdtime 1.540987475 and average waittime 0.000714864 [3].
> Comparing to [1], ata_scsi_queuecmd() holds port locks 2 times
> less and waits for locks 13 times less.

Hmmmmmm..... I'd normally apply this patch but block layer is just
growing multi-queue support and libata is likely to be converted to mq
in foreseeable future, so I'm a bit hesitant to make irq handling more
sophiscated right now.  Would you be interested in looking into
converting libata to blk mq support?  I'm pretty sure it'd yield far
better outcome if done properly.

Thanks.

-- 
tejun

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

On Wed, May 22, 2013 at 08:50:03AM +0900, Tejun Heo wrote:
> Hmmmmmm..... I'd normally apply this patch but block layer is just
> growing multi-queue support and libata is likely to be converted to mq
> in foreseeable future, so I'm a bit hesitant to make irq handling more
> sophiscated right now.  Would you be interested in looking into
> converting libata to blk mq support?  I'm pretty sure it'd yield far
> better outcome if done properly.

I am not committing, but will look into it, sure.

> Thanks.
> 
> -- 
> tejun

-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Jens Axboe]

On Wed, May 22 2013, Alexander Gordeev wrote:
> On Wed, May 22, 2013 at 08:50:03AM +0900, Tejun Heo wrote:
> > Hmmmmmm..... I'd normally apply this patch but block layer is just
> > growing multi-queue support and libata is likely to be converted to mq
> > in foreseeable future, so I'm a bit hesitant to make irq handling more
> > sophiscated right now.  Would you be interested in looking into
> > converting libata to blk mq support?  I'm pretty sure it'd yield far
> > better outcome if done properly.
> 
> I am not committing, but will look into it, sure.

Would be most awesome, I'm sure Nic would not mind a bit of help on the
SCSI/libata side :-)

And personally, can't wait to run it on the laptop! That's right, I
alpha test on the laptop.

-- 
Jens Axboe

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

On Wed, May 22, 2013 at 07:03:05PM +0200, Jens Axboe wrote:
> On Wed, May 22 2013, Alexander Gordeev wrote:
> > On Wed, May 22, 2013 at 08:50:03AM +0900, Tejun Heo wrote:
> > > Hmmmmmm..... I'd normally apply this patch but block layer is just
> > > growing multi-queue support and libata is likely to be converted to mq
> > > in foreseeable future, so I'm a bit hesitant to make irq handling more
> > > sophiscated right now.  Would you be interested in looking into
> > > converting libata to blk mq support?  I'm pretty sure it'd yield far
> > > better outcome if done properly.
> > 
> > I am not committing, but will look into it, sure.
> 
> Would be most awesome, I'm sure Nic would not mind a bit of help on the
> SCSI/libata side :-)

Hi Nicholas,

Could you please clarify the status of SCSI MQ support? Is it usable now?

I tried git://git.kernel.org/pub/scm/linux/kernel/git/nab/target-pending.git,
but it does not appear working without (at least) changes below to SCSI lib:

Thanks!

diff --git a/drivers/scsi/scsi-mq.c b/drivers/scsi/scsi-mq.c
index ca6ff67..d8cc7a4 100644
--- a/drivers/scsi/scsi-mq.c
+++ b/drivers/scsi/scsi-mq.c
@@ -155,6 +155,7 @@ void scsi_mq_done(struct scsi_cmnd *sc)
 static struct blk_mq_ops scsi_mq_ops = {
 	.queue_rq	= scsi_mq_queue_rq,
 	.map_queue	= blk_mq_map_queue,
+	.timeout	= scsi_times_out,
 	.alloc_hctx	= blk_mq_alloc_single_hw_queue,
 	.free_hctx	= blk_mq_free_single_hw_queue,
 };
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 65360db..33aa373 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -283,7 +283,10 @@ int scsi_execute(struct scsi_device *sdev, const unsigned char *cmd,
 	/*
 	 * head injection *required* here otherwise quiesce won't work
 	 */
-	blk_execute_rq(req->q, NULL, req, 1);
+	if (q->mq_ops)
+		blk_mq_execute_rq(req->q, req);
+	else
+		blk_execute_rq(req->q, NULL, req, 1);
 
 	/*
 	 * Some devices (USB mass-storage in particular) may transfer
@@ -298,12 +301,8 @@ int scsi_execute(struct scsi_device *sdev, const unsigned char *cmd,
 		*resid = req->resid_len;
 	ret = req->errors;
  out:
-	if (q->mq_ops) {
-		printk("scsi_execute(): Calling blk_mq_free_request >>>\n");
-		blk_mq_free_request(req);
-	} else {
+	if (!q->mq_ops)
 		blk_put_request(req);
-	}
 
 	return ret;
 }


> And personally, can't wait to run it on the laptop! That's right, I
> alpha test on the laptop.
> 
> -- 
> Jens Axboe
> 

-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

On Wed, May 22, 2013 at 07:03:05PM +0200, Jens Axboe wrote:
> On Wed, May 22 2013, Alexander Gordeev wrote:
> > On Wed, May 22, 2013 at 08:50:03AM +0900, Tejun Heo wrote:
> > > Hmmmmmm..... I'd normally apply this patch but block layer is just
> > > growing multi-queue support and libata is likely to be converted to mq
> > > in foreseeable future, so I'm a bit hesitant to make irq handling more
> > > sophiscated right now.  Would you be interested in looking into
> > > converting libata to blk mq support?  I'm pretty sure it'd yield far
> > > better outcome if done properly.
> > 
> > I am not committing, but will look into it, sure.
> 
> Would be most awesome, I'm sure Nic would not mind a bit of help on the
> SCSI/libata side :-)

Hi Nicholas,

Could you please clarify the status of SCSI MQ support? Is it usable now?

I tried git://git.kernel.org/pub/scm/linux/kernel/git/nab/target-pending.git,
but it does not appear working without (at least) changes below to SCSI lib:

Thanks!

diff --git a/drivers/scsi/scsi-mq.c b/drivers/scsi/scsi-mq.c
index ca6ff67..d8cc7a4 100644
--- a/drivers/scsi/scsi-mq.c
+++ b/drivers/scsi/scsi-mq.c
@@ -155,6 +155,7 @@ void scsi_mq_done(struct scsi_cmnd *sc)
 static struct blk_mq_ops scsi_mq_ops = {
 	.queue_rq	= scsi_mq_queue_rq,
 	.map_queue	= blk_mq_map_queue,
+	.timeout	= scsi_times_out,
 	.alloc_hctx	= blk_mq_alloc_single_hw_queue,
 	.free_hctx	= blk_mq_free_single_hw_queue,
 };
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 65360db..33aa373 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -283,7 +283,10 @@ int scsi_execute(struct scsi_device *sdev, const unsigned char *cmd,
 	/*
 	 * head injection *required* here otherwise quiesce won't work
 	 */
-	blk_execute_rq(req->q, NULL, req, 1);
+	if (q->mq_ops)
+		blk_mq_execute_rq(req->q, req);
+	else
+		blk_execute_rq(req->q, NULL, req, 1);
 
 	/*
 	 * Some devices (USB mass-storage in particular) may transfer
@@ -298,12 +301,8 @@ int scsi_execute(struct scsi_device *sdev, const unsigned char *cmd,
 		*resid = req->resid_len;
 	ret = req->errors;
  out:
-	if (q->mq_ops) {
-		printk("scsi_execute(): Calling blk_mq_free_request >>>\n");
-		blk_mq_free_request(req);
-	} else {
+	if (!q->mq_ops)
 		blk_put_request(req);
-	}
 
 	return ret;
 }


> And personally, can't wait to run it on the laptop! That's right, I
> alpha test on the laptop.
> 
> -- 
> Jens Axboe
> 

-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Nicholas A. Bellinger]

On Thu, 2013-07-11 at 12:26 +0200, Alexander Gordeev wrote:
> On Wed, May 22, 2013 at 07:03:05PM +0200, Jens Axboe wrote:
> > On Wed, May 22 2013, Alexander Gordeev wrote:
> > > On Wed, May 22, 2013 at 08:50:03AM +0900, Tejun Heo wrote:
> > > > Hmmmmmm..... I'd normally apply this patch but block layer is just
> > > > growing multi-queue support and libata is likely to be converted to mq
> > > > in foreseeable future, so I'm a bit hesitant to make irq handling more
> > > > sophiscated right now.  Would you be interested in looking into
> > > > converting libata to blk mq support?  I'm pretty sure it'd yield far
> > > > better outcome if done properly.
> > > 
> > > I am not committing, but will look into it, sure.
> > 
> > Would be most awesome, I'm sure Nic would not mind a bit of help on the
> > SCSI/libata side :-)
> 
> Hi Nicholas,
> 
> Could you please clarify the status of SCSI MQ support? Is it usable now?
> 

Hi Alexander,

Thanks for taking a look.  I've not made further progress in the last
weeks on scsi-mq, but am still using virtio-scsi + scsi-mq <->
vhost-scsi + per-cpu-ida for quite a bit for benchmarking purposes.

> I tried git://git.kernel.org/pub/scm/linux/kernel/git/nab/target-pending.git,
> but it does not appear working without (at least) changes below to SCSI lib:
> 

The only scsi-mq LLD conversions so far have been to virtio-scsi +
scsi_debug to nop REQ_TYPE_FS.

Just so I understand, your patch below is required in order to make what
LLD function with scsi-mq..?

Thanks!

--nab

> Thanks!
> 
> diff --git a/drivers/scsi/scsi-mq.c b/drivers/scsi/scsi-mq.c
> index ca6ff67..d8cc7a4 100644
> --- a/drivers/scsi/scsi-mq.c
> +++ b/drivers/scsi/scsi-mq.c
> @@ -155,6 +155,7 @@ void scsi_mq_done(struct scsi_cmnd *sc)
>  static struct blk_mq_ops scsi_mq_ops = {
>  	.queue_rq	= scsi_mq_queue_rq,
>  	.map_queue	= blk_mq_map_queue,
> +	.timeout	= scsi_times_out,
>  	.alloc_hctx	= blk_mq_alloc_single_hw_queue,
>  	.free_hctx	= blk_mq_free_single_hw_queue,
>  };
> diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
> index 65360db..33aa373 100644
> --- a/drivers/scsi/scsi_lib.c
> +++ b/drivers/scsi/scsi_lib.c
> @@ -283,7 +283,10 @@ int scsi_execute(struct scsi_device *sdev, const unsigned char *cmd,
>  	/*
>  	 * head injection *required* here otherwise quiesce won't work
>  	 */
> -	blk_execute_rq(req->q, NULL, req, 1);
> +	if (q->mq_ops)
> +		blk_mq_execute_rq(req->q, req);
> +	else
> +		blk_execute_rq(req->q, NULL, req, 1);
>  
>  	/*
>  	 * Some devices (USB mass-storage in particular) may transfer
> @@ -298,12 +301,8 @@ int scsi_execute(struct scsi_device *sdev, const unsigned char *cmd,
>  		*resid = req->resid_len;
>  	ret = req->errors;
>   out:
> -	if (q->mq_ops) {
> -		printk("scsi_execute(): Calling blk_mq_free_request >>>\n");
> -		blk_mq_free_request(req);
> -	} else {
> +	if (!q->mq_ops)
>  		blk_put_request(req);
> -	}
>  
>  	return ret;
>  }
> 
> 
> > And personally, can't wait to run it on the laptop! That's right, I
> > alpha test on the laptop.
> > 
> > -- 
> > Jens Axboe
> > 
> 

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

On Thu, Jul 11, 2013 at 04:00:37PM -0700, Nicholas A. Bellinger wrote:
> On Thu, 2013-07-11 at 12:26 +0200, Alexander Gordeev wrote:
> > On Wed, May 22, 2013 at 07:03:05PM +0200, Jens Axboe wrote:
> > > On Wed, May 22 2013, Alexander Gordeev wrote:
> > > > On Wed, May 22, 2013 at 08:50:03AM +0900, Tejun Heo wrote:
> > > > > Hmmmmmm..... I'd normally apply this patch but block layer is just
> > > > > growing multi-queue support and libata is likely to be converted to mq
> > > > > in foreseeable future, so I'm a bit hesitant to make irq handling more
> > > > > sophiscated right now.  Would you be interested in looking into
> > > > > converting libata to blk mq support?  I'm pretty sure it'd yield far
> > > > > better outcome if done properly.
> > > > 
> > > > I am not committing, but will look into it, sure.
> > > 
> > > Would be most awesome, I'm sure Nic would not mind a bit of help on the
> > > SCSI/libata side :-)
> > 
> > Hi Nicholas,
> > 
> > Could you please clarify the status of SCSI MQ support? Is it usable now?
> > 
> 
> Hi Alexander,
> 
> Thanks for taking a look.  I've not made further progress in the last
> weeks on scsi-mq, but am still using virtio-scsi + scsi-mq <->
> vhost-scsi + per-cpu-ida for quite a bit for benchmarking purposes.

Thanks for the clarification, Nicholas.

> > I tried git://git.kernel.org/pub/scm/linux/kernel/git/nab/target-pending.git,
> > but it does not appear working without (at least) changes below to SCSI lib:
> > 
> 
> The only scsi-mq LLD conversions so far have been to virtio-scsi +
> scsi_debug to nop REQ_TYPE_FS.

I see. Do you think the changes I made is a right way to go?

I had to make it to avoid a NULL-pointer assignent and make BIO bounce
buffers work, but I do not really understand the mixture of old and
new code ( neither in fact :) )

> Just so I understand, your patch below is required in order to make what
> LLD function with scsi-mq..?

ata_piix

> 
> Thanks!
> 
> --nab
> 
> > Thanks!
> > 
> > diff --git a/drivers/scsi/scsi-mq.c b/drivers/scsi/scsi-mq.c
> > index ca6ff67..d8cc7a4 100644
> > --- a/drivers/scsi/scsi-mq.c
> > +++ b/drivers/scsi/scsi-mq.c
> > @@ -155,6 +155,7 @@ void scsi_mq_done(struct scsi_cmnd *sc)
> >  static struct blk_mq_ops scsi_mq_ops = {
> >  	.queue_rq	= scsi_mq_queue_rq,
> >  	.map_queue	= blk_mq_map_queue,
> > +	.timeout	= scsi_times_out,
> >  	.alloc_hctx	= blk_mq_alloc_single_hw_queue,
> >  	.free_hctx	= blk_mq_free_single_hw_queue,
> >  };
> > diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
> > index 65360db..33aa373 100644
> > --- a/drivers/scsi/scsi_lib.c
> > +++ b/drivers/scsi/scsi_lib.c
> > @@ -283,7 +283,10 @@ int scsi_execute(struct scsi_device *sdev, const unsigned char *cmd,
> >  	/*
> >  	 * head injection *required* here otherwise quiesce won't work
> >  	 */
> > -	blk_execute_rq(req->q, NULL, req, 1);
> > +	if (q->mq_ops)
> > +		blk_mq_execute_rq(req->q, req);
> > +	else
> > +		blk_execute_rq(req->q, NULL, req, 1);
> >  
> >  	/*
> >  	 * Some devices (USB mass-storage in particular) may transfer
> > @@ -298,12 +301,8 @@ int scsi_execute(struct scsi_device *sdev, const unsigned char *cmd,
> >  		*resid = req->resid_len;
> >  	ret = req->errors;
> >   out:
> > -	if (q->mq_ops) {
> > -		printk("scsi_execute(): Calling blk_mq_free_request >>>\n");
> > -		blk_mq_free_request(req);
> > -	} else {
> > +	if (!q->mq_ops)
> >  		blk_put_request(req);
> > -	}
> >  
> >  	return ret;
> >  }
> > 
> > 
> > > And personally, can't wait to run it on the laptop! That's right, I
> > > alpha test on the laptop.
> > > 
> > > -- 
> > > Jens Axboe
> > > 
> > 
> 
> 

-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Nicholas A. Bellinger]

On Fri, 2013-07-12 at 09:46 +0200, Alexander Gordeev wrote:
> On Thu, Jul 11, 2013 at 04:00:37PM -0700, Nicholas A. Bellinger wrote:
> > On Thu, 2013-07-11 at 12:26 +0200, Alexander Gordeev wrote:
> > > On Wed, May 22, 2013 at 07:03:05PM +0200, Jens Axboe wrote:
> > > > On Wed, May 22 2013, Alexander Gordeev wrote:
> > > > > On Wed, May 22, 2013 at 08:50:03AM +0900, Tejun Heo wrote:
> > > > > > Hmmmmmm..... I'd normally apply this patch but block layer is just
> > > > > > growing multi-queue support and libata is likely to be converted to mq
> > > > > > in foreseeable future, so I'm a bit hesitant to make irq handling more
> > > > > > sophiscated right now.  Would you be interested in looking into
> > > > > > converting libata to blk mq support?  I'm pretty sure it'd yield far
> > > > > > better outcome if done properly.
> > > > > 
> > > > > I am not committing, but will look into it, sure.
> > > > 
> > > > Would be most awesome, I'm sure Nic would not mind a bit of help on the
> > > > SCSI/libata side :-)
> > > 
> > > Hi Nicholas,
> > > 
> > > Could you please clarify the status of SCSI MQ support? Is it usable now?
> > > 
> > 
> > Hi Alexander,
> > 
> > Thanks for taking a look.  I've not made further progress in the last
> > weeks on scsi-mq, but am still using virtio-scsi + scsi-mq <->
> > vhost-scsi + per-cpu-ida for quite a bit for benchmarking purposes.
> 
> Thanks for the clarification, Nicholas.
> 
> > > I tried git://git.kernel.org/pub/scm/linux/kernel/git/nab/target-pending.git,
> > > but it does not appear working without (at least) changes below to SCSI lib:
> > > 
> > 
> > The only scsi-mq LLD conversions so far have been to virtio-scsi +
> > scsi_debug to nop REQ_TYPE_FS.
> 
> I see. Do you think the changes I made is a right way to go?
> 
> I had to make it to avoid a NULL-pointer assignent and make BIO bounce
> buffers work, but I do not really understand the mixture of old and
> new code ( neither in fact :) )
> 

Hi Alexander,

Comments below.

> > Just so I understand, your patch below is required in order to make what
> > LLD function with scsi-mq..?
> 
> ata_piix
> 

<nod>

> > 
> > Thanks!
> > 
> > --nab
> > 
> > > Thanks!
> > > 
> > > diff --git a/drivers/scsi/scsi-mq.c b/drivers/scsi/scsi-mq.c
> > > index ca6ff67..d8cc7a4 100644
> > > --- a/drivers/scsi/scsi-mq.c
> > > +++ b/drivers/scsi/scsi-mq.c
> > > @@ -155,6 +155,7 @@ void scsi_mq_done(struct scsi_cmnd *sc)
> > >  static struct blk_mq_ops scsi_mq_ops = {
> > >  	.queue_rq	= scsi_mq_queue_rq,
> > >  	.map_queue	= blk_mq_map_queue,
> > > +	.timeout	= scsi_times_out,
> > >  	.alloc_hctx	= blk_mq_alloc_single_hw_queue,
> > >  	.free_hctx	= blk_mq_free_single_hw_queue,
> > >  };

So your actually triggering a blk-mq timeout with ata_piix..?

> > > diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
> > > index 65360db..33aa373 100644
> > > --- a/drivers/scsi/scsi_lib.c
> > > +++ b/drivers/scsi/scsi_lib.c
> > > @@ -283,7 +283,10 @@ int scsi_execute(struct scsi_device *sdev, const unsigned char *cmd,
> > >  	/*
> > >  	 * head injection *required* here otherwise quiesce won't work
> > >  	 */
> > > -	blk_execute_rq(req->q, NULL, req, 1);
> > > +	if (q->mq_ops)
> > > +		blk_mq_execute_rq(req->q, req);
> > > +	else
> > > +		blk_execute_rq(req->q, NULL, req, 1);
> > >  

The scsi_execute() -> REQ_TYPE_BLOCK_RQ special case (scsi_scan +
scsi_ioctl) has a small issue preventing it's conversion to use
blk_mq_execute_rq().

Namely that scsi_execute_cmd() currently expects to be able to check for
the existence of req->errors + req->resid_len *after*
blk_mq_execute_rq() -> blk_mq_finish_request() -> blk_mq_free_request()
has been called to mark the pre-allocated struct request as free for
blk-mq re-use.

That is why scsi-mq still uses blk_execute_rq() for this reason, and
this will need be addressed in order to safely use blk_mq_execute_rq()
in the above context.

> > >  	/*
> > >  	 * Some devices (USB mass-storage in particular) may transfer
> > > @@ -298,12 +301,8 @@ int scsi_execute(struct scsi_device *sdev, const unsigned char *cmd,
> > >  		*resid = req->resid_len;
> > >  	ret = req->errors;
> > >   out:
> > > -	if (q->mq_ops) {
> > > -		printk("scsi_execute(): Calling blk_mq_free_request >>>\n");
> > > -		blk_mq_free_request(req);
> > > -	} else {
> > > +	if (!q->mq_ops)
> > >  		blk_put_request(req);
> > > -	}
> > >  

Do you have an OOPs backtrace handy to post w/o the last two changes in
place..?

Also, just a heads up that so far I've been using IDE (/dev/hdX) for the
root-device in order to make scsi-mq debugging safer and easier.

I *very* much recommend doing the same if at all possible for ata_piix
scsi-mq development + testing, as you'll want to be very careful when
using a real file-system on top of this early alpha code.

--nab

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

[-- Attachment #1: Type: text/plain, Size: 1547 bytes --]

On Fri, Jul 12, 2013 at 10:20:12PM -0700, Nicholas A. Bellinger wrote:
> On Fri, 2013-07-12 at 09:46 +0200, Alexander Gordeev wrote:
> > > > diff --git a/drivers/scsi/scsi-mq.c b/drivers/scsi/scsi-mq.c
> > > > index ca6ff67..d8cc7a4 100644
> > > > --- a/drivers/scsi/scsi-mq.c
> > > > +++ b/drivers/scsi/scsi-mq.c
> > > > @@ -155,6 +155,7 @@ void scsi_mq_done(struct scsi_cmnd *sc)
> > > >  static struct blk_mq_ops scsi_mq_ops = {
> > > >  	.queue_rq	= scsi_mq_queue_rq,
> > > >  	.map_queue	= blk_mq_map_queue,
> > > > +	.timeout	= scsi_times_out,
> > > >  	.alloc_hctx	= blk_mq_alloc_single_hw_queue,
> > > >  	.free_hctx	= blk_mq_free_single_hw_queue,
> > > >  };
> 
> So your actually triggering a blk-mq timeout with ata_piix..?

No.
That is to avoid a NULL-pointer assignment from ->timeout elsewhere.
In fact I return -ENODEV for sr_probe() to not hit it.

> That is why scsi-mq still uses blk_execute_rq() for this reason, and
> this will need be addressed in order to safely use blk_mq_execute_rq()
> in the above context.

Got it.

> Do you have an OOPs backtrace handy to post w/o the last two changes in
> place..?

Attaching the output. No oops actually (due to aforementioned .timeout).

> I *very* much recommend doing the same if at all possible for ata_piix
> scsi-mq development + testing, as you'll want to be very careful when
> using a real file-system on top of this early alpha code.

Thank you for the warning.
Getting to writing CDBs would be of success :)

> --nab
> 

-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

[-- Attachment #2: output_for_nab.txt --]
[-- Type: text/plain, Size: 129996 bytes --]

Loading Fedora (3.10.0-rc5.nab+)
Loading initial ramdisk ...
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.10.0-rc5.nab+ (agordeev@dhcp47-148.lab.bos.redhat.com) (gcc version 4.7.2 20120921 (Red Hat 4.7.2-2) (GCC) ) #11 SMP Tue Jul 16 14:06:45 EDT 2013
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.10.0-rc5.nab+ root=/dev/mapper/vg_dhcp47--148-lv_root ro rd.md=0 rd.dm=0 KEYTABLE=us rd.lvm.lv=vg_dhcp47-148/lv_root console=ttyS0,115200 SYSFONT=True rd.luks=0 rd.lvm.lv=vg_dhcp47-148/lv_swap LANG=en_US.UTF-8 debug
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009bfff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009c000-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000008bd5cfff] usable
[    0.000000] BIOS-e820: [mem 0x000000008bd5d000-0x000000008be4efff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000008be4f000-0x000000008c26dfff] ACPI data
[    0.000000] BIOS-e820: [mem 0x000000008c26e000-0x000000008d66dfff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000008d66e000-0x000000008f601fff] ACPI data
[    0.000000] BIOS-e820: [mem 0x000000008f602000-0x000000008f64efff] reserved
[    0.000000] BIOS-e820: [mem 0x000000008f64f000-0x000000008f6e1fff] ACPI data
[    0.000000] BIOS-e820: [mem 0x000000008f6e2000-0x000000008f6ecfff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000008f6ed000-0x000000008f6effff] ACPI data
[    0.000000] BIOS-e820: [mem 0x000000008f6f0000-0x000000008f7cefff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000008f7cf000-0x000000008f7fffff] ACPI data
[    0.000000] BIOS-e820: [mem 0x000000008f800000-0x000000008fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000a0000000-0x00000000afffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fc000000-0x00000000fcffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000ff800000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000046fffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.5 present.
[    0.000000] DMI: Cisco Systems Inc R210-2121605W/R210-2121605W, BIOS C200.1.4.3j.0.020720132258 02/07/2013
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] No AGP bridge found
[    0.000000] e820: last_pfn = 0x470000 max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: uncachable
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-DFFFF uncachable
[    0.000000]   E0000-FFFFF write-protect
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 0000000000 mask FF80000000 write-back
[    0.000000]   1 base 0080000000 mask FFF0000000 write-back
[    0.000000]   2 base 0100000000 mask FF00000000 write-back
[    0.000000]   3 base 0200000000 mask FE00000000 write-back
[    0.000000]   4 base 0400000000 mask FFC0000000 write-back
[    0.000000]   5 base 0440000000 mask FFE0000000 write-back
[    0.000000]   6 base 0460000000 mask FFF0000000 write-back
[    0.000000]   7 base 00B0000000 mask FFFF000000 write-combining
[    0.000000]   8 disabled
[    0.000000]   9 disabled
[    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
[    0.000000] e820: last_pfn = 0x8bd5d max_arch_pfn = 0x400000000
[    0.000000] found SMP MP-table at [mem 0x000fc640-0x000fc64f] mapped at [ffff8800000fc640]
[    0.000000] Base memory trampoline at [ffff880000095000] 95000 size 24576
[    0.000000] Using GB pages for direct mapping
[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[    0.000000]  [mem 0x00000000-0x000fffff] page 4k
[    0.000000] BRK [0x01fba000, 0x01fbafff] PGTABLE
[    0.000000] BRK [0x01fbb000, 0x01fbbfff] PGTABLE
[    0.000000] BRK [0x01fbc000, 0x01fbcfff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0x46fe00000-0x46fffffff]
[    0.000000]  [mem 0x46fe00000-0x46fffffff] page 2M
[    0.000000] BRK [0x01fbd000, 0x01fbdfff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0x46c000000-0x46fdfffff]
[    0.000000]  [mem 0x46c000000-0x46fdfffff] page 2M
[    0.000000] init_memory_mapping: [mem 0x400000000-0x46bffffff]
[    0.000000]  [mem 0x400000000-0x43fffffff] page 1G
[    0.000000]  [mem 0x440000000-0x46bffffff] page 2M
[    0.000000] init_memory_mapping: [mem 0x00100000-0x8bd5cfff]
[    0.000000]  [mem 0x00100000-0x001fffff] page 4k
[    0.000000]  [mem 0x00200000-0x3fffffff] page 2M
[    0.000000]  [mem 0x40000000-0x7fffffff] page 1G
[    0.000000]  [mem 0x80000000-0x8bbfffff] page 2M
[    0.000000]  [mem 0x8bc00000-0x8bd5cfff] page 4k
[    0.000000] init_memory_mapping: [mem 0x100000000-0x3ffffffff]
[    0.000000]  [mem 0x100000000-0x3ffffffff] page 1G
[    0.000000] RAMDISK: [mem 0x363ca000-0x371dcfff]
[    0.000000] ACPI: RSDP 00000000000f0410 00024 (v02 Cisco0)
[    0.000000] ACPI: XSDT 000000008f7fe120 000A4 (v01 Cisco0 CiscoUCS 00000000      01000013)
[    0.000000] ACPI: FACP 000000008f7fc000 000F4 (v04 Cisco0 CiscoUCS 00000000 MSFT 0100000D)
[    0.000000] ACPI: DSDT 000000008f7f6000 05EC4 (v02 Cisco0 CiscoUCS 00000003 MSFT 0100000D)
[    0.000000] ACPI: FACS 000000008f6f0000 00040
[    0.000000] ACPI: APIC 000000008f7f5000 001A8 (v02 Cisco0 CiscoUCS 00000000 MSFT 0100000D)
[    0.000000] ACPI: MCFG 000000008f7f4000 0003C (v01 Cisco0 CiscoUCS 00000001 MSFT 0100000D)
[    0.000000] ACPI: HPET 000000008f7f3000 00038 (v01 Cisco0 CiscoUCS 00000001 MSFT 0100000D)
[    0.000000] ACPI: SLIT 000000008f7f2000 00030 (v01 Cisco0 CiscoUCS 00000001 MSFT 0100000D)
[    0.000000] ACPI: SRAT 000000008f7f1000 00430 (v02 Cisco0 CiscoUCS 00000001 MSFT 0100000D)
[    0.000000] ACPI: SPCR 000000008f7f0000 00050 (v01 Cisco0 CiscoUCS 00000000 MSFT 0100000D)
[    0.000000] ACPI: WDDT 000000008f7ef000 00040 (v01 Cisco0 CiscoUCS 00000000 MSFT 0100000D)
[    0.000000] ACPI: SSDT 000000008f7d4000 1AFC4 (v02  Cisco SSDT  PM 00004000 INTL 20090730)
[    0.000000] ACPI: SSDT 000000008f7d3000 001D8 (v02  Cisco IPMI     00004000 INTL 20090730)
[    0.000000] ACPI: SSDT 000000008f7d2000 00962 (v02 CISCO  PMETER   00004000 INTL 20090730)
[    0.000000] ACPI: HEST 000000008f7d0000 000A8 (v01 Cisco  CiscoTbl 00000001 CISC 00000001)
[    0.000000] ACPI: BERT 000000008f7cf000 00030 (v01 Cisco  CiscoTbl 00000001 CISC 00000001)
[    0.000000] ACPI: ERST 000000008f6ef000 00230 (v01 Cisco  CiscoTbl 00000001 CISC 00000001)
[    0.000000] ACPI: EINJ 000000008f6ee000 00130 (v01 Cisco  CiscoTbl 00000001 CISC 00000001)
[    0.000000] ACPI: DMAR 000000008f6ed000 001A8 (v01 Cisco0 CiscoUCS 00000001 MSFT 0100000D)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] SRAT: PXM 0 -> APIC 0x00 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x20 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x02 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x22 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x12 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x32 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x14 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x34 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x01 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x21 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x03 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x23 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x13 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x33 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x15 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x35 -> Node 1
[    0.000000] SRAT: Node 0 PXM 0 [mem 0x00000000-0x8fffffff]
[    0.000000] SRAT: Node 0 PXM 0 [mem 0x100000000-0x26fffffff]
[    0.000000] SRAT: Node 1 PXM 1 [mem 0x270000000-0x46fffffff]
[    0.000000] NUMA: Initialized distance table, cnt=2
[    0.000000] NUMA: Node 0 [mem 0x00000000-0x8fffffff] + [mem 0x100000000-0x26fffffff] -> [mem 0x00000000-0x26fffffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000-0x26fffffff]
[    0.000000]   NODE_DATA [mem 0x26ffec000-0x26fffffff]
[    0.000000] Initmem setup node 1 [mem 0x270000000-0x46fffffff]
[    0.000000]   NODE_DATA [mem 0x46ffe9000-0x46fffcfff]
[    0.000000]  [ffffea0000000000-ffffea0009bfffff] PMD -> [ffff880267e00000-ffff88026fdfffff] on node 0
[    0.000000]  [ffffea0009c00000-ffffea0011bfffff] PMD -> [ffff880467600000-ffff88046f5fffff] on node 1
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
[    0.000000]   Normal   [mem 0x100000000-0x46fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00001000-0x0009bfff]
[    0.000000]   node   0: [mem 0x00100000-0x8bd5cfff]
[    0.000000]   node   0: [mem 0x100000000-0x26fffffff]
[    0.000000]   node   1: [mem 0x270000000-0x46fffffff]
[    0.000000] On node 0 totalpages: 2079992
[    0.000000]   DMA zone: 64 pages used for memmap
[    0.000000]   DMA zone: 22 pages reserved
[    0.000000]   DMA zone: 3995 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 8886 pages used for memmap
[    0.000000]   DMA32 zone: 568669 pages, LIFO batch:31
[    0.000000]   Normal zone: 23552 pages used for memmap
[    0.000000]   Normal zone: 1507328 pages, LIFO batch:31
[    0.000000] On node 1 totalpages: 2097152
[    0.000000]   Normal zone: 32768 pages used for memmap
[    0.000000]   Normal zone: 2097152 pages, LIFO batch:31
[    0.000000] ACPI: PM-Timer IO Port: 0x408
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x20] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x22] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x12] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x32] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x14] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x34] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x01] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x21] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x03] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x23] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x13] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x33] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x15] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x35] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x10] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x11] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x12] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x13] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x14] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x15] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x16] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x17] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x05] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x06] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x07] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x08] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x09] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x0a] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x0b] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x0c] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x0d] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x0e] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x0f] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x10] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x11] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x12] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x13] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x14] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x15] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x16] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x17] high level lint[0x1])
[    0.000000] ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
[    0.000000] ACPI: IOAPIC (id[0x09] address[0xfec90000] gsi_base[24])
[    0.000000] IOAPIC[1]: apic_id 9, version 32, address 0xfec90000, GSI 24-47
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ2 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x8086a401 base: 0xfed00000
[    0.000000] smpboot: Allowing 24 CPUs, 8 hotplug CPUs
[    0.000000] nr_irqs_gsi: 64
[    0.000000] PM: Registered nosave memory: 000000000009c000 - 00000000000a0000
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
[    0.000000] PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
[    0.000000] PM: Registered nosave memory: 000000008bd5d000 - 000000008be4f000
[    0.000000] PM: Registered nosave memory: 000000008be4f000 - 000000008c26e000
[    0.000000] PM: Registered nosave memory: 000000008c26e000 - 000000008d66e000
[    0.000000] PM: Registered nosave memory: 000000008d66e000 - 000000008f602000
[    0.000000] PM: Registered nosave memory: 000000008f602000 - 000000008f64f000
[    0.000000] PM: Registered nosave memory: 000000008f64f000 - 000000008f6e2000
[    0.000000] PM: Registered nosave memory: 000000008f6e2000 - 000000008f6ed000
[    0.000000] PM: Registered nosave memory: 000000008f6ed000 - 000000008f6f0000
[    0.000000] PM: Registered nosave memory: 000000008f6f0000 - 000000008f7cf000
[    0.000000] PM: Registered nosave memory: 000000008f7cf000 - 000000008f800000
[    0.000000] PM: Registered nosave memory: 000000008f800000 - 0000000090000000
[    0.000000] PM: Registered nosave memory: 0000000090000000 - 00000000a0000000
[    0.000000] PM: Registered nosave memory: 00000000a0000000 - 00000000b0000000
[    0.000000] PM: Registered nosave memory: 00000000b0000000 - 00000000fc000000
[    0.000000] PM: Registered nosave memory: 00000000fc000000 - 00000000fd000000
[    0.000000] PM: Registered nosave memory: 00000000fd000000 - 00000000fed1c000
[    0.000000] PM: Registered nosave memory: 00000000fed1c000 - 00000000fed20000
[    0.000000] PM: Registered nosave memory: 00000000fed20000 - 00000000ff800000
[    0.000000] PM: Registered nosave memory: 00000000ff800000 - 0000000100000000
[    0.000000] e820: [mem 0xb0000000-0xfbffffff] available for PCI devices
[    0.000000] setup_percpu: NR_CPUS:128 nr_cpumask_bits:128 nr_cpu_ids:24 nr_node_ids:2
[    0.000000] PERCPU: Embedded 27 pages/cpu @ffff880267c00000 s81216 r8192 d21184 u131072
[    0.000000] pcpu-alloc: s81216 r8192 d21184 u131072 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 00 02 04 06 08 10 12 14 16 18 20 22 -- -- -- -- 
[    0.000000] pcpu-alloc: [1] 01 03 05 07 09 11 13 15 17 19 21 23 -- -- -- -- 
[    0.000000] Built 2 zonelists in Zone order, mobility grouping on.  Total pages: 4111852
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-rc5.nab+ root=/dev/mapper/vg_dhcp47--148-lv_root ro rd.md=0 rd.dm=0 KEYTABLE=us rd.lvm.lv=vg_dhcp47-148/lv_root console=ttyS0,115200 SYSFONT=True rd.luks=0 rd.lvm.lv=vg_dhcp47-148/lv_swap LANG=en_US.UTF-8 debug
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Memory: 16346740k/18612224k available (6396k kernel code, 1903648k absent, 361836k reserved, 6834k data, 1312k init)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=24, Nodes=2
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	RCU restricting CPUs from NR_CPUS=128 to nr_cpu_ids=24.
[    0.000000] NR_IRQS:8448 nr_irqs:1280 16
[    0.000000] Console: colour VGA+ 80x25
[    0.000000] console [ttyS0] enabled
[    0.000000] allocated 67108864 bytes of page_cgroup
[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
[    0.000000] Enabling automatic NUMA balancing. Configure with numa_balancing= or sysctl
[    0.000000] hpet clockevent registered
[    0.001000] tsc: Fast TSC calibration using PIT
[    0.002000] tsc: Detected 2400.120 MHz processor
[    0.000006] Calibrating delay loop (skipped), value calculated using timer frequency.. 4800.24 BogoMIPS (lpj=2400120)
[    0.011880] pid_max: default: 32768 minimum: 301
[    0.017103] Security Framework initialized
[    0.021690] SELinux:  Initializing.
[    0.025604] SELinux:  Starting in permissive mode
[    0.032441] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
[    0.044914] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[    0.054586] Mount-cache hash table entries: 256
[    0.059896] Initializing cgroup subsys memory
[    0.064787] Initializing cgroup subsys devices
[    0.069754] Initializing cgroup subsys freezer
[    0.074720] Initializing cgroup subsys net_cls
[    0.079686] Initializing cgroup subsys blkio
[    0.084459] Initializing cgroup subsys perf_event
[    0.089749] CPU: Physical Processor ID: 0
[    0.094230] CPU: Processor Core ID: 0
[    0.098326] mce: CPU supports 9 MCE banks
[    0.102814] CPU0: Thermal monitoring enabled (TM1)
[    0.108176] Last level iTLB entries: 4KB 512, 2MB 7, 4MB 7
[    0.108176] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32
[    0.108176] tlb_flushall_shift: 6
[    0.124388] Freeing SMP alternatives: 24k freed
[    0.129457] ACPI: Core revision 20130328
[    0.146166] ACPI: All ACPI Tables successfully acquired
[    0.157144] ftrace: allocating 24325 entries in 96 pages
[    0.175780] dmar: Host address width 40
[    0.180071] dmar: DRHD base: 0x000000fe710000 flags: 0x1
[    0.186014] dmar: IOMMU 0: reg_base_addr fe710000 ver 1:0 cap c90780106f0462 ecap f020fe
[    0.195056] dmar: RMRR base: 0x0000008f62f000 end: 0x0000008f631fff
[    0.202061] dmar: RMRR base: 0x0000008f61a000 end: 0x0000008f61afff
[    0.209065] dmar: RMRR base: 0x0000008f617000 end: 0x0000008f617fff
[    0.216068] dmar: RMRR base: 0x0000008f614000 end: 0x0000008f614fff
[    0.223072] dmar: RMRR base: 0x0000008f611000 end: 0x0000008f611fff
[    0.230077] dmar: RMRR base: 0x0000008f60e000 end: 0x0000008f60efff
[    0.237082] dmar: RMRR base: 0x0000008f60b000 end: 0x0000008f60bfff
[    0.244085] dmar: RMRR base: 0x0000008f608000 end: 0x0000008f608fff
[    0.251089] dmar: RMRR base: 0x0000008f605000 end: 0x0000008f605fff
[    0.258195] IOAPIC id 8 under DRHD base  0xfe710000 IOMMU 0
[    0.264407] IOAPIC id 9 under DRHD base  0xfe710000 IOMMU 0
[    0.270820] Enabled IRQ remapping in xapic mode
[    0.275886] Switched APIC routing to physical flat.
[    0.281928] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.298646] smpboot: CPU0: Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz (fam: 06, model: 2c, stepping: 02)
[    0.411839] Performance Events: PEBS fmt1+, 16-deep LBR, Westmere events, Intel PMU driver.
[    0.421215] perf_event_intel: CPUID marked event: 'bus cycles' unavailable
[    0.428893] ... version:                3
[    0.433367] ... bit width:              48
[    0.437940] ... generic registers:      4
[    0.442416] ... value mask:             0000ffffffffffff
[    0.448347] ... max period:             000000007fffffff
[    0.454279] ... fixed-purpose events:   3
[    0.458755] ... event mask:             000000070000000f
[    0.466272] smpboot: Booting Node   1, Processors  #1[    0.562463] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
 OK
[    0.571999] smpboot: Booting Node   0, Processors  #2 OK
[    0.591330] smpboot: Booting Node   1, Processors  #3 OK
[    0.610596] smpboot: Booting Node   0, Processors  #4 OK
[    0.629920] smpboot: Booting Node   1, Processors  #5 OK
[    0.649202] smpboot: Booting Node   0, Processors  #6 OK
[    0.668532] smpboot: Booting Node   1, Processors  #7 OK
[    0.687810] smpboot: Booting Node   0, Processors  #8 OK
[    0.707145] smpboot: Booting Node   1, Processors  #9 OK
[    0.726403] smpboot: Booting Node   0, Processors  #10 OK
[    0.745806] smpboot: Booting Node   1, Processors  #11 OK
[    0.765195] smpboot: Booting Node   0, Processors  #12 OK
[    0.784617] smpboot: Booting Node   1, Processors  #13 OK
[    0.803988] smpboot: Booting Node   0, Processors  #14 OK
[    0.823413] smpboot: Booting Node   1, Processors  #15
[    0.842210] Brought up 16 CPUs
[    0.845824] smpboot: Total of 16 processors activated (76801.07 BogoMIPS)
[    0.870916] devtmpfs: initialized
[    0.874904] PM: Registering ACPI NVS region [mem 0x8bd5d000-0x8be4efff] (991232 bytes)
[    0.883778] PM: Registering ACPI NVS region [mem 0x8c26e000-0x8d66dfff] (20971520 bytes)
[    0.893458] PM: Registering ACPI NVS region [mem 0x8f6e2000-0x8f6ecfff] (45056 bytes)
[    0.902205] PM: Registering ACPI NVS region [mem 0x8f6f0000-0x8f7cefff] (913408 bytes)
[    0.912322] atomic64 test passed for x86-64 platform with CX8 and with SSE
[    0.920026] RTC time: 14:17:10, date: 07/16/13
[    0.925053] NET: Registered protocol family 16
[    0.930189] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[    0.938647] ACPI: bus type PCI registered
[    0.943123] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    0.950399] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xa0000000-0xafffffff] (base 0xa0000000)
[    0.960797] PCI: MMCONFIG at [mem 0xa0000000-0xafffffff] reserved in E820
[    0.991889] PCI: Using configuration type 1 for base access
[    0.999361] bio: create slab <bio-0> at 0
[    1.004097] ACPI: Added _OSI(Module Device)
[    1.008770] ACPI: Added _OSI(Processor Device)
[    1.013734] ACPI: Added _OSI(3.0 _SCP Extensions)
[    1.018988] ACPI: Added _OSI(Processor Aggregator Device)
[    1.026534] ACPI: EC: Look up EC in DSDT
[    1.052643] ACPI: Interpreter enabled
[    1.056728] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20130328/hwxface-568)
[    1.067052] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20130328/hwxface-568)
[    1.077376] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S3_] (20130328/hwxface-568)
[    1.087701] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S4_] (20130328/hwxface-568)
[    1.098029] ACPI: (supports S0 S5)
[    1.101827] ACPI: Using IOAPIC for interrupt routing
[    1.107414] HEST: Table parsing has been initialized.
[    1.113056] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    1.131160] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-fd])
[    1.138489] acpi PNP0A08:00: ignoring host bridge window [mem 0x000c4000-0x000cbfff] (conflicts with Video ROM [mem 0x000c0000-0x000c7fff])
[    1.152629] PCI host bridge to bus 0000:00
[    1.157204] pci_bus 0000:00: root bus resource [bus 00-fd]
[    1.163332] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7]
[    1.170236] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff]
[    1.177139] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff]
[    1.184819] pci_bus 0000:00: root bus resource [mem 0xfed40000-0xfedfffff]
[    1.192499] pci_bus 0000:00: root bus resource [mem 0xb0000000-0xfdffffff]
[    1.200191] pci 0000:00:00.0: [8086:3406] type 00 class 0x060000
[    1.206952] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
[    1.213809] pci 0000:00:01.0: [8086:3408] type 01 class 0x060400
[    1.220570] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
[    1.227397] pci 0000:00:01.0: System wakeup disabled by ACPI
[    1.233751] pci 0000:00:02.0: [8086:3409] type 01 class 0x060400
[    1.240512] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
[    1.247373] pci 0000:00:03.0: [8086:340a] type 01 class 0x060400
[    1.254135] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
[    1.260963] pci 0000:00:03.0: System wakeup disabled by ACPI
[    1.267316] pci 0000:00:04.0: [8086:340b] type 01 class 0x060400
[    1.274075] pci 0000:00:04.0: PME# supported from D0 D3hot D3cold
[    1.280904] pci 0000:00:04.0: System wakeup disabled by ACPI
[    1.287257] pci 0000:00:05.0: [8086:340c] type 01 class 0x060400
[    1.294016] pci 0000:00:05.0: PME# supported from D0 D3hot D3cold
[    1.300845] pci 0000:00:05.0: System wakeup disabled by ACPI
[    1.307200] pci 0000:00:06.0: [8086:340d] type 01 class 0x060400
[    1.313960] pci 0000:00:06.0: PME# supported from D0 D3hot D3cold
[    1.320795] pci 0000:00:06.0: System wakeup disabled by ACPI
[    1.327150] pci 0000:00:07.0: [8086:340e] type 01 class 0x060400
[    1.333911] pci 0000:00:07.0: PME# supported from D0 D3hot D3cold
[    1.340750] pci 0000:00:07.0: System wakeup disabled by ACPI
[    1.347103] pci 0000:00:08.0: [8086:340f] type 01 class 0x060400
[    1.353863] pci 0000:00:08.0: PME# supported from D0 D3hot D3cold
[    1.360695] pci 0000:00:08.0: System wakeup disabled by ACPI
[    1.367050] pci 0000:00:09.0: [8086:3410] type 01 class 0x060400
[    1.373811] pci 0000:00:09.0: PME# supported from D0 D3hot D3cold
[    1.380645] pci 0000:00:09.0: System wakeup disabled by ACPI
[    1.386994] pci 0000:00:0a.0: [8086:3411] type 01 class 0x060400
[    1.393744] pci 0000:00:0a.0: PME# supported from D0 D3hot D3cold
[    1.400581] pci 0000:00:0a.0: System wakeup disabled by ACPI
[    1.406937] pci 0000:00:10.0: [8086:3425] type 00 class 0x080000
[    1.413778] pci 0000:00:10.1: [8086:3426] type 00 class 0x080000
[    1.420600] pci 0000:00:11.0: [8086:3427] type 00 class 0x080000
[    1.427436] pci 0000:00:11.1: [8086:3428] type 00 class 0x080000
[    1.434265] pci 0000:00:13.0: [8086:342d] type 00 class 0x080020
[    1.440984] pci 0000:00:13.0: reg 10: [mem 0xb2423000-0xb2423fff]
[    1.447833] pci 0000:00:13.0: PME# supported from D0 D3hot D3cold
[    1.454725] pci 0000:00:14.0: [8086:342e] type 00 class 0x080000
[    1.461561] pci 0000:00:14.1: [8086:3422] type 00 class 0x080000
[    1.468354] pci 0000:00:14.2: [8086:3423] type 00 class 0x080000
[    1.475189] pci 0000:00:14.3: [8086:3438] type 00 class 0x080000
[    1.482018] pci 0000:00:16.0: [8086:3430] type 00 class 0x088000
[    1.488739] pci 0000:00:16.0: reg 10: [mem 0xb2400000-0xb2403fff 64bit]
[    1.496222] pci 0000:00:16.1: [8086:3431] type 00 class 0x088000
[    1.502942] pci 0000:00:16.1: reg 10: [mem 0xb2404000-0xb2407fff 64bit]
[    1.510450] pci 0000:00:16.2: [8086:3432] type 00 class 0x088000
[    1.517170] pci 0000:00:16.2: reg 10: [mem 0xb2408000-0xb240bfff 64bit]
[    1.524685] pci 0000:00:16.3: [8086:3433] type 00 class 0x088000
[    1.531404] pci 0000:00:16.3: reg 10: [mem 0xb240c000-0xb240ffff 64bit]
[    1.538924] pci 0000:00:16.4: [8086:3429] type 00 class 0x088000
[    1.545645] pci 0000:00:16.4: reg 10: [mem 0xb2410000-0xb2413fff 64bit]
[    1.553167] pci 0000:00:16.5: [8086:342a] type 00 class 0x088000
[    1.559885] pci 0000:00:16.5: reg 10: [mem 0xb2414000-0xb2417fff 64bit]
[    1.567398] pci 0000:00:16.6: [8086:342b] type 00 class 0x088000
[    1.574117] pci 0000:00:16.6: reg 10: [mem 0xb2418000-0xb241bfff 64bit]
[    1.581639] pci 0000:00:16.7: [8086:342c] type 00 class 0x088000
[    1.588357] pci 0000:00:16.7: reg 10: [mem 0xb241c000-0xb241ffff 64bit]
[    1.595878] pci 0000:00:1a.0: [8086:3a37] type 00 class 0x0c0300
[    1.602627] pci 0000:00:1a.0: reg 20: [io  0x50c0-0x50df]
[    1.608742] pci 0000:00:1a.0: System wakeup disabled by ACPI
[    1.615104] pci 0000:00:1a.1: [8086:3a38] type 00 class 0x0c0300
[    1.621853] pci 0000:00:1a.1: reg 20: [io  0x50a0-0x50bf]
[    1.627965] pci 0000:00:1a.1: System wakeup disabled by ACPI
[    1.634321] pci 0000:00:1a.2: [8086:3a39] type 00 class 0x0c0300
[    1.641069] pci 0000:00:1a.2: reg 20: [io  0x5080-0x509f]
[    1.647181] pci 0000:00:1a.2: System wakeup disabled by ACPI
[    1.653546] pci 0000:00:1a.7: [8086:3a3c] type 00 class 0x0c0320
[    1.660273] pci 0000:00:1a.7: reg 10: [mem 0xb2421000-0xb24213ff]
[    1.667162] pci 0000:00:1a.7: PME# supported from D0 D3hot D3cold
[    1.674027] pci 0000:00:1a.7: System wakeup disabled by ACPI
[    1.680384] pci 0000:00:1c.0: [8086:3a40] type 01 class 0x060400
[    1.687158] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
[    1.694019] pci 0000:00:1c.0: System wakeup disabled by ACPI
[    1.700383] pci 0000:00:1c.4: [8086:3a48] type 01 class 0x060400
[    1.707157] pci 0000:00:1c.4: PME# supported from D0 D3hot D3cold
[    1.714011] pci 0000:00:1c.4: System wakeup disabled by ACPI
[    1.720371] pci 0000:00:1c.5: [8086:3a4a] type 01 class 0x060400
[    1.727145] pci 0000:00:1c.5: PME# supported from D0 D3hot D3cold
[    1.734002] pci 0000:00:1c.5: System wakeup disabled by ACPI
[    1.740363] pci 0000:00:1d.0: [8086:3a34] type 00 class 0x0c0300
[    1.747111] pci 0000:00:1d.0: reg 20: [io  0x5060-0x507f]
[    1.753234] pci 0000:00:1d.0: System wakeup disabled by ACPI
[    1.759591] pci 0000:00:1d.1: [8086:3a35] type 00 class 0x0c0300
[    1.766339] pci 0000:00:1d.1: reg 20: [io  0x5040-0x505f]
[    1.772472] pci 0000:00:1d.1: System wakeup disabled by ACPI
[    1.778829] pci 0000:00:1d.2: [8086:3a36] type 00 class 0x0c0300
[    1.785568] pci 0000:00:1d.2: reg 20: [io  0x5020-0x503f]
[    1.791691] pci 0000:00:1d.2: System wakeup disabled by ACPI
[    1.798062] pci 0000:00:1d.7: [8086:3a3a] type 00 class 0x0c0320
[    1.804790] pci 0000:00:1d.7: reg 10: [mem 0xb2420000-0xb24203ff]
[    1.811679] pci 0000:00:1d.7: PME# supported from D0 D3hot D3cold
[    1.818557] pci 0000:00:1d.7: System wakeup disabled by ACPI
[    1.824915] pci 0000:00:1e.0: [8086:244e] type 01 class 0x060401
[    1.831718] pci 0000:00:1e.0: System wakeup disabled by ACPI
[    1.838076] pci 0000:00:1f.0: [8086:3a16] type 00 class 0x060100
[    1.844858] pci 0000:00:1f.0: quirk: [io  0x0400-0x047f] claimed by ICH6 ACPI/GPIO/TCO
[    1.853704] pci 0000:00:1f.0: quirk: [io  0x0500-0x053f] claimed by ICH6 GPIO
[    1.861668] pci 0000:00:1f.0: ICH7 LPC Generic IO decode 1 PIO at 0680 (mask 000f)
[    1.870124] pci 0000:00:1f.0: ICH7 LPC Generic IO decode 2 PIO at 0ca0 (mask 000f)
[    1.878582] pci 0000:00:1f.0: ICH7 LPC Generic IO decode 3 PIO at 0600 (mask 003f)
[    1.887135] pci 0000:00:1f.2: [8086:3a20] type 00 class 0x01018f
[    1.893858] pci 0000:00:1f.2: reg 10: [io  0x5138-0x513f]
[    1.899893] pci 0000:00:1f.2: reg 14: [io  0x514c-0x514f]
[    1.905928] pci 0000:00:1f.2: reg 18: [io  0x5130-0x5137]
[    1.911964] pci 0000:00:1f.2: reg 1c: [io  0x5148-0x514b]
[    1.917999] pci 0000:00:1f.2: reg 20: [io  0x5110-0x511f]
[    1.924035] pci 0000:00:1f.2: reg 24: [io  0x5100-0x510f]
[    1.930182] pci 0000:00:1f.3: [8086:3a30] type 00 class 0x0c0500
[    1.936905] pci 0000:00:1f.3: reg 10: [mem 0xb2422000-0xb24220ff 64bit]
[    1.944310] pci 0000:00:1f.3: reg 20: [io  0x5000-0x501f]
[    1.950443] pci 0000:00:1f.5: [8086:3a26] type 00 class 0x010185
[    1.957167] pci 0000:00:1f.5: reg 10: [io  0x5128-0x512f]
[    1.963203] pci 0000:00:1f.5: reg 14: [io  0x5144-0x5147]
[    1.969238] pci 0000:00:1f.5: reg 18: [io  0x5120-0x5127]
[    1.975272] pci 0000:00:1f.5: reg 1c: [io  0x5140-0x5143]
[    1.981307] pci 0000:00:1f.5: reg 20: [io  0x50f0-0x50ff]
[    1.987342] pci 0000:00:1f.5: reg 24: [io  0x50e0-0x50ef]
[    1.993531] pci 0000:00:01.0: PCI bridge to [bus 01]
[    1.999131] pci 0000:00:02.0: PCI bridge to [bus 02]
[    2.004722] pci 0000:00:03.0: PCI bridge to [bus 03]
[    2.010319] pci 0000:00:04.0: PCI bridge to [bus 04]
[    2.015921] pci 0000:05:00.0: [10b5:8624] type 01 class 0x060400
[    2.022639] pci 0000:05:00.0: reg 10: [mem 0xb2300000-0xb231ffff]
[    2.029499] pci 0000:05:00.0: PME# supported from D0 D3hot D3cold
[    2.036635] pci 0000:00:05.0: PCI bridge to [bus 05-14]
[    2.042473] pci 0000:00:05.0:   bridge window [io  0x3000-0x4fff]
[    2.049272] pci 0000:00:05.0:   bridge window [mem 0xb1c00000-0xb23fffff]
[    2.056916] pci 0000:06:04.0: [10b5:8624] type 01 class 0x060400
[    2.063685] pci 0000:06:04.0: PME# supported from D0 D3hot D3cold
[    2.070542] pci 0000:06:05.0: [10b5:8624] type 01 class 0x060400
[    2.077311] pci 0000:06:05.0: PME# supported from D0 D3hot D3cold
[    2.084172] pci 0000:06:08.0: [10b5:8624] type 01 class 0x060400
[    2.090941] pci 0000:06:08.0: PME# supported from D0 D3hot D3cold
[    2.097803] pci 0000:05:00.0: PCI bridge to [bus 06-14]
[    2.103644] pci 0000:05:00.0:   bridge window [io  0x3000-0x4fff]
[    2.110451] pci 0000:05:00.0:   bridge window [mem 0xb1c00000-0xb22fffff]
[    2.118084] pci 0000:06:04.0: PCI bridge to [bus 07]
[    2.123688] pci 0000:06:05.0: PCI bridge to [bus 08]
[    2.129301] pci 0000:09:00.0: [10b5:8632] type 01 class 0x060400
[    2.136023] pci 0000:09:00.0: reg 10: [mem 0xb2200000-0xb221ffff]
[    2.142899] pci 0000:09:00.0: PME# supported from D0 D3hot D3cold
[    2.151136] pci 0000:06:08.0: PCI bridge to [bus 09-14]
[    2.156976] pci 0000:06:08.0:   bridge window [io  0x3000-0x4fff]
[    2.163786] pci 0000:06:08.0:   bridge window [mem 0xb1c00000-0xb22fffff]
[    2.171448] pci 0000:0a:01.0: [10b5:8632] type 01 class 0x060400
[    2.178228] pci 0000:0a:01.0: PME# supported from D0 D3hot D3cold
[    2.185094] pci 0000:0a:04.0: [10b5:8632] type 01 class 0x060400
[    2.191881] pci 0000:0a:04.0: PME# supported from D0 D3hot D3cold
[    2.198760] pci 0000:09:00.0: PCI bridge to [bus 0a-14]
[    2.204600] pci 0000:09:00.0:   bridge window [io  0x3000-0x4fff]
[    2.211408] pci 0000:09:00.0:   bridge window [mem 0xb1c00000-0xb21fffff]
[    2.219051] pci 0000:0a:01.0: PCI bridge to [bus 0b]
[    2.224681] pci 0000:0c:00.0: [1137:0023] type 01 class 0x060400
[    2.231479] pci 0000:0c:00.0: PME# supported from D0 D3hot D3cold
[    2.240201] pci 0000:0a:04.0: PCI bridge to [bus 0c-14]
[    2.246042] pci 0000:0a:04.0:   bridge window [io  0x3000-0x4fff]
[    2.252851] pci 0000:0a:04.0:   bridge window [mem 0xb1c00000-0xb21fffff]
[    2.260544] pci 0000:0d:00.0: [1137:0041] type 01 class 0x060400
[    2.267330] pci 0000:0d:00.0: reg 10: [mem 0xb2100000-0xb210ffff 64bit]
[    2.274816] pci 0000:0d:00.0: PME# supported from D0 D3hot D3cold
[    2.281696] pci 0000:0d:01.0: [1137:0041] type 01 class 0x060400
[    2.288549] pci 0000:0d:01.0: PME# supported from D0 D3hot D3cold
[    2.295442] pci 0000:0c:00.0: PCI bridge to [bus 0d-14]
[    2.301282] pci 0000:0c:00.0:   bridge window [io  0x3000-0x4fff]
[    2.308092] pci 0000:0c:00.0:   bridge window [mem 0xb1c00000-0xb21fffff]
[    2.315782] pci 0000:0e:00.0: [1137:0042] type 00 class 0x00ff00
[    2.322563] pci 0000:0e:00.0: reg 10: [mem 0xb2002000-0xb2002fff 64bit]
[    2.330008] pci 0000:0e:00.0: reg 18: [mem 0xb2000000-0xb2001fff 64bit]
[    2.337521] pci 0000:0e:00.0: reg 30: [mem 0xffff8000-0xffffffff pref]
[    2.344955] pci 0000:0d:00.0: PCI bridge to [bus 0e]
[    2.350505] pci 0000:0d:00.0:   bridge window [mem 0xb2000000-0xb20fffff]
[    2.358187] pci 0000:0f:00.0: [1137:0040] type 01 class 0x060400
[    2.365042] pci 0000:0f:00.0: PME# supported from D0 D3hot D3cold
[    2.371903] pci 0000:0d:01.0: PCI bridge to [bus 0f-14]
[    2.377742] pci 0000:0d:01.0:   bridge window [io  0x3000-0x4fff]
[    2.384553] pci 0000:0d:01.0:   bridge window [mem 0xb1c00000-0xb1ffffff]
[    2.392227] pci 0000:10:00.0: [1137:0041] type 01 class 0x060400
[    2.399075] pci 0000:10:00.0: PME# supported from D0 D3hot D3cold
[    2.405948] pci 0000:10:01.0: [1137:0041] type 01 class 0x060400
[    2.412812] pci 0000:10:01.0: PME# supported from D0 D3hot D3cold
[    2.419685] pci 0000:10:02.0: [1137:0041] type 01 class 0x060400
[    2.426533] pci 0000:10:02.0: PME# supported from D0 D3hot D3cold
[    2.433419] pci 0000:10:03.0: [1137:0041] type 01 class 0x060400
[    2.440265] pci 0000:10:03.0: PME# supported from D0 D3hot D3cold
[    2.447158] pci 0000:0f:00.0: PCI bridge to [bus 10-14]
[    2.452996] pci 0000:0f:00.0:   bridge window [io  0x3000-0x4fff]
[    2.459796] pci 0000:0f:00.0:   bridge window [mem 0xb1c00000-0xb1ffffff]
[    2.467471] pci 0000:11:00.0: [1137:0043] type 00 class 0x020000
[    2.474260] pci 0000:11:00.0: reg 10: [mem 0xb1f00000-0xb1f07fff 64bit]
[    2.481714] pci 0000:11:00.0: reg 18: [mem 0xb1f08000-0xb1f09fff 64bit]
[    2.489155] pci 0000:11:00.0: reg 20: [io  0x4000-0x407f]
[    2.495286] pci 0000:11:00.0: reg 30: [mem 0xffff0000-0xffffffff pref]
[    2.502733] pci 0000:10:00.0: PCI bridge to [bus 11]
[    2.508281] pci 0000:10:00.0:   bridge window [io  0x4000-0x4fff]
[    2.515090] pci 0000:10:00.0:   bridge window [mem 0xb1f00000-0xb1ffffff]
[    2.522777] pci 0000:12:00.0: [1137:0043] type 00 class 0x020000
[    2.529564] pci 0000:12:00.0: reg 10: [mem 0xb1e00000-0xb1e07fff 64bit]
[    2.537016] pci 0000:12:00.0: reg 18: [mem 0xb1e08000-0xb1e09fff 64bit]
[    2.544460] pci 0000:12:00.0: reg 20: [io  0x3000-0x307f]
[    2.550591] pci 0000:12:00.0: reg 30: [mem 0xffff0000-0xffffffff pref]
[    2.558034] pci 0000:10:01.0: PCI bridge to [bus 12]
[    2.563584] pci 0000:10:01.0:   bridge window [io  0x3000-0x3fff]
[    2.570394] pci 0000:10:01.0:   bridge window [mem 0xb1e00000-0xb1efffff]
[    2.578083] pci 0000:13:00.0: [1137:0045] type 00 class 0x0c0400
[    2.584875] pci 0000:13:00.0: reg 10: [mem 0xb1d00000-0xb1d07fff 64bit]
[    2.592323] pci 0000:13:00.0: reg 18: [mem 0xb1d08000-0xb1d09fff 64bit]
[    2.600010] pci 0000:10:02.0: PCI bridge to [bus 13]
[    2.605561] pci 0000:10:02.0:   bridge window [mem 0xb1d00000-0xb1dfffff]
[    2.613255] pci 0000:14:00.0: [1137:0045] type 00 class 0x0c0400
[    2.620041] pci 0000:14:00.0: reg 10: [mem 0xb1c00000-0xb1c07fff 64bit]
[    2.627491] pci 0000:14:00.0: reg 18: [mem 0xb1c08000-0xb1c09fff 64bit]
[    2.635158] pci 0000:10:03.0: PCI bridge to [bus 14]
[    2.640708] pci 0000:10:03.0:   bridge window [mem 0xb1c00000-0xb1cfffff]
[    2.648476] pci 0000:00:06.0: PCI bridge to [bus 15]
[    2.654095] pci 0000:16:00.0: [1077:2432] type 00 class 0x0c0400
[    2.660816] pci 0000:16:00.0: reg 10: [io  0x2200-0x22ff]
[    2.666858] pci 0000:16:00.0: reg 14: [mem 0xb1b00000-0xb1b03fff 64bit]
[    2.674275] pci 0000:16:00.0: reg 30: [mem 0xfffc0000-0xffffffff pref]
[    2.681658] pci 0000:16:00.1: [1077:2432] type 00 class 0x0c0400
[    2.688379] pci 0000:16:00.1: reg 10: [io  0x2000-0x20ff]
[    2.694421] pci 0000:16:00.1: reg 14: [mem 0xb1b04000-0xb1b07fff 64bit]
[    2.701839] pci 0000:16:00.1: reg 30: [mem 0xfffc0000-0xffffffff pref]
[    2.709497] pci 0000:00:07.0: PCI bridge to [bus 16]
[    2.715035] pci 0000:00:07.0:   bridge window [io  0x2000-0x2fff]
[    2.721842] pci 0000:00:07.0:   bridge window [mem 0xb1b00000-0xb1bfffff]
[    2.729478] pci 0000:00:08.0: PCI bridge to [bus 17]
[    2.735072] pci 0000:00:09.0: PCI bridge to [bus 18]
[    2.740669] pci 0000:00:0a.0: PCI bridge to [bus 19]
[    2.746295] pci 0000:1a:00.0: [8086:10c9] type 00 class 0x020000
[    2.753018] pci 0000:1a:00.0: reg 10: [mem 0xb1960000-0xb197ffff]
[    2.759835] pci 0000:1a:00.0: reg 14: [mem 0xb1940000-0xb195ffff]
[    2.766652] pci 0000:1a:00.0: reg 18: [io  0x1020-0x103f]
[    2.772691] pci 0000:1a:00.0: reg 1c: [mem 0xb1a04000-0xb1a07fff]
[    2.779526] pci 0000:1a:00.0: reg 30: [mem 0xfffe0000-0xffffffff pref]
[    2.786874] pci 0000:1a:00.0: PME# supported from D0 D3hot D3cold
[    2.793723] pci 0000:1a:00.0: reg 184: [mem 0xb1980000-0xb1983fff 64bit]
[    2.801229] pci 0000:1a:00.0: reg 190: [mem 0xb19a0000-0xb19a3fff 64bit]
[    2.808786] pci 0000:1a:00.1: [8086:10c9] type 00 class 0x020000
[    2.815500] pci 0000:1a:00.1: reg 10: [mem 0xb1920000-0xb193ffff]
[    2.822316] pci 0000:1a:00.1: reg 14: [mem 0xb1900000-0xb191ffff]
[    2.829133] pci 0000:1a:00.1: reg 18: [io  0x1000-0x101f]
[    2.835163] pci 0000:1a:00.1: reg 1c: [mem 0xb1a00000-0xb1a03fff]
[    2.841997] pci 0000:1a:00.1: reg 30: [mem 0xfffe0000-0xffffffff pref]
[    2.849337] pci 0000:1a:00.1: PME# supported from D0 D3hot D3cold
[    2.856185] pci 0000:1a:00.1: reg 184: [mem 0xb19c0000-0xb19c3fff 64bit]
[    2.863692] pci 0000:1a:00.1: reg 190: [mem 0xb19e0000-0xb19e3fff 64bit]
[    2.872673] pci 0000:00:1c.0: PCI bridge to [bus 1a-1c]
[    2.878513] pci 0000:00:1c.0:   bridge window [io  0x1000-0x1fff]
[    2.885322] pci 0000:00:1c.0:   bridge window [mem 0xb1900000-0xb1afffff]
[    2.892986] pci 0000:1d:00.0: [102b:0522] type 00 class 0x030000
[    2.899715] pci 0000:1d:00.0: reg 10: [mem 0xb0000000-0xb0ffffff pref]
[    2.907011] pci 0000:1d:00.0: reg 14: [mem 0xb1800000-0xb1803fff]
[    2.913829] pci 0000:1d:00.0: reg 18: [mem 0xb1000000-0xb17fffff]
[    2.920686] pci 0000:1d:00.0: reg 30: [mem 0xffff0000-0xffffffff pref]
[    2.929714] pci 0000:00:1c.4: PCI bridge to [bus 1d]
[    2.935264] pci 0000:00:1c.4:   bridge window [mem 0xb1000000-0xb18fffff]
[    2.942852] pci 0000:00:1c.4:   bridge window [mem 0xb0000000-0xb0ffffff 64bit pref]
[    2.951551] pci 0000:00:1c.5: PCI bridge to [bus 1e]
[    2.957175] pci 0000:00:1e.0: PCI bridge to [bus 1f] (subtractive decode)
[    2.964766] pci 0000:00:1e.0:   bridge window [io  0x0000-0x0cf7] (subtractive decode)
[    2.973609] pci 0000:00:1e.0:   bridge window [io  0x0d00-0xffff] (subtractive decode)
[    2.982454] pci 0000:00:1e.0:   bridge window [mem 0x000a0000-0x000bffff] (subtractive decode)
[    2.992074] pci 0000:00:1e.0:   bridge window [mem 0xfed40000-0xfedfffff] (subtractive decode)
[    3.001693] pci 0000:00:1e.0:   bridge window [mem 0xb0000000-0xfdffffff] (subtractive decode)
[    3.011408] acpi PNP0A08:00: Requesting ACPI _OSC control (0x1d)
[    3.018207] acpi PNP0A08:00: ACPI _OSC control (0x1d) granted
[    3.024769] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 *11 12 14 15)
[    3.033015] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 *10 11 12 14 15)
[    3.041259] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 *9 10 11 12 14 15)
[    3.049503] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 *5 6 7 9 10 11 12 14 15)
[    3.057750] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled.
[    3.067280] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 *11 12 14 15)
[    3.075520] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled.
[    3.085046] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled.
[    3.094830] acpi root: \_SB_.PCI0 notify handler is installed
[    3.101284] Found 1 acpi root devices
[    3.105475] ACPI: No dock devices found.
[    3.109940] vgaarb: device added: PCI:0000:1d:00.0,decodes=io+mem,owns=io+mem,locks=none
[    3.118980] vgaarb: loaded
[    3.122002] vgaarb: bridge control possible 0000:1d:00.0
[    3.128017] SCSI subsystem initialized
[    3.132204] ACPI: bus type ATA registered
[    3.136744] libata version 3.00 loaded.
[    3.141056] ACPI: bus type USB registered
[    3.145539] usbcore: registered new interface driver usbfs
[    3.151672] usbcore: registered new interface driver hub
[    3.157632] usbcore: registered new device driver usb
[    3.163406] PCI: Using ACPI for IRQ routing
[    3.173349] PCI: Discovered peer bus fe
[    3.177631] PCI: root bus fe: using default resources
[    3.183272] PCI: Probing PCI hardware (bus fe)
[    3.188259] PCI host bridge to bus 0000:fe
[    3.192834] pci_bus 0000:fe: root bus resource [io  0x0000-0xffff]
[    3.199739] pci_bus 0000:fe: root bus resource [mem 0x00000000-0xffffffffff]
[    3.207612] pci_bus 0000:fe: No busn resource found for root bus, will use [bus fe-ff]
[    3.216459] pci 0000:fe:00.0: [8086:2c70] type 00 class 0x060000
[    3.223212] pci 0000:fe:00.1: [8086:2d81] type 00 class 0x060000
[    3.229967] pci 0000:fe:02.0: [8086:2d90] type 00 class 0x060000
[    3.236718] pci 0000:fe:02.1: [8086:2d91] type 00 class 0x060000
[    3.243472] pci 0000:fe:02.2: [8086:2d92] type 00 class 0x060000
[    3.250222] pci 0000:fe:02.3: [8086:2d93] type 00 class 0x060000
[    3.256973] pci 0000:fe:02.4: [8086:2d94] type 00 class 0x060000
[    3.263723] pci 0000:fe:02.5: [8086:2d95] type 00 class 0x060000
[    3.270475] pci 0000:fe:03.0: [8086:2d98] type 00 class 0x060000
[    3.277226] pci 0000:fe:03.1: [8086:2d99] type 00 class 0x060000
[    3.283979] pci 0000:fe:03.2: [8086:2d9a] type 00 class 0x060000
[    3.290732] pci 0000:fe:03.4: [8086:2d9c] type 00 class 0x060000
[    3.297484] pci 0000:fe:04.0: [8086:2da0] type 00 class 0x060000
[    3.304232] pci 0000:fe:04.1: [8086:2da1] type 00 class 0x060000
[    3.310982] pci 0000:fe:04.2: [8086:2da2] type 00 class 0x060000
[    3.317724] pci 0000:fe:04.3: [8086:2da3] type 00 class 0x060000
[    3.324473] pci 0000:fe:05.0: [8086:2da8] type 00 class 0x060000
[    3.331220] pci 0000:fe:05.1: [8086:2da9] type 00 class 0x060000
[    3.337967] pci 0000:fe:05.2: [8086:2daa] type 00 class 0x060000
[    3.344715] pci 0000:fe:05.3: [8086:2dab] type 00 class 0x060000
[    3.351466] pci 0000:fe:06.0: [8086:2db0] type 00 class 0x060000
[    3.358212] pci 0000:fe:06.1: [8086:2db1] type 00 class 0x060000
[    3.364949] pci 0000:fe:06.2: [8086:2db2] type 00 class 0x060000
[    3.371687] pci 0000:fe:06.3: [8086:2db3] type 00 class 0x060000
[    3.378449] pci_bus 0000:fe: busn_res: [bus fe-ff] end is updated to fe
[    3.385841] PCI: Discovered peer bus ff
[    3.390124] PCI: root bus ff: using default resources
[    3.395763] PCI: Probing PCI hardware (bus ff)
[    3.400748] PCI host bridge to bus 0000:ff
[    3.405322] pci_bus 0000:ff: root bus resource [io  0x0000-0xffff]
[    3.412226] pci_bus 0000:ff: root bus resource [mem 0x00000000-0xffffffffff]
[    3.420101] pci_bus 0000:ff: No busn resource found for root bus, will use [bus ff-ff]
[    3.428948] pci 0000:ff:00.0: [8086:2c70] type 00 class 0x060000
[    3.435690] pci 0000:ff:00.1: [8086:2d81] type 00 class 0x060000
[    3.442439] pci 0000:ff:02.0: [8086:2d90] type 00 class 0x060000
[    3.449183] pci 0000:ff:02.1: [8086:2d91] type 00 class 0x060000
[    3.455929] pci 0000:ff:02.2: [8086:2d92] type 00 class 0x060000
[    3.462666] pci 0000:ff:02.3: [8086:2d93] type 00 class 0x060000
[    3.469415] pci 0000:ff:02.4: [8086:2d94] type 00 class 0x060000
[    3.476160] pci 0000:ff:02.5: [8086:2d95] type 00 class 0x060000
[    3.482904] pci 0000:ff:03.0: [8086:2d98] type 00 class 0x060000
[    3.489650] pci 0000:ff:03.1: [8086:2d99] type 00 class 0x060000
[    3.496394] pci 0000:ff:03.2: [8086:2d9a] type 00 class 0x060000
[    3.503141] pci 0000:ff:03.4: [8086:2d9c] type 00 class 0x060000
[    3.509890] pci 0000:ff:04.0: [8086:2da0] type 00 class 0x060000
[    3.516631] pci 0000:ff:04.1: [8086:2da1] type 00 class 0x060000
[    3.523375] pci 0000:ff:04.2: [8086:2da2] type 00 class 0x060000
[    3.530123] pci 0000:ff:04.3: [8086:2da3] type 00 class 0x060000
[    3.536870] pci 0000:ff:05.0: [8086:2da8] type 00 class 0x060000
[    3.543615] pci 0000:ff:05.1: [8086:2da9] type 00 class 0x060000
[    3.550365] pci 0000:ff:05.2: [8086:2daa] type 00 class 0x060000
[    3.557114] pci 0000:ff:05.3: [8086:2dab] type 00 class 0x060000
[    3.563860] pci 0000:ff:06.0: [8086:2db0] type 00 class 0x060000
[    3.570605] pci 0000:ff:06.1: [8086:2db1] type 00 class 0x060000
[    3.577351] pci 0000:ff:06.2: [8086:2db2] type 00 class 0x060000
[    3.584098] pci 0000:ff:06.3: [8086:2db3] type 00 class 0x060000
[    3.590854] pci_bus 0000:ff: busn_res: [bus ff] end is updated to ff
[    3.597954] PCI: pci_cache_line_size set to 64 bytes
[    3.603770] e820: reserve RAM buffer [mem 0x0009c000-0x0009ffff]
[    3.610478] e820: reserve RAM buffer [mem 0x8bd5d000-0x8bffffff]
[    3.617275] NetLabel: Initializing
[    3.621064] NetLabel:  domain hash size = 128
[    3.625929] NetLabel:  protocols = UNLABELED CIPSOv4
[    3.631482] NetLabel:  unlabeled traffic allowed by default
[    3.637747] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0
[    3.643534] hpet0: 4 comparators, 64-bit 14.318180 MHz counter
[    3.652074] Switching to clocksource hpet
[    3.663947] pnp: PnP ACPI init
[    3.667378] ACPI: bus type PNP registered
[    3.671942] pnp 00:00: Plug and Play ACPI device, IDs PNP0003 (active)
[    3.679412] pnp 00:01: [dma 4]
[    3.682852] pnp 00:01: Plug and Play ACPI device, IDs PNP0200 (active)
[    3.690190] pnp 00:02: Plug and Play ACPI device, IDs PNP0b00 (active)
[    3.697515] pnp 00:03: Plug and Play ACPI device, IDs PNP0c04 (active)
[    3.704839] pnp 00:04: Plug and Play ACPI device, IDs PNP0800 (active)
[    3.712201] pnp 00:05: Plug and Play ACPI device, IDs PNP0103 (active)
[    3.719662] system 00:06: [io  0x0500-0x057f] could not be reserved
[    3.726664] system 00:06: [io  0x0400-0x047f] has been reserved
[    3.733271] system 00:06: [io  0x0800-0x081f] has been reserved
[    3.739885] system 00:06: [io  0x0ca2-0x0ca3] has been reserved
[    3.746490] system 00:06: [io  0x0600-0x063f] has been reserved
[    3.753108] system 00:06: [io  0x0880-0x0883] has been reserved
[    3.759723] system 00:06: [io  0x0ca4-0x0ca5] has been reserved
[    3.766339] system 00:06: [mem 0xfed1c000-0xfed3fffe] could not be reserved
[    3.774118] system 00:06: [mem 0xff000000-0xffffffff] could not be reserved
[    3.781897] system 00:06: [mem 0xfee00000-0xfeefffff] has been reserved
[    3.789288] system 00:06: [mem 0xfe900000-0xfe90001f] has been reserved
[    3.796678] system 00:06: [mem 0xfea00000-0xfea0001f] has been reserved
[    3.804068] system 00:06: [mem 0xfed1b000-0xfed1bfff] has been reserved
[    3.811459] system 00:06: Plug and Play ACPI device, IDs PNP0c02 (active)
[    3.819214] pnp 00:07: Plug and Play ACPI device, IDs PNP0501 (active)
[    3.826573] pnp 00:08: Plug and Play ACPI device, IDs IPI0001 (active)
[    3.833960] pnp: PnP ACPI: found 9 devices
[    3.838536] ACPI: bus type PNP unregistered
[    3.849623] pci 0000:0e:00.0: no compatible bridge window for [mem 0xffff8000-0xffffffff pref]
[    3.859248] pci 0000:11:00.0: no compatible bridge window for [mem 0xffff0000-0xffffffff pref]
[    3.868871] pci 0000:12:00.0: no compatible bridge window for [mem 0xffff0000-0xffffffff pref]
[    3.878495] pci 0000:16:00.0: no compatible bridge window for [mem 0xfffc0000-0xffffffff pref]
[    3.888117] pci 0000:16:00.1: no compatible bridge window for [mem 0xfffc0000-0xffffffff pref]
[    3.897743] pci 0000:1a:00.0: no compatible bridge window for [mem 0xfffe0000-0xffffffff pref]
[    3.907366] pci 0000:1a:00.1: no compatible bridge window for [mem 0xfffe0000-0xffffffff pref]
[    3.916987] pci 0000:1d:00.0: no compatible bridge window for [mem 0xffff0000-0xffffffff pref]
[    3.926649] pci 0000:06:05.0: bridge window [io  0x1000-0x0fff] to [bus 08] add_size 1000
[    3.935788] pci 0000:06:05.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 08] add_size 200000
[    3.946962] pci 0000:06:05.0: bridge window [mem 0x00100000-0x000fffff] to [bus 08] add_size 200000
[    3.957077] pci 0000:0a:01.0: bridge window [io  0x1000-0x0fff] to [bus 0b] add_size 1000
[    3.966216] pci 0000:0a:01.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 0b] add_size 200000
[    3.977392] pci 0000:0a:01.0: bridge window [mem 0x00100000-0x000fffff] to [bus 0b] add_size 200000
[    3.987624] pci 0000:0a:04.0: bridge window [mem 0x00100000-0x003fffff pref] to [bus 0c-14] add_size 200000
[    3.998515] pci 0000:0a:01.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
[    4.009593] pci 0000:0a:04.0: res[15]=[mem 0x00100000-0x003fffff pref] get_res_add_size add_size 200000
[    4.020089] pci 0000:09:00.0: bridge window [mem 0x00100000-0x003fffff pref] to [bus 0a-14] add_size 400000
[    4.030979] pci 0000:09:00.0: res[15]=[mem 0x00100000-0x003fffff pref] get_res_add_size add_size 400000
[    4.041475] pci 0000:06:08.0: bridge window [mem 0x00100000-0x003fffff pref] to [bus 09-14] add_size 400000
[    4.052355] pci 0000:06:05.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
[    4.063433] pci 0000:06:08.0: res[15]=[mem 0x00100000-0x003fffff pref] get_res_add_size add_size 400000
[    4.073929] pci 0000:05:00.0: bridge window [mem 0x00100000-0x003fffff pref] to [bus 06-14] add_size 600000
[    4.084808] pci 0000:05:00.0: res[15]=[mem 0x00100000-0x003fffff pref] get_res_add_size add_size 600000
[    4.095305] pci 0000:00:05.0: bridge window [mem 0x00100000-0x003fffff pref] to [bus 05-14] add_size 600000
[    4.106217] pci 0000:00:1c.0: bridge window [mem 0x00100000-0x001fffff pref] to [bus 1a-1c] add_size 200000
[    4.117106] pci 0000:00:1c.4: bridge window [io  0x1000-0x0fff] to [bus 1d] add_size 1000
[    4.126248] pci 0000:00:1c.5: bridge window [io  0x1000-0x0fff] to [bus 1e] add_size 1000
[    4.135386] pci 0000:00:1c.5: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 1e] add_size 200000
[    4.146562] pci 0000:00:1c.5: bridge window [mem 0x00100000-0x000fffff] to [bus 1e] add_size 200000
[    4.156690] pci 0000:00:05.0: res[15]=[mem 0x00100000-0x003fffff pref] get_res_add_size add_size 600000
[    4.167187] pci 0000:00:1c.0: res[15]=[mem 0x00100000-0x001fffff pref] get_res_add_size add_size 200000
[    4.177683] pci 0000:00:1c.5: res[14]=[mem 0x00100000-0x000fffff] get_res_add_size add_size 200000
[    4.187695] pci 0000:00:1c.5: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
[    4.198774] pci 0000:00:1c.4: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
[    4.207814] pci 0000:00:1c.5: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
[    4.216848] pci 0000:00:05.0: BAR 15: assigned [mem 0xb2500000-0xb2dfffff pref]
[    4.225015] pci 0000:00:07.0: BAR 15: assigned [mem 0xb2e00000-0xb2efffff pref]
[    4.233183] pci 0000:00:1c.0: BAR 15: assigned [mem 0xb2f00000-0xb31fffff pref]
[    4.241350] pci 0000:00:1c.5: BAR 14: assigned [mem 0xb3200000-0xb33fffff]
[    4.249024] pci 0000:00:1c.5: BAR 15: assigned [mem 0xb3400000-0xb35fffff 64bit pref]
[    4.257774] pci 0000:00:1c.4: BAR 13: assigned [io  0x6000-0x6fff]
[    4.264679] pci 0000:00:1c.5: BAR 13: assigned [io  0x7000-0x7fff]
[    4.271576] pci 0000:00:01.0: PCI bridge to [bus 01]
[    4.277131] pci 0000:00:02.0: PCI bridge to [bus 02]
[    4.282682] pci 0000:00:03.0: PCI bridge to [bus 03]
[    4.288235] pci 0000:00:04.0: PCI bridge to [bus 04]
[    4.293787] pci 0000:05:00.0: res[15]=[mem 0x00100000-0x003fffff pref] get_res_add_size add_size 600000
[    4.304283] pci 0000:05:00.0: BAR 15: assigned [mem 0xb2500000-0xb2dfffff pref]
[    4.312442] pci 0000:06:05.0: res[14]=[mem 0x00100000-0x000fffff] get_res_add_size add_size 200000
[    4.322455] pci 0000:06:05.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
[    4.333534] pci 0000:06:08.0: res[15]=[mem 0x00100000-0x003fffff pref] get_res_add_size add_size 400000
[    4.344029] pci 0000:06:05.0: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
[    4.353069] pci 0000:06:05.0: BAR 14: can't assign mem (size 0x200000)
[    4.360363] pci 0000:06:05.0: BAR 15: assigned [mem 0xb2500000-0xb26fffff 64bit pref]
[    4.369113] pci 0000:06:08.0: BAR 15: assigned [mem 0xb2700000-0xb2dfffff pref]
[    4.377281] pci 0000:06:05.0: BAR 13: can't assign io (size 0x1000)
[    4.384284] pci 0000:06:08.0: BAR 15: assigned [mem 0xb2500000-0xb27fffff pref]
[    4.392450] pci 0000:06:08.0: BAR 15: reassigned [mem 0xb2500000-0xb2bfffff pref]
[    4.400810] pci 0000:06:05.0: BAR 14: can't assign mem (size 0x200000)
[    4.408104] pci 0000:06:05.0: BAR 15: assigned [mem 0xb2c00000-0xb2dfffff 64bit pref]
[    4.416845] pci 0000:06:05.0: BAR 13: can't assign io (size 0x1000)
[    4.423848] pci 0000:06:04.0: PCI bridge to [bus 07]
[    4.429400] pci 0000:06:05.0: PCI bridge to [bus 08]
[    4.434949] pci 0000:06:05.0:   bridge window [mem 0xb2c00000-0xb2dfffff 64bit pref]
[    4.443606] pci 0000:09:00.0: res[15]=[mem 0x00100000-0x003fffff pref] get_res_add_size add_size 400000
[    4.454101] pci 0000:09:00.0: BAR 15: assigned [mem 0xb2500000-0xb2bfffff pref]
[    4.462269] pci 0000:0a:01.0: res[14]=[mem 0x00100000-0x000fffff] get_res_add_size add_size 200000
[    4.472280] pci 0000:0a:01.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
[    4.483358] pci 0000:0a:04.0: res[15]=[mem 0x00100000-0x003fffff pref] get_res_add_size add_size 200000
[    4.493853] pci 0000:0a:01.0: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
[    4.502891] pci 0000:0a:01.0: BAR 14: can't assign mem (size 0x200000)
[    4.510186] pci 0000:0a:01.0: BAR 15: assigned [mem 0xb2500000-0xb26fffff 64bit pref]
[    4.518933] pci 0000:0a:04.0: BAR 15: assigned [mem 0xb2700000-0xb2bfffff pref]
[    4.527099] pci 0000:0a:01.0: BAR 13: can't assign io (size 0x1000)
[    4.534101] pci 0000:0a:04.0: BAR 15: assigned [mem 0xb2500000-0xb27fffff pref]
[    4.542269] pci 0000:0a:04.0: BAR 15: reassigned [mem 0xb2500000-0xb29fffff pref]
[    4.550631] pci 0000:0a:01.0: BAR 14: can't assign mem (size 0x200000)
[    4.557925] pci 0000:0a:01.0: BAR 15: assigned [mem 0xb2a00000-0xb2bfffff 64bit pref]
[    4.566675] pci 0000:0a:01.0: BAR 13: can't assign io (size 0x1000)
[    4.573677] pci 0000:0a:01.0: PCI bridge to [bus 0b]
[    4.579221] pci 0000:0a:01.0:   bridge window [mem 0xb2a00000-0xb2bfffff 64bit pref]
[    4.587879] pci 0000:0c:00.0: BAR 15: assigned [mem 0xb2500000-0xb27fffff pref]
[    4.596046] pci 0000:0d:00.0: BAR 15: assigned [mem 0xb2500000-0xb25fffff pref]
[    4.604213] pci 0000:0d:01.0: BAR 15: assigned [mem 0xb2600000-0xb27fffff pref]
[    4.612371] pci 0000:0e:00.0: BAR 6: assigned [mem 0xb2500000-0xb2507fff pref]
[    4.620432] pci 0000:0d:00.0: PCI bridge to [bus 0e]
[    4.626002] pci 0000:0d:00.0:   bridge window [mem 0xb2000000-0xb20fffff]
[    4.633594] pci 0000:0d:00.0:   bridge window [mem 0xb2500000-0xb25fffff pref]
[    4.641682] pci 0000:0f:00.0: BAR 15: assigned [mem 0xb2600000-0xb27fffff pref]
[    4.649850] pci 0000:10:00.0: BAR 15: assigned [mem 0xb2600000-0xb26fffff pref]
[    4.658016] pci 0000:10:01.0: BAR 15: assigned [mem 0xb2700000-0xb27fffff pref]
[    4.666184] pci 0000:11:00.0: BAR 6: assigned [mem 0xb2600000-0xb260ffff pref]
[    4.674253] pci 0000:10:00.0: PCI bridge to [bus 11]
[    4.679800] pci 0000:10:00.0:   bridge window [io  0x4000-0x4fff]
[    4.686618] pci 0000:10:00.0:   bridge window [mem 0xb1f00000-0xb1ffffff]
[    4.694212] pci 0000:10:00.0:   bridge window [mem 0xb2600000-0xb26fffff pref]
[    4.702299] pci 0000:12:00.0: BAR 6: assigned [mem 0xb2700000-0xb270ffff pref]
[    4.710368] pci 0000:10:01.0: PCI bridge to [bus 12]
[    4.715917] pci 0000:10:01.0:   bridge window [io  0x3000-0x3fff]
[    4.722743] pci 0000:10:01.0:   bridge window [mem 0xb1e00000-0xb1efffff]
[    4.730338] pci 0000:10:01.0:   bridge window [mem 0xb2700000-0xb27fffff pref]
[    4.738426] pci 0000:10:02.0: PCI bridge to [bus 13]
[    4.743987] pci 0000:10:02.0:   bridge window [mem 0xb1d00000-0xb1dfffff]
[    4.751596] pci 0000:10:03.0: PCI bridge to [bus 14]
[    4.757154] pci 0000:10:03.0:   bridge window [mem 0xb1c00000-0xb1cfffff]
[    4.764763] pci 0000:0f:00.0: PCI bridge to [bus 10-14]
[    4.770602] pci 0000:0f:00.0:   bridge window [io  0x3000-0x4fff]
[    4.777428] pci 0000:0f:00.0:   bridge window [mem 0xb1c00000-0xb1ffffff]
[    4.785012] pci 0000:0f:00.0:   bridge window [mem 0xb2600000-0xb27fffff pref]
[    4.793103] pci 0000:0d:01.0: PCI bridge to [bus 0f-14]
[    4.798943] pci 0000:0d:01.0:   bridge window [io  0x3000-0x4fff]
[    4.805764] pci 0000:0d:01.0:   bridge window [mem 0xb1c00000-0xb1ffffff]
[    4.813356] pci 0000:0d:01.0:   bridge window [mem 0xb2600000-0xb27fffff pref]
[    4.821447] pci 0000:0c:00.0: PCI bridge to [bus 0d-14]
[    4.827286] pci 0000:0c:00.0:   bridge window [io  0x3000-0x4fff]
[    4.834097] pci 0000:0c:00.0:   bridge window [mem 0xb1c00000-0xb21fffff]
[    4.841684] pci 0000:0c:00.0:   bridge window [mem 0xb2500000-0xb27fffff pref]
[    4.849759] pci 0000:0a:04.0: PCI bridge to [bus 0c-14]
[    4.855598] pci 0000:0a:04.0:   bridge window [io  0x3000-0x4fff]
[    4.862409] pci 0000:0a:04.0:   bridge window [mem 0xb1c00000-0xb21fffff]
[    4.869996] pci 0000:0a:04.0:   bridge window [mem 0xb2500000-0xb29fffff pref]
[    4.878070] pci 0000:09:00.0: PCI bridge to [bus 0a-14]
[    4.883909] pci 0000:09:00.0:   bridge window [io  0x3000-0x4fff]
[    4.890711] pci 0000:09:00.0:   bridge window [mem 0xb1c00000-0xb21fffff]
[    4.898297] pci 0000:09:00.0:   bridge window [mem 0xb2500000-0xb2bfffff pref]
[    4.906370] pci 0000:06:08.0: PCI bridge to [bus 09-14]
[    4.912208] pci 0000:06:08.0:   bridge window [io  0x3000-0x4fff]
[    4.919010] pci 0000:06:08.0:   bridge window [mem 0xb1c00000-0xb22fffff]
[    4.926596] pci 0000:06:08.0:   bridge window [mem 0xb2500000-0xb2bfffff pref]
[    4.934668] pci 0000:05:00.0: PCI bridge to [bus 06-14]
[    4.940506] pci 0000:05:00.0:   bridge window [io  0x3000-0x4fff]
[    4.947316] pci 0000:05:00.0:   bridge window [mem 0xb1c00000-0xb22fffff]
[    4.954900] pci 0000:05:00.0:   bridge window [mem 0xb2500000-0xb2dfffff pref]
[    4.962965] pci 0000:00:05.0: PCI bridge to [bus 05-14]
[    4.968803] pci 0000:00:05.0:   bridge window [io  0x3000-0x4fff]
[    4.975604] pci 0000:00:05.0:   bridge window [mem 0xb1c00000-0xb23fffff]
[    4.983189] pci 0000:00:05.0:   bridge window [mem 0xb2500000-0xb2dfffff pref]
[    4.991252] pci 0000:00:06.0: PCI bridge to [bus 15]
[    4.996804] pci 0000:16:00.0: BAR 6: assigned [mem 0xb2e00000-0xb2e3ffff pref]
[    5.004874] pci 0000:16:00.1: BAR 6: assigned [mem 0xb2e40000-0xb2e7ffff pref]
[    5.012936] pci 0000:00:07.0: PCI bridge to [bus 16]
[    5.018483] pci 0000:00:07.0:   bridge window [io  0x2000-0x2fff]
[    5.025293] pci 0000:00:07.0:   bridge window [mem 0xb1b00000-0xb1bfffff]
[    5.032878] pci 0000:00:07.0:   bridge window [mem 0xb2e00000-0xb2efffff pref]
[    5.040950] pci 0000:00:08.0: PCI bridge to [bus 17]
[    5.046501] pci 0000:00:09.0: PCI bridge to [bus 18]
[    5.052055] pci 0000:00:0a.0: PCI bridge to [bus 19]
[    5.057608] pci 0000:1a:00.0: BAR 6: assigned [mem 0xb2f00000-0xb2f1ffff pref]
[    5.065679] pci 0000:1a:00.1: BAR 6: assigned [mem 0xb2f20000-0xb2f3ffff pref]
[    5.073749] pci 0000:00:1c.0: PCI bridge to [bus 1a-1c]
[    5.079587] pci 0000:00:1c.0:   bridge window [io  0x1000-0x1fff]
[    5.086389] pci 0000:00:1c.0:   bridge window [mem 0xb1900000-0xb1afffff]
[    5.093974] pci 0000:00:1c.0:   bridge window [mem 0xb2f00000-0xb31fffff pref]
[    5.102039] pci 0000:1d:00.0: BAR 6: assigned [mem 0xb1810000-0xb181ffff pref]
[    5.110109] pci 0000:00:1c.4: PCI bridge to [bus 1d]
[    5.115657] pci 0000:00:1c.4:   bridge window [io  0x6000-0x6fff]
[    5.122469] pci 0000:00:1c.4:   bridge window [mem 0xb1000000-0xb18fffff]
[    5.130054] pci 0000:00:1c.4:   bridge window [mem 0xb0000000-0xb0ffffff 64bit pref]
[    5.138710] pci 0000:00:1c.5: PCI bridge to [bus 1e]
[    5.144257] pci 0000:00:1c.5:   bridge window [io  0x7000-0x7fff]
[    5.151067] pci 0000:00:1c.5:   bridge window [mem 0xb3200000-0xb33fffff]
[    5.158652] pci 0000:00:1c.5:   bridge window [mem 0xb3400000-0xb35fffff 64bit pref]
[    5.167307] pci 0000:00:1e.0: PCI bridge to [bus 1f]
[    5.179614] pci 0000:00:1e.0: setting latency timer to 64
[    5.185651] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7]
[    5.191878] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff]
[    5.198105] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff]
[    5.205106] pci_bus 0000:00: resource 7 [mem 0xfed40000-0xfedfffff]
[    5.212109] pci_bus 0000:00: resource 8 [mem 0xb0000000-0xfdffffff]
[    5.219114] pci_bus 0000:05: resource 0 [io  0x3000-0x4fff]
[    5.225338] pci_bus 0000:05: resource 1 [mem 0xb1c00000-0xb23fffff]
[    5.232339] pci_bus 0000:05: resource 2 [mem 0xb2500000-0xb2dfffff pref]
[    5.239825] pci_bus 0000:06: resource 0 [io  0x3000-0x4fff]
[    5.246050] pci_bus 0000:06: resource 1 [mem 0xb1c00000-0xb22fffff]
[    5.253052] pci_bus 0000:06: resource 2 [mem 0xb2500000-0xb2dfffff pref]
[    5.260539] pci_bus 0000:08: resource 2 [mem 0xb2c00000-0xb2dfffff 64bit pref]
[    5.268600] pci_bus 0000:09: resource 0 [io  0x3000-0x4fff]
[    5.274827] pci_bus 0000:09: resource 1 [mem 0xb1c00000-0xb22fffff]
[    5.281828] pci_bus 0000:09: resource 2 [mem 0xb2500000-0xb2bfffff pref]
[    5.289315] pci_bus 0000:0a: resource 0 [io  0x3000-0x4fff]
[    5.295541] pci_bus 0000:0a: resource 1 [mem 0xb1c00000-0xb21fffff]
[    5.302544] pci_bus 0000:0a: resource 2 [mem 0xb2500000-0xb2bfffff pref]
[    5.310031] pci_bus 0000:0b: resource 2 [mem 0xb2a00000-0xb2bfffff 64bit pref]
[    5.318102] pci_bus 0000:0c: resource 0 [io  0x3000-0x4fff]
[    5.324326] pci_bus 0000:0c: resource 1 [mem 0xb1c00000-0xb21fffff]
[    5.331329] pci_bus 0000:0c: resource 2 [mem 0xb2500000-0xb29fffff pref]
[    5.338815] pci_bus 0000:0d: resource 0 [io  0x3000-0x4fff]
[    5.345042] pci_bus 0000:0d: resource 1 [mem 0xb1c00000-0xb21fffff]
[    5.352045] pci_bus 0000:0d: resource 2 [mem 0xb2500000-0xb27fffff pref]
[    5.359532] pci_bus 0000:0e: resource 1 [mem 0xb2000000-0xb20fffff]
[    5.366535] pci_bus 0000:0e: resource 2 [mem 0xb2500000-0xb25fffff pref]
[    5.374021] pci_bus 0000:0f: resource 0 [io  0x3000-0x4fff]
[    5.380246] pci_bus 0000:0f: resource 1 [mem 0xb1c00000-0xb1ffffff]
[    5.387249] pci_bus 0000:0f: resource 2 [mem 0xb2600000-0xb27fffff pref]
[    5.394736] pci_bus 0000:10: resource 0 [io  0x3000-0x4fff]
[    5.400962] pci_bus 0000:10: resource 1 [mem 0xb1c00000-0xb1ffffff]
[    5.407956] pci_bus 0000:10: resource 2 [mem 0xb2600000-0xb27fffff pref]
[    5.415435] pci_bus 0000:11: resource 0 [io  0x4000-0x4fff]
[    5.421651] pci_bus 0000:11: resource 1 [mem 0xb1f00000-0xb1ffffff]
[    5.428653] pci_bus 0000:11: resource 2 [mem 0xb2600000-0xb26fffff pref]
[    5.436141] pci_bus 0000:12: resource 0 [io  0x3000-0x3fff]
[    5.442367] pci_bus 0000:12: resource 1 [mem 0xb1e00000-0xb1efffff]
[    5.449370] pci_bus 0000:12: resource 2 [mem 0xb2700000-0xb27fffff pref]
[    5.456857] pci_bus 0000:13: resource 1 [mem 0xb1d00000-0xb1dfffff]
[    5.463859] pci_bus 0000:14: resource 1 [mem 0xb1c00000-0xb1cfffff]
[    5.470853] pci_bus 0000:16: resource 0 [io  0x2000-0x2fff]
[    5.477078] pci_bus 0000:16: resource 1 [mem 0xb1b00000-0xb1bfffff]
[    5.484079] pci_bus 0000:16: resource 2 [mem 0xb2e00000-0xb2efffff pref]
[    5.491566] pci_bus 0000:1a: resource 0 [io  0x1000-0x1fff]
[    5.497783] pci_bus 0000:1a: resource 1 [mem 0xb1900000-0xb1afffff]
[    5.504776] pci_bus 0000:1a: resource 2 [mem 0xb2f00000-0xb31fffff pref]
[    5.512263] pci_bus 0000:1d: resource 0 [io  0x6000-0x6fff]
[    5.518480] pci_bus 0000:1d: resource 1 [mem 0xb1000000-0xb18fffff]
[    5.525481] pci_bus 0000:1d: resource 2 [mem 0xb0000000-0xb0ffffff 64bit pref]
[    5.533550] pci_bus 0000:1e: resource 0 [io  0x7000-0x7fff]
[    5.539777] pci_bus 0000:1e: resource 1 [mem 0xb3200000-0xb33fffff]
[    5.546778] pci_bus 0000:1e: resource 2 [mem 0xb3400000-0xb35fffff 64bit pref]
[    5.554849] pci_bus 0000:1f: resource 4 [io  0x0000-0x0cf7]
[    5.561075] pci_bus 0000:1f: resource 5 [io  0x0d00-0xffff]
[    5.567300] pci_bus 0000:1f: resource 6 [mem 0x000a0000-0x000bffff]
[    5.574301] pci_bus 0000:1f: resource 7 [mem 0xfed40000-0xfedfffff]
[    5.581302] pci_bus 0000:1f: resource 8 [mem 0xb0000000-0xfdffffff]
[    5.588306] pci_bus 0000:fe: resource 4 [io  0x0000-0xffff]
[    5.594533] pci_bus 0000:fe: resource 5 [mem 0x00000000-0xffffffffff]
[    5.601730] pci_bus 0000:ff: resource 4 [io  0x0000-0xffff]
[    5.607956] pci_bus 0000:ff: resource 5 [mem 0x00000000-0xffffffffff]
[    5.615218] NET: Registered protocol family 2
[    5.620472] TCP established hash table entries: 131072 (order: 9, 2097152 bytes)
[    5.629206] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    5.636879] TCP: Hash tables configured (established 131072 bind 65536)
[    5.644305] TCP: reno registered
[    5.647943] UDP hash table entries: 8192 (order: 6, 262144 bytes)
[    5.654826] UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes)
[    5.662298] NET: Registered protocol family 1
[    5.669299] pci 0000:1d:00.0: Boot video device
[    5.674414] PCI: CLS 64 bytes, default 64
[    5.678930] Unpacking initramfs...
[    5.950385] Freeing initrd memory: 14412k freed
[    5.959081] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    5.966287] software IO TLB [mem 0x87d5d000-0x8bd5d000] (64MB) mapped at [ffff880087d5d000-ffff88008bd5cfff]
[    5.981757] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni)
[    5.989377] audit: initializing netlink socket (disabled)
[    5.995432] type=2000 audit(1373984231.242:1): initialized
[    6.037818] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    6.046864] VFS: Disk quotas dquot_6.5.2
[    6.051307] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    6.058995] msgmni has been set to 31955
[    6.063460] SELinux:  Registering netfilter hooks
[    6.069889] alg: No test for stdrng (krng)
[    6.074477] NET: Registered protocol family 38
[    6.079482] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[    6.087816] io scheduler noop registered
[    6.092200] io scheduler deadline registered
[    6.097005] io scheduler cfq registered (default)
[    6.102528] pcieport 0000:00:01.0: irq 65 for MSI/MSI-X
[    6.108444] pcieport 0000:00:02.0: irq 66 for MSI/MSI-X
[    6.114346] pcieport 0000:00:03.0: irq 67 for MSI/MSI-X
[    6.120260] pcieport 0000:00:04.0: irq 68 for MSI/MSI-X
[    6.126168] pcieport 0000:00:05.0: irq 69 for MSI/MSI-X
[    6.132078] pcieport 0000:00:06.0: irq 70 for MSI/MSI-X
[    6.137999] pcieport 0000:00:07.0: irq 71 for MSI/MSI-X
[    6.143908] pcieport 0000:00:08.0: irq 72 for MSI/MSI-X
[    6.149816] pcieport 0000:00:09.0: irq 73 for MSI/MSI-X
[    6.155725] pcieport 0000:00:0a.0: irq 74 for MSI/MSI-X
[    6.161653] pcieport 0000:00:1c.0: irq 75 for MSI/MSI-X
[    6.167584] pcieport 0000:00:1c.4: irq 76 for MSI/MSI-X
[    6.173509] pcieport 0000:00:1c.5: irq 77 for MSI/MSI-X
[    6.179428] pcieport 0000:05:00.0: irq 78 for MSI/MSI-X
[    6.185345] pcieport 0000:06:04.0: irq 79 for MSI/MSI-X
[    6.191264] pcieport 0000:06:05.0: irq 80 for MSI/MSI-X
[    6.197196] pcieport 0000:06:08.0: irq 81 for MSI/MSI-X
[    6.203127] pcieport 0000:09:00.0: irq 82 for MSI/MSI-X
[    6.209069] pcieport 0000:0a:01.0: irq 83 for MSI/MSI-X
[    6.215014] pcieport 0000:0a:04.0: irq 84 for MSI/MSI-X
[    6.220955] pcieport 0000:0c:00.0: irq 85 for MSI/MSI-X
[    6.227250] aer 0000:00:01.0:pcie02: service driver aer loaded
[    6.233787] aer 0000:00:02.0:pcie02: service driver aer loaded
[    6.240321] aer 0000:00:03.0:pcie02: service driver aer loaded
[    6.246855] aer 0000:00:04.0:pcie02: service driver aer loaded
[    6.253419] aer 0000:00:05.0:pcie02: service driver aer loaded
[    6.259955] aer 0000:00:06.0:pcie02: service driver aer loaded
[    6.266491] aer 0000:00:07.0:pcie02: service driver aer loaded
[    6.273026] aer 0000:00:08.0:pcie02: service driver aer loaded
[    6.279557] aer 0000:00:09.0:pcie02: service driver aer loaded
[    6.286091] aer 0000:00:0a.0:pcie02: service driver aer loaded
[    6.292622] pcieport 0000:00:01.0: Signaling PME through PCIe PME interrupt
[    6.300403] pcie_pme 0000:00:01.0:pcie01: service driver pcie_pme loaded
[    6.307900] pcieport 0000:00:02.0: Signaling PME through PCIe PME interrupt
[    6.315680] pcie_pme 0000:00:02.0:pcie01: service driver pcie_pme loaded
[    6.323173] pcieport 0000:00:03.0: Signaling PME through PCIe PME interrupt
[    6.330951] pcie_pme 0000:00:03.0:pcie01: service driver pcie_pme loaded
[    6.338442] pcieport 0000:00:04.0: Signaling PME through PCIe PME interrupt
[    6.346223] pcie_pme 0000:00:04.0:pcie01: service driver pcie_pme loaded
[    6.353720] pcieport 0000:00:05.0: Signaling PME through PCIe PME interrupt
[    6.361499] pcieport 0000:05:00.0: Signaling PME through PCIe PME interrupt
[    6.369269] pcieport 0000:06:04.0: Signaling PME through PCIe PME interrupt
[    6.377038] pcieport 0000:06:05.0: Signaling PME through PCIe PME interrupt
[    6.384816] pcieport 0000:06:08.0: Signaling PME through PCIe PME interrupt
[    6.392585] pcieport 0000:09:00.0: Signaling PME through PCIe PME interrupt
[    6.400362] pcieport 0000:0a:01.0: Signaling PME through PCIe PME interrupt
[    6.408139] pcieport 0000:0a:04.0: Signaling PME through PCIe PME interrupt
[    6.415918] pcieport 0000:0c:00.0: Signaling PME through PCIe PME interrupt
[    6.423696] pcieport 0000:0d:00.0: Signaling PME through PCIe PME interrupt
[    6.431466] pci 0000:0e:00.0: Signaling PME through PCIe PME interrupt
[    6.438758] pcieport 0000:0d:01.0: Signaling PME through PCIe PME interrupt
[    6.446534] pcieport 0000:0f:00.0: Signaling PME through PCIe PME interrupt
[    6.454311] pcieport 0000:10:00.0: Signaling PME through PCIe PME interrupt
[    6.462087] pci 0000:11:00.0: Signaling PME through PCIe PME interrupt
[    6.469379] pcieport 0000:10:01.0: Signaling PME through PCIe PME interrupt
[    6.477157] pci 0000:12:00.0: Signaling PME through PCIe PME interrupt
[    6.484448] pcieport 0000:10:02.0: Signaling PME through PCIe PME interrupt
[    6.492226] pci 0000:13:00.0: Signaling PME through PCIe PME interrupt
[    6.499518] pcieport 0000:10:03.0: Signaling PME through PCIe PME interrupt
[    6.507294] pci 0000:14:00.0: Signaling PME through PCIe PME interrupt
[    6.514588] pcie_pme 0000:00:05.0:pcie01: service driver pcie_pme loaded
[    6.522071] pcieport 0000:00:06.0: Signaling PME through PCIe PME interrupt
[    6.529850] pcie_pme 0000:00:06.0:pcie01: service driver pcie_pme loaded
[    6.537332] pcieport 0000:00:07.0: Signaling PME through PCIe PME interrupt
[    6.545110] pci 0000:16:00.0: Signaling PME through PCIe PME interrupt
[    6.552404] pci 0000:16:00.1: Signaling PME through PCIe PME interrupt
[    6.559690] pcie_pme 0000:00:07.0:pcie01: service driver pcie_pme loaded
[    6.567181] pcieport 0000:00:08.0: Signaling PME through PCIe PME interrupt
[    6.574963] pcie_pme 0000:00:08.0:pcie01: service driver pcie_pme loaded
[    6.582455] pcieport 0000:00:09.0: Signaling PME through PCIe PME interrupt
[    6.590236] pcie_pme 0000:00:09.0:pcie01: service driver pcie_pme loaded
[    6.597728] pcieport 0000:00:0a.0: Signaling PME through PCIe PME interrupt
[    6.605507] pcie_pme 0000:00:0a.0:pcie01: service driver pcie_pme loaded
[    6.613016] pcieport 0000:00:1c.0: Signaling PME through PCIe PME interrupt
[    6.620795] pci 0000:1a:00.0: Signaling PME through PCIe PME interrupt
[    6.628080] pci 0000:1a:00.1: Signaling PME through PCIe PME interrupt
[    6.635366] pcie_pme 0000:00:1c.0:pcie01: service driver pcie_pme loaded
[    6.642865] pcieport 0000:00:1c.4: Signaling PME through PCIe PME interrupt
[    6.650645] pci 0000:1d:00.0: Signaling PME through PCIe PME interrupt
[    6.657940] pcie_pme 0000:00:1c.4:pcie01: service driver pcie_pme loaded
[    6.665440] pcieport 0000:00:1c.5: Signaling PME through PCIe PME interrupt
[    6.673219] pcie_pme 0000:00:1c.5:pcie01: service driver pcie_pme loaded
[    6.680731] ioapic: probe of 0000:00:13.0 failed with error -22
[    6.687356] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    6.693638] pciehp 0000:00:1c.0:pcie04: HPC vendor_id 8086 device_id 3a40 ss_vid 1137 ss_did 101
[    6.703504] pciehp 0000:00:1c.0:pcie04: service driver pciehp loaded
[    6.710615] pciehp 0000:00:1c.4:pcie04: HPC vendor_id 8086 device_id 3a48 ss_vid 1137 ss_did 101
[    6.720481] pciehp 0000:00:1c.4:pcie04: service driver pciehp loaded
[    6.727591] pciehp 0000:00:1c.5:pcie04: HPC vendor_id 8086 device_id 3a4a ss_vid 1137 ss_did 101
[    6.737449] pciehp 0000:00:1c.5:pcie04: service driver pciehp loaded
[    6.744561] pciehp 0000:06:05.0:pcie24: HPC vendor_id 10b5 device_id 8624 ss_vid 10b5 ss_did 8624
[    6.754545] pciehp 0000:06:05.0:pcie24: service driver pciehp loaded
[    6.761668] pciehp 0000:06:08.0:pcie24: HPC vendor_id 10b5 device_id 8624 ss_vid 10b5 ss_did 8624
[    6.771649] pciehp 0000:06:08.0:pcie24: service driver pciehp loaded
[    6.778764] pciehp 0000:0a:01.0:pcie24: HPC vendor_id 10b5 device_id 8632 ss_vid 10b5 ss_did 8632
[    6.788736] pciehp 0000:0a:01.0:pcie24: service driver pciehp loaded
[    6.795850] pciehp 0000:0a:04.0:pcie24: HPC vendor_id 10b5 device_id 8632 ss_vid 10b5 ss_did 8632
[    6.805823] pciehp 0000:0a:04.0:pcie24: service driver pciehp loaded
[    6.812936] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[    6.820407] intel_idle: MWAIT substates: 0x1120
[    6.825470] intel_idle: v0.4 model 0x2C
[    6.829746] intel_idle: lapic_timer_reliable_states 0xffffffff
[    6.836414] ACPI Error: No handler for Region [POWS] (ffff88046484cb88) [IPMI] (20130328/evregion-161)
[    6.846843] ACPI Error: Region IPMI (ID=7) has no handler (20130328/exfldio-305)
[    6.855133] ACPI Error: Method parse/execution failed [\_SB_.PCI0.LPC0.P111._PSR] (Node ffff88046483d5f0), AE_NOT_EXIST (20130328/psparse-537)
[    6.869466] ACPI Exception: AE_NOT_EXIST, Error reading AC Adapter state (20130328/ac-126)
[    6.878755] ACPI Error: No handler for Region [POWS] (ffff88046484cbd0) [IPMI] (20130328/evregion-161)
[    6.889183] ACPI Error: Region IPMI (ID=7) has no handler (20130328/exfldio-305)
[    6.897477] ACPI Error: Method parse/execution failed [\_SB_.PCI0.LPC0.P112._PSR] (Node ffff88046483d7a8), AE_NOT_EXIST (20130328/psparse-537)
[    6.911816] ACPI Exception: AE_NOT_EXIST, Error reading AC Adapter state (20130328/ac-126)
[    6.921128] input: Sleep Button as /devices/LNXSYSTM:00/device:00/PNP0C0E:00/input/input0
[    6.930275] ACPI: Sleep Button [SLPB]
[    6.934403] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1
[    6.942667] ACPI: Power Button [PWRF]
[    6.946795] ACPI: Requesting acpi_cpufreq
[    6.955727] ERST: Error Record Serialization Table (ERST) support is initialized.
[    6.964304] GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
[    6.972636] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    6.980307] tsc: Refined TSC clocksource calibration: 2400.084 MHz
[    7.000267] 00:07: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    7.013378] Switching to clocksource tsc
[    7.028126] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[    7.035260] Non-volatile memory driver v1.3
[    7.039934] Linux agpgart interface v0.103
[    7.045560] loop: module loaded
[    7.049227] ata_piix 0000:00:1f.2: version 2.13
[    7.054453] ata_piix 0000:00:1f.2: MAP [
[    7.058834]  P0 P2 P1 P3 ]
[    7.061937] ata_piix 0000:00:1f.2: setting latency timer to 64
[    7.068672] scsi0 : ata_piix
[    7.072204] scsi1 : ata_piix
[    7.075654] ata1: SATA max UDMA/133 cmd 0x5138 ctl 0x514c bmdma 0x5110 irq 18
[    7.083629] ata2: SATA max UDMA/133 cmd 0x5130 ctl 0x5148 bmdma 0x5118 irq 18
[    7.091733] ata_piix 0000:00:1f.5: MAP [
[    7.096116]  P0 -- P1 -- ]
[    7.249526] ata_piix 0000:00:1f.5: SCR access via SIDPR is available but doesn't work
[    7.258281] ata_piix 0000:00:1f.5: setting latency timer to 64
[    7.264981] scsi2 : ata_piix
[    7.268387] scsi3 : ata_piix
[    7.271797] ata3: SATA max UDMA/133 cmd 0x5128 ctl 0x5144 bmdma 0x50f0 irq 21
[    7.279770] ata4: SATA max UDMA/133 cmd 0x5120 ctl 0x5140 bmdma 0x50f8 irq 21
[    7.287792] libphy: Fixed MDIO Bus: probed
[    7.292447] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    7.299741] ehci-pci: EHCI PCI platform driver
[    7.304844] ehci-pci 0000:00:1a.7: setting latency timer to 64
[    7.311376] ehci-pci 0000:00:1a.7: EHCI Host Controller
[    7.317250] ehci-pci 0000:00:1a.7: new USB bus registered, assigned bus number 1
[    7.325524] ehci-pci 0000:00:1a.7: debug port 1
[    7.334483] ehci-pci 0000:00:1a.7: cache line size of 64 is not supported
[    7.342076] ehci-pci 0000:00:1a.7: irq 19, io mem 0xb2421000
[    7.353598] ehci-pci 0000:00:1a.7: USB 2.0 started, EHCI 1.00
[    7.360039] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    7.367621] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    7.375688] usb usb1: Product: EHCI Host Controller
[    7.381136] usb usb1: Manufacturer: Linux 3.10.0-rc5.nab+ ehci_hcd
[    7.388038] usb usb1: SerialNumber: 0000:00:1a.7
[    7.393283] hub 1-0:1.0: USB hub found
[    7.397467] hub 1-0:1.0: 6 ports detected
[    7.402194] ehci-pci 0000:00:1d.7: setting latency timer to 64
[    7.408725] ehci-pci 0000:00:1d.7: EHCI Host Controller
[    7.414600] ehci-pci 0000:00:1d.7: new USB bus registered, assigned bus number 2
[    7.422875] ehci-pci 0000:00:1d.7: debug port 1
[    7.431821] ehci-pci 0000:00:1d.7: cache line size of 64 is not supported
[    7.439416] ehci-pci 0000:00:1d.7: irq 16, io mem 0xb2420000
[    7.451678] ehci-pci 0000:00:1d.7: USB 2.0 started, EHCI 1.00
[    7.458116] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
[    7.465702] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    7.467885] ata4.00: ATAPI: Optiarc DVD RW AD-7580S, FX04, max UDMA/100
[    7.481153] usb usb2: Product: EHCI Host Controller
[    7.486599] usb usb2: Manufacturer: Linux 3.10.0-rc5.nab+ ehci_hcd
[    7.489900] ata4.00: configured for UDMA/100
[    7.498270] usb usb2: SerialNumber: 0000:00:1d.7
[    7.503525] hub 2-0:1.0: USB hub found
[    7.507719] hub 2-0:1.0: 6 ports detected
[    7.512308] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    7.519224] uhci_hcd: USB Universal Host Controller Interface driver
[    7.526451] uhci_hcd 0000:00:1a.0: setting latency timer to 64
[    7.532971] uhci_hcd 0000:00:1a.0: UHCI Host Controller
[    7.538845] uhci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 3
[    7.547131] uhci_hcd 0000:00:1a.0: irq 19, io base 0x000050c0
[    7.553580] usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
[    7.561164] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    7.569230] usb usb3: Product: UHCI Host Controller
[    7.574675] usb usb3: Manufacturer: Linux 3.10.0-rc5.nab+ uhci_hcd
[    7.581579] usb usb3: SerialNumber: 0000:00:1a.0
[    7.586811] hub 3-0:1.0: USB hub found
[    7.591001] hub 3-0:1.0: 2 ports detected
[    7.595669] uhci_hcd 0000:00:1a.1: setting latency timer to 64
[    7.602189] uhci_hcd 0000:00:1a.1: UHCI Host Controller
[    7.608073] uhci_hcd 0000:00:1a.1: new USB bus registered, assigned bus number 4
[    7.616382] uhci_hcd 0000:00:1a.1: irq 19, io base 0x000050a0
[    7.622827] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
[    7.630409] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    7.638475] usb usb4: Product: UHCI Host Controller
[    7.643925] usb usb4: Manufacturer: Linux 3.10.0-rc5.nab+ uhci_hcd
[    7.650819] usb usb4: SerialNumber: 0000:00:1a.1
[    7.656051] hub 4-0:1.0: USB hub found
[    7.660241] hub 4-0:1.0: 2 ports detected
[    7.664907] uhci_hcd 0000:00:1a.2: setting latency timer to 64
[    7.671424] uhci_hcd 0000:00:1a.2: UHCI Host Controller
[    7.677293] uhci_hcd 0000:00:1a.2: new USB bus registered, assigned bus number 5
[    7.685578] uhci_hcd 0000:00:1a.2: irq 19, io base 0x00005080
[    7.692021] usb usb5: New USB device found, idVendor=1d6b, idProduct=0001
[    7.699606] usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    7.707687] usb usb5: Product: UHCI Host Controller
[    7.713134] usb usb5: Manufacturer: Linux 3.10.0-rc5.nab+ uhci_hcd
[    7.720038] usb usb5: SerialNumber: 0000:00:1a.2
[    7.725308] hub 5-0:1.0: USB hub found
[    7.729516] hub 5-0:1.0: 2 ports detected
[    7.734226] uhci_hcd 0000:00:1d.0: setting latency timer to 64
[    7.740747] uhci_hcd 0000:00:1d.0: UHCI Host Controller
[    7.746623] uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 6
[    7.754906] uhci_hcd 0000:00:1d.0: irq 16, io base 0x00005060
[    7.761355] usb usb6: New USB device found, idVendor=1d6b, idProduct=0001
[    7.768937] usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    7.777005] usb usb6: Product: UHCI Host Controller
[    7.782453] usb usb6: Manufacturer: Linux 3.10.0-rc5.nab+ uhci_hcd
[    7.789354] usb usb6: SerialNumber: 0000:00:1d.0
[    7.794585] hub 6-0:1.0: USB hub found
[    7.798775] hub 6-0:1.0: 2 ports detected
[    7.803440] uhci_hcd 0000:00:1d.1: setting latency timer to 64
[    7.809958] uhci_hcd 0000:00:1d.1: UHCI Host Controller
[    7.815828] uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 7
[    7.824112] uhci_hcd 0000:00:1d.1: irq 16, io base 0x00005040
[    7.830555] usb usb7: New USB device found, idVendor=1d6b, idProduct=0001
[    7.838140] usb usb7: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    7.846207] usb usb7: Product: UHCI Host Controller
[    7.851654] usb usb7: Manufacturer: Linux 3.10.0-rc5.nab+ uhci_hcd
[    7.853087] ata2.00: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    7.853104] ata2.01: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    7.859545] ata2.00: ATA-8: ST9500530NS, CC04, max UDMA/133
[    7.859547] ata2.00: 976773168 sectors, multi 16: LBA48 NCQ (depth 0/32)
[    7.859888] ata2.01: ATA-8: ST9500530NS, CC04, max UDMA/133
[    7.859890] ata2.01: 976773168 sectors, multi 16: LBA48 NCQ (depth 0/32)
[    7.866571] ata2.00: configured for UDMA/133
[    7.872535] ata2.01: configured for UDMA/133
[    7.909880] usb usb7: SerialNumber: 0000:00:1d.1
[    7.909965] ata1.00: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    7.909981] ata1.01: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    7.915578] ata1.00: ATA-8: ST9500530NS, CC03, max UDMA/133
[    7.915579] ata1.00: 976773168 sectors, multi 16: LBA48 NCQ (depth 0/32)
[    7.915919] ata1.01: ATA-8: ST9500530NS, CC04, max UDMA/133
[    7.915920] ata1.01: 976773168 sectors, multi 16: LBA48 NCQ (depth 0/32)
[    7.921573] ata1.00: configured for UDMA/133
[    7.927581] ata1.01: configured for UDMA/133
[    7.927596] Calling blk_mq_init_queue: scsi_mq_ops: ffffffff81ca13e0, queue_depth: 64, cmd_size: 296 SCSI cmd_size: 0
[    7.927639] blk-mq: CPU -> queue map
[    7.927640]   CPU 0 -> Queue 0
[    7.927640]   CPU 1 -> Queue 0
[    7.927640]   CPU 2 -> Queue 0
[    7.927641]   CPU 3 -> Queue 0
[    7.927641]   CPU 4 -> Queue 0
[    7.927642]   CPU 5 -> Queue 0
[    7.927642]   CPU 6 -> Queue 0
[    7.927643]   CPU 7 -> Queue 0
[    7.927643]   CPU 8 -> Queue 0
[    7.927643]   CPU 9 -> Queue 0
[    7.927644]   CPU10 -> Queue 0
[    7.927644]   CPU11 -> Queue 0
[    7.927645]   CPU12 -> Queue 0
[    7.927645]   CPU13 -> Queue 0
[    7.927646]   CPU14 -> Queue 0
[    7.927646]   CPU15 -> Queue 0
[    7.927673] Performing sc map setup on q: ffff880462430000 hctx: ffff880462a14200 i: 0
[    7.927780] scsi_mq_alloc_queue() complete !! >>>>>>>>>>>>>>>>>>>>>>>>>>>
[    7.927784] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.927785] Allocated blk-mq req: ffff88046244ad40, req->tag: 63
[    7.927790] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.927803] scsi_execute(): Calling blk_mq_free_request >>>
[    7.927815] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.927816] Allocated blk-mq req: ffff88046244ad40, req->tag: 63
[    7.927817] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.927818] scsi_execute(): Calling blk_mq_free_request >>>
[    7.927826] scsi 0:0:0:0: Direct-Access     ATA      ST9500530NS      CC03 PQ: 0 ANSI: 5
[    7.927944] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.927946] Allocated blk-mq req: ffff88046244a7c0, req->tag: 61
[    7.927946] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.927949] scsi_execute(): Calling blk_mq_free_request >>>
[    7.927951] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.927952] Allocated blk-mq req: ffff88046244a7c0, req->tag: 61
[    7.927955] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.927958] scsi_execute(): Calling blk_mq_free_request >>>
[    7.927960] sd 0:0:0:0: [sda] Sector size 0 reported, assuming 512.
[    7.927964] sd 0:0:0:0: [sda] 1 512-byte logical blocks: (512 B/512 B)
[    7.927965] sd 0:0:0:0: [sda] 0-byte physical blocks
[    7.927966] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.927967] Allocated blk-mq req: ffff88046244a7c0, req->tag: 61
[    7.927968] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.927970] scsi_execute(): Calling blk_mq_free_request >>>
[    7.927970] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.927971] Allocated blk-mq req: ffff88046244a7c0, req->tag: 61
[    7.927972] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.927973] scsi_execute(): Calling blk_mq_free_request >>>
[    7.927975] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.927976] Allocated blk-mq req: ffff88046244a7c0, req->tag: 61
[    7.927977] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.927979] scsi_execute(): Calling blk_mq_free_request >>>
[    7.927981] sd 0:0:0:0: [sda] Write Protect is off
[    7.927982] sd 0:0:0:0: [sda] Mode Sense: 00 00 00 00
[    7.927983] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.927984] Allocated blk-mq req: ffff88046244a7c0, req->tag: 61
[    7.927985] sd 0:0:0:0: Attached scsi generic sg0 type 0
[    7.927986] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.927987] scsi_execute(): Calling blk_mq_free_request >>>
[    7.927989] sd 0:0:0:0: [sda] Asking for cache data failed
[    7.927990] sd 0:0:0:0: [sda] Assuming drive cache: write through
[    7.927995] Calling blk_mq_init_queue: scsi_mq_ops: ffffffff81ca13e0, queue_depth: 64, cmd_size: 296 SCSI cmd_size: 0
[    7.928030] blk-mq: CPU -> queue map
[    7.928030]   CPU 0 -> Queue 0
[    7.928031]   CPU 1 -> Queue 0
[    7.928031]   CPU 2 -> Queue 0
[    7.928032]   CPU 3 -> Queue 0
[    7.928032]   CPU 4 -> Queue 0
[    7.928032]   CPU 5 -> Queue 0
[    7.928033]   CPU 6 -> Queue 0
[    7.928033]   CPU 7 -> Queue 0
[    7.928034]   CPU 8 -> Queue 0
[    7.928034]   CPU 9 -> Queue 0
[    7.928035]   CPU10 -> Queue 0
[    7.928035]   CPU11 -> Queue 0
[    7.928035]   CPU12 -> Queue 0
[    7.928036]   CPU13 -> Queue 0
[    7.928036]   CPU14 -> Queue 0
[    7.928037]   CPU15 -> Queue 0
[    7.928055] Performing sc map setup on q: ffff8804624f0000 hctx: ffff880462938400 i: 0
[    7.928087] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928088] Allocated blk-mq req: ffff88046244a240, req->tag: 59
[    7.928089] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928092] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928093] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928094] Allocated blk-mq req: ffff88046244a240, req->tag: 59
[    7.928096] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928097] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928099] sd 0:0:0:0: [sda] Sector size 0 reported, assuming 512.
[    7.928100] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928101] Allocated blk-mq req: ffff88046244a240, req->tag: 59
[    7.928102] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928103] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928104] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928105] Allocated blk-mq req: ffff88046244a240, req->tag: 59
[    7.928106] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928107] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928108] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928109] Allocated blk-mq req: ffff88046244a240, req->tag: 59
[    7.928110] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928112] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928113] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928113] Allocated blk-mq req: ffff88046244a240, req->tag: 59
[    7.928114] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928116] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928117] sd 0:0:0:0: [sda] Asking for cache data failed
[    7.928118] sd 0:0:0:0: [sda] Assuming drive cache: write through
[    7.928174] scsi_mq_alloc_queue() complete !! >>>>>>>>>>>>>>>>>>>>>>>>>>>
[    7.928177] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928178] Allocated blk-mq req: ffff88046250ad40, req->tag: 63
[    7.928180] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928232] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928237] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928238] Allocated blk-mq req: ffff88046250ad40, req->tag: 63
[    7.928238] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928240] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928246] scsi 0:0:1:0: Direct-Access     ATA      ST9500530NS      CC04 PQ: 0 ANSI: 5
[    7.928336] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928337] Allocated blk-mq req: ffff88046250a7c0, req->tag: 61
[    7.928338] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928340] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928341] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928342] Allocated blk-mq req: ffff88046250a7c0, req->tag: 61
[    7.928344] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928346] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928347] sd 0:0:1:0: [sdb] Sector size 0 reported, assuming 512.
[    7.928349] sd 0:0:1:0: [sdb] 1 512-byte logical blocks: (512 B/512 B)
[    7.928351] sd 0:0:1:0: Attached scsi generic sg1 type 0
[    7.928351] sd 0:0:1:0: [sdb] 0-byte physical blocks
[    7.928352] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928353] Allocated blk-mq req: ffff88046250a7c0, req->tag: 61
[    7.928354] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928356] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928356] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928357] Allocated blk-mq req: ffff88046250a7c0, req->tag: 61
[    7.928362] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928364] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928366] Calling blk_mq_init_queue: scsi_mq_ops: ffffffff81ca13e0, queue_depth: 64, cmd_size: 296 SCSI cmd_size: 0
[    7.928366] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928367] Allocated blk-mq req: ffff88046250a7c0, req->tag: 61
[    7.928368] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928369] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928371] sd 0:0:1:0: [sdb] Write Protect is off
[    7.928372] sd 0:0:1:0: [sdb] Mode Sense: 00 00 00 00
[    7.928373] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928374] Allocated blk-mq req: ffff88046250a7c0, req->tag: 61
[    7.928375] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928376] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928377] sd 0:0:1:0: [sdb] Asking for cache data failed
[    7.928378] sd 0:0:1:0: [sdb] Assuming drive cache: write through
[    7.928394] blk-mq: CPU -> queue map
[    7.928394]   CPU 0 -> Queue 0
[    7.928395]   CPU 1 -> Queue 0
[    7.928395]   CPU 2 -> Queue 0
[    7.928396]   CPU 3 -> Queue 0
[    7.928396]   CPU 4 -> Queue 0
[    7.928396]   CPU 5 -> Queue 0
[    7.928397]   CPU 6 -> Queue 0
[    7.928397]   CPU 7 -> Queue 0
[    7.928398]   CPU 8 -> Queue 0
[    7.928398]   CPU 9 -> Queue 0
[    7.928399]   CPU10 -> Queue 0
[    7.928399]   CPU11 -> Queue 0
[    7.928399]   CPU12 -> Queue 0
[    7.928400]   CPU13 -> Queue 0
[    7.928400]   CPU14 -> Queue 0
[    7.928401]   CPU15 -> Queue 0
[    7.928420] Performing sc map setup on q: ffff8804624f08e0 hctx: ffff880462938a00 i: 0
[    7.928435] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928436] Allocated blk-mq req: ffff88046250a240, req->tag: 59
[    7.928437] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928439] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928441] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928441] Allocated blk-mq req: ffff88046250a240, req->tag: 59
[    7.928443] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928444] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928446] sd 0:0:1:0: [sdb] Sector size 0 reported, assuming 512.
[    7.928448] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928448] Allocated blk-mq req: ffff88046250a240, req->tag: 59
[    7.928450] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928451] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928452] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928452] Allocated blk-mq req: ffff88046250a240, req->tag: 59
[    7.928457] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928458] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928459] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928460] Allocated blk-mq req: ffff88046250a240, req->tag: 59
[    7.928461] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928462] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928463] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928464] Allocated blk-mq req: ffff88046250a240, req->tag: 59
[    7.928465] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928466] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928468] sd 0:0:1:0: [sdb] Asking for cache data failed
[    7.928469] sd 0:0:1:0: [sdb] Assuming drive cache: write through
[    7.928487] ------------[ cut here ]------------
[    7.928492] WARNING: at drivers/ata/libata-core.c:5038 ata_qc_issue+0x266/0x380()
[    7.928494] Modules linked in:
[    7.928496] CPU: 9 PID: 153 Comm: kworker/u50:5 Not tainted 3.10.0-rc5.nab+ #11
[    7.928497] Hardware name: Cisco Systems Inc R210-2121605W/R210-2121605W, BIOS C200.1.4.3j.0.020720132258 02/07/2013
[    7.928502] Workqueue: events_unbound async_run_entry_fn
[    7.928505]  0000000000000009 ffff88046241f588 ffffffff8162cc58 ffff88046241f5c8
[    7.928507]  ffffffff8104a200 ffff88046241f5d8 ffff880462b90000 0000000000000003
[    7.928509]  ffff880462b91c68 ffffffff81415450 ffff880462b90230 ffff88046241f5d8
[    7.928510] Call Trace:
[    7.928514]  [<ffffffff8162cc58>] dump_stack+0x19/0x1b
[    7.928518]  [<ffffffff8104a200>] warn_slowpath_common+0x70/0xa0
[    7.928521]  [<ffffffff81415450>] ? ata_scsi_set_sense.constprop.26+0x30/0x30
[    7.928523]  [<ffffffff8104a24a>] warn_slowpath_null+0x1a/0x20
[    7.928525]  [<ffffffff8140f586>] ata_qc_issue+0x266/0x380
[    7.928526]  [<ffffffff814155b3>] ? ata_scsi_rw_xlat+0x163/0x210
[    7.928528]  [<ffffffff81415450>] ? ata_scsi_set_sense.constprop.26+0x30/0x30
[    7.928530]  [<ffffffff814140d7>] ata_scsi_translate+0xa7/0x180
[    7.928531] scsi_mq_alloc_queue() complete !! >>>>>>>>>>>>>>>>>>>>>>>>>>>
[    7.928533]  [<ffffffff814181f9>] ata_scsi_queuecmd+0xa9/0x2b0
[    7.928534] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928537]  [<ffffffff813ed956>] scsi_dispatch_cmd+0x1c6/0x310
[    7.928537] Allocated blk-mq req: ffff88046259ad40, req->tag: 63
[    7.928540]  [<ffffffff813f63bb>] scsi_mq_queue_rq+0x17b/0x280
[    7.928541] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928546]  [<ffffffff812c15c5>] __blk_mq_run_hw_queue+0x1b5/0x3a0
[    7.928549]  [<ffffffff812c1c65>] blk_mq_run_hw_queue+0x35/0x40
[    7.928550]  [<ffffffff812c1fbb>] blk_mq_make_request+0x34b/0x4a0
[    7.928554]  [<ffffffff812b74f2>] generic_make_request+0xc2/0x110
[    7.928556]  [<ffffffff812b7a1b>] submit_bio+0x7b/0x160
[    7.928560]  [<ffffffff811baa8d>] ? bio_alloc_bioset+0x9d/0x1b0
[    7.928562]  [<ffffffff811b576e>] _submit_bh+0x13e/0x200
[    7.928564]  [<ffffffff811b5840>] submit_bh+0x10/0x20
[    7.928566]  [<ffffffff811b754d>] block_read_full_page+0x21d/0x350
[    7.928568]  [<ffffffff811bbff0>] ? I_BDEV+0x10/0x10
[    7.928571]  [<ffffffff8113be73>] ? __inc_zone_page_state+0x33/0x40
[    7.928573]  [<ffffffff8111f4bf>] ? add_to_page_cache_locked+0xdf/0x190
[    7.928575]  [<ffffffff811bc4a0>] ? blkdev_write_begin+0x30/0x30
[    7.928577]  [<ffffffff811bc4b8>] blkdev_readpage+0x18/0x20
[    7.928579]  [<ffffffff8111fcfa>] do_read_cache_page+0x7a/0x170
[    7.928581]  [<ffffffff8112837a>] ? __alloc_pages_nodemask+0x17a/0xad0
[    7.928583]  [<ffffffff8111fe09>] read_cache_page_async+0x19/0x20
[    7.928585]  [<ffffffff8112015e>] read_cache_page+0xe/0x20
[    7.928588]  [<ffffffff812c80cd>] read_dev_sector+0x2d/0x90
[    7.928590]  [<ffffffff812cdc4c>] read_lba+0xec/0x190
[    7.928592]  [<ffffffff812ce255>] ? efi_partition+0xe5/0x5f0
[    7.928594]  [<ffffffff812ce26f>] efi_partition+0xff/0x5f0
[    7.928596]  [<ffffffff812e9c34>] ? snprintf+0x34/0x40
[    7.928598]  [<ffffffff812ce170>] ? is_gpt_valid+0x480/0x480
[    7.928600]  [<ffffffff812c9138>] check_partition+0x108/0x220
[    7.928602]  [<ffffffff812c8d44>] rescan_partitions+0xb4/0x2c0
[    7.928604]  [<ffffffff811bda35>] __blkdev_get+0x375/0x4b0
[    7.928606]  [<ffffffff8119e447>] ? inode_init_always+0x107/0x1c0
[    7.928608]  [<ffffffff811bc010>] ? blkdev_get_block+0x20/0x20
[    7.928610]  [<ffffffff811bdd05>] blkdev_get+0x195/0x2e0
[    7.928612]  [<ffffffff8119f0c7>] ? unlock_new_inode+0x47/0x70
[    7.928613]  [<ffffffff811bcff0>] ? bdget+0x120/0x140
[    7.928615]  [<ffffffff812c6531>] add_disk+0x391/0x490
[    7.928618]  [<ffffffff81402c4a>] sd_probe_async+0x13a/0x230
[    7.928620]  [<ffffffff81075de6>] async_run_entry_fn+0x46/0x140
[    7.928623]  [<ffffffff810683c4>] process_one_work+0x174/0x400
[    7.928624]  [<ffffffff81068acc>] worker_thread+0x11c/0x370
[    7.928626]  [<ffffffff810689b0>] ? rescuer_thread+0x320/0x320
[    7.928629]  [<ffffffff8106f410>] kthread+0xc0/0xd0
[    7.928631]  [<ffffffff8106f350>] ? flush_kthread_worker+0x80/0x80
[    7.928633]  [<ffffffff8163b2dc>] ret_from_fork+0x7c/0xb0
[    7.928635]  [<ffffffff8106f350>] ? flush_kthread_worker+0x80/0x80
[    7.928638] ---[ end trace 9f4b3fe3fb787a07 ]---
[    7.931709] scsi_execute(): Calling blk_mq_free_request >>>
[    7.931715] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.931715] Allocated blk-mq req: ffff88046259ad40, req->tag: 63
[    7.931716] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.931718] scsi_execute(): Calling blk_mq_free_request >>>
[    7.931724] scsi 1:0:0:0: Direct-Access     ATA      ST9500530NS      CC04 PQ: 0 ANSI: 5
[    7.931786] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.931787] Allocated blk-mq req: ffff88046259a7c0, req->tag: 61
[    7.931788] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.931791] scsi_execute(): Calling blk_mq_free_request >>>
[    7.931792] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.931792] Allocated blk-mq req: ffff88046259a7c0, req->tag: 61
[    7.931795] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.931796] scsi_execute(): Calling blk_mq_free_request >>>
[    7.931798] sd 1:0:0:0: [sdc] Sector size 0 reported, assuming 512.
[    7.931800] sd 1:0:0:0: [sdc] 1 512-byte logical blocks: (512 B/512 B)
[    7.931801] sd 1:0:0:0: [sdc] 0-byte physical blocks
[    7.931802] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.931803] Allocated blk-mq req: ffff88046259a7c0, req->tag: 61
[    7.931804] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.931807] scsi_execute(): Calling blk_mq_free_request >>>
[    7.931808] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.931808] Allocated blk-mq req: ffff88046259a7c0, req->tag: 61
[    7.931809] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.931813] scsi_execute(): Calling blk_mq_free_request >>>
[    7.931814] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.931815] Allocated blk-mq req: ffff88046259a7c0, req->tag: 61
[    7.931816] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.931821] scsi_execute(): Calling blk_mq_free_request >>>
[    7.931822] sd 1:0:0:0: [sdc] Write Protect is off
[    7.931823] sd 1:0:0:0: [sdc] Mode Sense: 00 00 00 00
[    7.931824] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.931825] Allocated blk-mq req: ffff88046259a7c0, req->tag: 61
[    7.931826] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.931827] scsi_execute(): Calling blk_mq_free_request >>>
[    7.931829] sd 1:0:0:0: [sdc] Asking for cache data failed
[    7.931830] sd 1:0:0:0: Attached scsi generic sg2 type 0
[    7.931831] sd 1:0:0:0: [sdc] Assuming drive cache: write through
[    7.931837] Calling blk_mq_init_queue: scsi_mq_ops: ffffffff81ca13e0, queue_depth: 64, cmd_size: 296 SCSI cmd_size: 0
[    7.931860] blk-mq: CPU -> queue map
[    7.931860]   CPU 0 -> Queue 0
[    7.931861]   CPU 1 -> Queue 0
[    7.931861]   CPU 2 -> Queue 0
[    7.931862]   CPU 3 -> Queue 0
[    7.931862]   CPU 4 -> Queue 0
[    7.931863]   CPU 5 -> Queue 0
[    7.931863]   CPU 6 -> Queue 0
[    7.931863]   CPU 7 -> Queue 0
[    7.931864]   CPU 8 -> Queue 0
[    7.931864]   CPU 9 -> Queue 0
[    7.931865]   CPU10 -> Queue 0
[    7.931865]   CPU11 -> Queue 0
[    7.931866]   CPU12 -> Queue 0
[    7.931866]   CPU13 -> Queue 0
[    7.931866]   CPU14 -> Queue 0
[    7.931867]   CPU15 -> Queue 0
[    7.931886] Performing sc map setup on q: ffff8804624308e0 hctx: ffff880462a14600 i: 0
[    7.931895] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.931896] Allocated blk-mq req: ffff88046259a7c0, req->tag: 61
[    7.931896] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.931899] scsi_execute(): Calling blk_mq_free_request >>>
[    7.931900] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.931901] Allocated blk-mq req: ffff88046259a7c0, req->tag: 61
[    7.931902] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.931904] scsi_execute(): Calling blk_mq_free_request >>>
[    7.931906] sd 1:0:0:0: [sdc] Sector size 0 reported, assuming 512.
[    7.931907] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.931908] Allocated blk-mq req: ffff88046259a7c0, req->tag: 61
[    7.931909] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.931910] scsi_execute(): Calling blk_mq_free_request >>>
[    7.931911] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.931912] Allocated blk-mq req: ffff88046259a7c0, req->tag: 61
[    7.931913] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.931917] scsi_execute(): Calling blk_mq_free_request >>>
[    7.931918] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.931919] Allocated blk-mq req: ffff88046259a7c0, req->tag: 61
[    7.931930] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.931932] scsi_execute(): Calling blk_mq_free_request >>>
[    7.931933] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.931933] Allocated blk-mq req: ffff88046259a7c0, req->tag: 61
[    7.931934] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.931937] scsi_execute(): Calling blk_mq_free_request >>>
[    7.931938] sd 1:0:0:0: [sdc] Asking for cache data failed
[    7.931940] sd 1:0:0:0: [sdc] Assuming drive cache: write through
[    7.932002] scsi_mq_alloc_queue() complete !! >>>>>>>>>>>>>>>>>>>>>>>>>>>
[    7.932006] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.932007] Allocated blk-mq req: ffff88046264ad40, req->tag: 63
[    7.932008] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.932042] scsi_execute(): Calling blk_mq_free_request >>>
[    7.932047] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.932048] Allocated blk-mq req: ffff88046264ad40, req->tag: 63
[    7.932048] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.932050] scsi_execute(): Calling blk_mq_free_request >>>
[    7.932056] scsi 1:0:1:0: Direct-Access     ATA      ST9500530NS      CC04 PQ: 0 ANSI: 5
[    7.932160] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.932161] Allocated blk-mq req: ffff88046264ad40, req->tag: 63
[    7.932161] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.932168] sd 1:0:1:0: Attached scsi generic sg3 type 0
[    7.932168] scsi_execute(): Calling blk_mq_free_request >>>
[    7.932170] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.932170] Allocated blk-mq req: ffff88046264ad40, req->tag: 63
[    7.932172] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.932175] scsi_execute(): Calling blk_mq_free_request >>>
[    7.932177] sd 1:0:1:0: [sdd] Sector size 0 reported, assuming 512.
[    7.932179] sd 1:0:1:0: [sdd] 1 512-byte logical blocks: (512 B/512 B)
[    7.932180] sd 1:0:1:0: [sdd] 0-byte physical blocks
[    7.932181] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.932182] Allocated blk-mq req: ffff88046264ad40, req->tag: 63
[    7.932183] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.932184] scsi_execute(): Calling blk_mq_free_request >>>
[    7.932185] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.932186] Allocated blk-mq req: ffff88046264ad40, req->tag: 63
[    7.932187] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.932190] scsi_execute(): Calling blk_mq_free_request >>>
[    7.932191] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.932192] Allocated blk-mq req: ffff88046264ad40, req->tag: 63
[    7.932193] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.932198] scsi_execute(): Calling blk_mq_free_request >>>
[    7.932200] sd 1:0:1:0: [sdd] Write Protect is off
[    7.932201] sd 1:0:1:0: [sdd] Mode Sense: 00 00 00 00
[    7.932202] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.932203] Allocated blk-mq req: ffff88046264ad40, req->tag: 63
[    7.932204] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.932206] scsi_execute(): Calling blk_mq_free_request >>>
[    7.932207] sd 1:0:1:0: [sdd] Asking for cache data failed
[    7.932208] sd 1:0:1:0: [sdd] Assuming drive cache: write through
[    7.932218] Calling blk_mq_init_queue: scsi_mq_ops: ffffffff81ca13e0, queue_depth: 64, cmd_size: 296 SCSI cmd_size: 0
[    7.932251] blk-mq: CPU -> queue map
[    7.932252]   CPU 0 -> Queue 0
[    7.932253]   CPU 1 -> Queue 0
[    7.932253]   CPU 2 -> Queue 0
[    7.932254]   CPU 3 -> Queue 0
[    7.932255]   CPU 4 -> Queue 0
[    7.932255]   CPU 5 -> Queue 0
[    7.932256]   CPU 6 -> Queue 0
[    7.932256]   CPU 7 -> Queue 0
[    7.932257]   CPU 8 -> Queue 0
[    7.932258]   CPU 9 -> Queue 0
[    7.932259]   CPU10 -> Queue 0
[    7.932259]   CPU11 -> Queue 0
[    7.932260]   CPU12 -> Queue 0
[    7.932260]   CPU13 -> Queue 0
[    7.932261]   CPU14 -> Queue 0
[    7.932262]   CPU15 -> Queue 0
[    7.932282] Performing sc map setup on q: ffff8804626f8000 hctx: ffff8802628f0200 i: 0
[    7.932286] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.932287] Allocated blk-mq req: ffff88046264ad40, req->tag: 63
[    7.932288] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.932293] scsi_execute(): Calling blk_mq_free_request >>>
[    7.932294] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.932295] Allocated blk-mq req: ffff88046264ad40, req->tag: 63
[    7.932296] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.932300] scsi_execute(): Calling blk_mq_free_request >>>
[    7.932302] sd 1:0:1:0: [sdd] Sector size 0 reported, assuming 512.
[    7.932303] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.932304] Allocated blk-mq req: ffff88046264ad40, req->tag: 63
[    7.932305] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.932308] scsi_execute(): Calling blk_mq_free_request >>>
[    7.932309] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.932310] Allocated blk-mq req: ffff88046264ad40, req->tag: 63
[    7.932311] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.932316] scsi_execute(): Calling blk_mq_free_request >>>
[    7.932317] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.932318] Allocated blk-mq req: ffff88046264ad40, req->tag: 63
[    7.932318] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.932324] scsi_execute(): Calling blk_mq_free_request >>>
[    7.932325] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.932325] Allocated blk-mq req: ffff88046264ad40, req->tag: 63
[    7.932326] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.932331] scsi_execute(): Calling blk_mq_free_request >>>
[    7.932333] sd 1:0:1:0: [sdd] Asking for cache data failed
[    7.932334] sd 1:0:1:0: [sdd] Assuming drive cache: write through
[    7.932394] scsi_mq_alloc_queue() complete !! >>>>>>>>>>>>>>>>>>>>>>>>>>>
[    7.932398] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.932399] Allocated blk-mq req: ffff88046271ad40, req->tag: 63
[    7.932400] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.943187]  sdb: sdb1
[    7.943191] sdb: p1 start 2048 is beyond EOD, enabling native capacity
[    7.943222] scsi_execute(): Calling blk_mq_free_request >>>
[    7.943230] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.943231] Allocated blk-mq req: ffff88046271ad40, req->tag: 63
[    7.943232] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.944750] scsi_execute(): Calling blk_mq_free_request >>>
[    7.944757] scsi 3:0:0:0: CD-ROM            Optiarc  DVD RW AD-7580S  FX04 PQ: 0 ANSI: 5
[    7.944813] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.944814] Allocated blk-mq req: ffff88046271ad40, req->tag: 63
[    7.944815] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.957010]  sdd: sdd1
[    7.957013] sdd: p1 start 2048 is beyond EOD, enabling native capacity
[    7.963106] usb 1-4: new high-speed USB device number 3 using ehci-pci
[    8.218646] usb 1-4: config 1 has no interfaces?
[    8.219146] usb 1-4: New USB device found, idVendor=0624, idProduct=0249
[    8.219148] usb 1-4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    8.219149] usb 1-4: Product: USB Composite Device-1
[    8.219150] usb 1-4: Manufacturer: Avocent
[    8.426474] usb 2-5: new high-speed USB device number 2 using ehci-pci
[    8.719067] usb 2-5: New USB device found, idVendor=04b4, idProduct=6560
[    8.719069] usb 2-5: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    8.719215] hub 2-5:1.0: USB hub found
[    8.719315] hub 2-5:1.0: 4 ports detected
[    8.957907] usb 4-1: new full-speed USB device number 2 using uhci_hcd
[    9.112050] usb 4-1: New USB device found, idVendor=0624, idProduct=0248
[    9.112051] usb 4-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    9.112053] usb 4-1: Product: USB Composite Device-0
[    9.112054] usb 4-1: Manufacturer: Avocent
[    9.112055] usb 4-1: SerialNumber: 20080930
[    9.329209] usb 2-3: new high-speed USB device number 3 using ehci-pci
[    9.470178] usb 2-3: New USB device found, idVendor=14dd, idProduct=0002
[    9.470180] usb 2-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    9.470181] usb 2-3: Product: Multidevice
[    9.470182] usb 2-3: Manufacturer: Peppercon AG
[    9.470183] usb 2-3: SerialNumber: 18EB7D234B1BCF34ADCBEA6091EC9869
[   10.986077] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   10.986079] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   10.986082] Allocated blk-mq req: ffff88046264a7c0, req->tag: 61
[   10.986083] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   10.986088] scsi_execute(): Calling blk_mq_free_request >>>
[   10.986090] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   10.986091] Allocated blk-mq req: ffff88046264a7c0, req->tag: 61
[   10.986094] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   10.986097] scsi_execute(): Calling blk_mq_free_request >>>
[   10.986100] sd 1:0:1:0: [sdd] Sector size 0 reported, assuming 512.
[   10.986101] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   10.986102] Allocated blk-mq req: ffff88046264a7c0, req->tag: 61
[   10.986104] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   10.986107] scsi_execute(): Calling blk_mq_free_request >>>
[   10.986110] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   10.986111] Allocated blk-mq req: ffff88046264a7c0, req->tag: 61
[   10.986113] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   10.986115] scsi_execute(): Calling blk_mq_free_request >>>
[   10.986116] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   10.986117] Allocated blk-mq req: ffff88046264a7c0, req->tag: 61
[   10.986118] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   10.986121] scsi_execute(): Calling blk_mq_free_request >>>
[   10.986124] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   10.986124] Allocated blk-mq req: ffff88046264a7c0, req->tag: 61
[   10.986126] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   10.986129] scsi_execute(): Calling blk_mq_free_request >>>
[   10.986132] sd 1:0:1:0: [sdd] Asking for cache data failed
[   10.986133] sd 1:0:1:0: [sdd] Assuming drive cache: write through
[   10.986155] hub 7-0:1.0: USB hub found
[   10.986158] hub 7-0:1.0: 2 ports detected
[   10.986355] uhci_hcd 0000:00:1d.2: setting latency timer to 64
[   10.986358] uhci_hcd 0000:00:1d.2: UHCI Host Controller
[   10.986397] uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 8
[   10.986428] uhci_hcd 0000:00:1d.2: irq 16, io base 0x00005020
[   10.986429]  sdd: sdd1
[   10.986432] sdd: p1 start 2048 is beyond EOD, truncated
[   10.986462] usb usb8: New USB device found, idVendor=1d6b, idProduct=0001
[   10.986464] usb usb8: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   10.986465] usb usb8: Product: UHCI Host Controller
[   10.986466] usb usb8: Manufacturer: Linux 3.10.0-rc5.nab+ uhci_hcd
[   10.986467] usb usb8: SerialNumber: 0000:00:1d.2
[   10.986547] hub 8-0:1.0: USB hub found
[   10.986550] hub 8-0:1.0: 2 ports detected
[   10.986570] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   10.986572] Allocated blk-mq req: ffff88046264a500, req->tag: 60
[   10.986572] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   10.986576] scsi_execute(): Calling blk_mq_free_request >>>
[   10.986581] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   10.986582] Allocated blk-mq req: ffff88046264a500, req->tag: 60
[   10.986582] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   10.986584] scsi_execute(): Calling blk_mq_free_request >>>
[   10.986585] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   10.986586] Allocated blk-mq req: ffff88046264a500, req->tag: 60
[   10.986588] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   10.986590] scsi_execute(): Calling blk_mq_free_request >>>
[   10.986593] sd 1:0:1:0: [sdd] Sector size 0 reported, assuming 512.
[   10.986595] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   10.986595] Allocated blk-mq req: ffff88046264a500, req->tag: 60
[   10.986598] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   10.986600] scsi_execute(): Calling blk_mq_free_request >>>
[   10.986601] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   10.986602] Allocated blk-mq req: ffff88046264a500, req->tag: 60
[   10.986603] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   10.986606] scsi_execute(): Calling blk_mq_free_request >>>
[   10.986607] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   10.986608] Allocated blk-mq req: ffff88046264a500, req->tag: 60
[   10.986609] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   10.986612] scsi_execute(): Calling blk_mq_free_request >>>
[   10.986614] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   10.986615] Allocated blk-mq req: ffff88046264a500, req->tag: 60
[   10.986617] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   10.986619] scsi_execute(): Calling blk_mq_free_request >>>
[   10.986620] sd 1:0:1:0: [sdd] Asking for cache data failed
[   10.986622] sd 1:0:1:0: [sdd] Assuming drive cache: write through
[   10.986623] sd 1:0:1:0: [sdd] Attached SCSI disk
[   10.986693] usbcore: registered new interface driver usbserial
[   10.986699] usbcore: registered new interface driver usbserial_generic
[   10.986706] usbserial: USB Serial support registered for generic
[   10.986730] i8042: PNP: No PS/2 controller found. Probing ports directly.
[   11.523406] Allocated blk-mq req: ffff880462509f80, req->tag: 58
[   11.530116] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   11.538869] scsi_execute(): Calling blk_mq_free_request >>>
[   11.545095] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   11.551996] Allocated blk-mq req: ffff880462509f80, req->tag: 58
[   11.558705] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   11.567452] scsi_execute(): Calling blk_mq_free_request >>>
[   11.573677] sd 0:0:1:0: [sdb] Sector size 0 reported, assuming 512.
[   11.580679] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   11.587580] Allocated blk-mq req: ffff880462509f80, req->tag: 58
[   11.594288] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   11.603035] scsi_execute(): Calling blk_mq_free_request >>>
[   11.609259] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   11.616161] Allocated blk-mq req: ffff880462509f80, req->tag: 58
[   11.622871] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   11.631617] scsi_execute(): Calling blk_mq_free_request >>>
[   11.637840] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   11.644741] Allocated blk-mq req: ffff880462509f80, req->tag: 58
[   11.651450] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   11.660198] scsi_execute(): Calling blk_mq_free_request >>>
[   11.666421] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   11.673324] Allocated blk-mq req: ffff880462509f80, req->tag: 58
[   11.680034] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   11.688782] scsi_execute(): Calling blk_mq_free_request >>>
[   11.695005] sd 0:0:1:0: [sdb] Asking for cache data failed
[   11.701132] sd 0:0:1:0: [sdb] Assuming drive cache: write through
[   11.708212]  sdb: sdb1
[   11.710851] sdb: p1 start 2048 is beyond EOD, truncated
[   11.716790] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   11.723696] Allocated blk-mq req: ffff880462509cc0, req->tag: 57
[   11.723744] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   11.723745] Allocated blk-mq req: ffff880462509740, req->tag: 55
[   11.723746] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   11.723749] scsi_execute(): Calling blk_mq_free_request >>>
[   11.723750] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   11.723751] Allocated blk-mq req: ffff880462509740, req->tag: 55
[   11.723754] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   11.723755] scsi_execute(): Calling blk_mq_free_request >>>
[   11.723757] sd 0:0:1:0: [sdb] Sector size 0 reported, assuming 512.
[   11.723759] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   11.723760] Allocated blk-mq req: ffff880462509740, req->tag: 55
[   11.723761] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   11.723762] scsi_execute(): Calling blk_mq_free_request >>>
[   11.723763] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   11.723764] Allocated blk-mq req: ffff880462509740, req->tag: 55
[   11.723765] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   11.723766] scsi_execute(): Calling blk_mq_free_request >>>
[   11.723767] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   11.723768] Allocated blk-mq req: ffff880462509740, req->tag: 55
[   11.723769] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   11.723770] scsi_execute(): Calling blk_mq_free_request >>>
[   11.723771] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[   11.723772] Allocated blk-mq req: ffff880462509740, req->tag: 55
[   11.723773] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   11.723774] scsi_execute(): Calling blk_mq_free_request >>>
[   11.723776] sd 0:0:1:0: [sdb] Asking for cache data failed
[   11.723777] sd 0:0:1:0: [sdb] Assuming drive cache: write through
[   11.723778] sd 0:0:1:0: [sdb] Attached SCSI disk
[   11.926903] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[   11.935652] scsi_execute(): Calling blk_mq_free_request >>>
[   12.033094] i8042: No controller found
[   12.037373] mousedev: PS/2 mouse device common for all mice
[   12.043839] rtc_cmos 00:02: RTC can wake from S4
[   12.049118] rtc_cmos 00:02: rtc core: registered rtc_cmos as rtc0
[   12.055954] rtc_cmos 00:02: alarms up to one month, y3k, 114 bytes nvram, hpet irqs
[   12.064566] device-mapper: uevent: version 1.0.3
[   12.069788] device-mapper: ioctl: 4.24.0-ioctl (2013-01-15) initialised: dm-devel@redhat.com
[   12.079541] cpuidle: using governor ladder
[   12.084416] cpuidle: using governor menu
[   12.089357] hidraw: raw HID events driver (C) Jiri Kosina
[   12.100665] input: Avocent USB Composite Device-0 as /devices/pci0000:00/0000:00:1a.1/usb4/4-1/4-1:1.0/input/input2
[   12.112415] hid-generic 0003:0624:0248.0001: input,hidraw0: USB HID v1.00 Keyboard [Avocent USB Composite Device-0] on usb-0000:00:1a.1-1/input0
[   12.130629] input: Avocent USB Composite Device-0 as /devices/pci0000:00/0000:00:1a.1/usb4/4-1/4-1:1.1/input/input3
[   12.142391] hid-generic 0003:0624:0248.0002: input,hidraw1: USB HID v1.00 Mouse [Avocent USB Composite Device-0] on usb-0000:00:1a.1-1/input1
[   12.161647] input: Avocent USB Composite Device-0 as /devices/pci0000:00/0000:00:1a.1/usb4/4-1/4-1:1.2/input/input4
[   12.173406] hid-generic 0003:0624:0248.0003: input,hidraw2: USB HID v1.00 Mouse [Avocent USB Composite Device-0] on usb-0000:00:1a.1-1/input2
[   12.188165] input: Peppercon AG Multidevice as /devices/pci0000:00/0000:00:1d.7/usb2/2-3/2-3:1.0/input/input5
[   12.199328] hid-generic 0003:14DD:0002.0004: input,hidraw3: USB HID v1.01 Keyboard [Peppercon AG Multidevice] on usb-0000:00:1d.7-3/input0
[   12.214075] input: Peppercon AG Multidevice as /devices/pci0000:00/0000:00:1d.7/usb2/2-3/2-3:1.1/input/input6
[   12.225243] hid-generic 0003:14DD:0002.0005: input,hidraw4: USB HID v1.01 Mouse [Peppercon AG Multidevice] on usb-0000:00:1d.7-3/input1
[   12.238856] usbcore: registered new interface driver usbhid
[   12.245087] usbhid: USB HID core driver
[   12.249415] drop_monitor: Initializing network drop monitor service
[   12.256511] ip_tables: (C) 2000-2006 Netfilter Core Team
[   12.262688] TCP: cubic registered
[   12.266394] Initializing XFRM netlink socket
[   12.271252] NET: Registered protocol family 10
[   12.276437] mip6: Mobile IPv6
[   12.279754] NET: Registered protocol family 17
[   12.285687] PM: Hibernation image not present or could not be loaded.
[   12.292888] registered taskstats version 1
[   12.299282]   Magic number: 5:412:283
[   12.304146] rtc_cmos 00:02: setting system clock to 2013-07-16 14:17:22 UTC (1373984242)

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Nicholas A. Bellinger]

On Tue, 2013-07-16 at 20:32 +0200, Alexander Gordeev wrote:
> On Fri, Jul 12, 2013 at 10:20:12PM -0700, Nicholas A. Bellinger wrote:
> > On Fri, 2013-07-12 at 09:46 +0200, Alexander Gordeev wrote:
> > > > > diff --git a/drivers/scsi/scsi-mq.c b/drivers/scsi/scsi-mq.c
> > > > > index ca6ff67..d8cc7a4 100644
> > > > > --- a/drivers/scsi/scsi-mq.c
> > > > > +++ b/drivers/scsi/scsi-mq.c
> > > > > @@ -155,6 +155,7 @@ void scsi_mq_done(struct scsi_cmnd *sc)
> > > > >  static struct blk_mq_ops scsi_mq_ops = {
> > > > >  	.queue_rq	= scsi_mq_queue_rq,
> > > > >  	.map_queue	= blk_mq_map_queue,
> > > > > +	.timeout	= scsi_times_out,
> > > > >  	.alloc_hctx	= blk_mq_alloc_single_hw_queue,
> > > > >  	.free_hctx	= blk_mq_free_single_hw_queue,
> > > > >  };
> > 
> > So your actually triggering a blk-mq timeout with ata_piix..?
> 
> No.
> That is to avoid a NULL-pointer assignment from ->timeout elsewhere.
> In fact I return -ENODEV for sr_probe() to not hit it.
> 
> > That is why scsi-mq still uses blk_execute_rq() for this reason, and
> > this will need be addressed in order to safely use blk_mq_execute_rq()
> > in the above context.
> 
> Got it.
> 
> > Do you have an OOPs backtrace handy to post w/o the last two changes in
> > place..?
> 
> Attaching the output. No oops actually (due to aforementioned .timeout).
> 

Hi Alexander,

Thanks for the logs.  I'm In-lining some of the output here for
reference:

[    7.927596] Calling blk_mq_init_queue: scsi_mq_ops: ffffffff81ca13e0, queue_depth: 64, cmd_size: 296 SCSI cmd_size: 0

Just FYI, a SCSI cmd_size of zero here means that scsi-mq will not be
providing pre-allocated LLD descriptors (located at scsi_cmnd->SCp.ptr)
for use by libata driver code.

That is fine for initial testing, but libata will eventually want to
take advantage of scsi_host_template->cmd_size = sizeof(ata_queued_cmd)
in order to remove (all) memory allocations from the I/O fast-path.

[    7.927639] blk-mq: CPU -> queue map
[    7.927640]   CPU 0 -> Queue 0
[    7.927640]   CPU 1 -> Queue 0
[    7.927640]   CPU 2 -> Queue 0
[    7.927641]   CPU 3 -> Queue 0
[    7.927641]   CPU 4 -> Queue 0
[    7.927642]   CPU 5 -> Queue 0
[    7.927642]   CPU 6 -> Queue 0
[    7.927643]   CPU 7 -> Queue 0
[    7.927643]   CPU 8 -> Queue 0
[    7.927643]   CPU 9 -> Queue 0
[    7.927644]   CPU10 -> Queue 0
[    7.927644]   CPU11 -> Queue 0
[    7.927645]   CPU12 -> Queue 0
[    7.927645]   CPU13 -> Queue 0
[    7.927646]   CPU14 -> Queue 0
[    7.927646]   CPU15 -> Queue 0
[    7.927673] Performing sc map setup on q: ffff880462430000 hctx: ffff880462a14200 i: 0
[    7.927780] scsi_mq_alloc_queue() complete !! >>>>>>>>>>>>>>>>>>>>>>>>>>>
[    7.927784] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.927785] Allocated blk-mq req: ffff88046244ad40, req->tag: 63
[    7.927790] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.927803] scsi_execute(): Calling blk_mq_free_request >>>
[    7.927815] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.927816] Allocated blk-mq req: ffff88046244ad40, req->tag: 63
[    7.927817] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.927818] scsi_execute(): Calling blk_mq_free_request >>>
[    7.927826] scsi 0:0:0:0: Direct-Access     ATA      ST9500530NS      CC03 PQ: 0 ANSI: 5

OK, so INQUIRY response payload is looking as expected here.

[    7.927944] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.927946] Allocated blk-mq req: ffff88046244a7c0, req->tag: 61
[    7.927946] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.927949] scsi_execute(): Calling blk_mq_free_request >>>
[    7.927951] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.927952] Allocated blk-mq req: ffff88046244a7c0, req->tag: 61
[    7.927955] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.927958] scsi_execute(): Calling blk_mq_free_request >>>
[    7.927960] sd 0:0:0:0: [sda] Sector size 0 reported, assuming 512.
[    7.927964] sd 0:0:0:0: [sda] 1 512-byte logical blocks: (512 B/512 B)
[    7.927965] sd 0:0:0:0: [sda] 0-byte physical blocks

Strange..  READ_CAPACITY appears to be returning a payload as zeros..?

[    7.927966] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.927967] Allocated blk-mq req: ffff88046244a7c0, req->tag: 61
[    7.927968] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.927970] scsi_execute(): Calling blk_mq_free_request >>>
[    7.927970] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.927971] Allocated blk-mq req: ffff88046244a7c0, req->tag: 61
[    7.927972] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.927973] scsi_execute(): Calling blk_mq_free_request >>>
[    7.927975] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.927976] Allocated blk-mq req: ffff88046244a7c0, req->tag: 61
[    7.927977] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.927979] scsi_execute(): Calling blk_mq_free_request >>>
[    7.927981] sd 0:0:0:0: [sda] Write Protect is off
[    7.927982] sd 0:0:0:0: [sda] Mode Sense: 00 00 00 00

Ditto here..

[    7.927983] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.927984] Allocated blk-mq req: ffff88046244a7c0, req->tag: 61
[    7.927985] sd 0:0:0:0: Attached scsi generic sg0 type 0
[    7.927986] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.927987] scsi_execute(): Calling blk_mq_free_request >>>
[    7.927989] sd 0:0:0:0: [sda] Asking for cache data failed
[    7.927990] sd 0:0:0:0: [sda] Assuming drive cache: write through

and here as well..

Not sure why yet some control CDBs are getting back the expected
payload, while others are returning zeros..

Also, looking at the included stack back-trace:

[    7.928394] blk-mq: CPU -> queue map
[    7.928394]   CPU 0 -> Queue 0
[    7.928395]   CPU 1 -> Queue 0
[    7.928395]   CPU 2 -> Queue 0
[    7.928396]   CPU 3 -> Queue 0
[    7.928396]   CPU 4 -> Queue 0
[    7.928396]   CPU 5 -> Queue 0
[    7.928397]   CPU 6 -> Queue 0
[    7.928397]   CPU 7 -> Queue 0
[    7.928398]   CPU 8 -> Queue 0
[    7.928398]   CPU 9 -> Queue 0
[    7.928399]   CPU10 -> Queue 0
[    7.928399]   CPU11 -> Queue 0
[    7.928399]   CPU12 -> Queue 0
[    7.928400]   CPU13 -> Queue 0
[    7.928400]   CPU14 -> Queue 0
[    7.928401]   CPU15 -> Queue 0
[    7.928420] Performing sc map setup on q: ffff8804624f08e0 hctx: ffff880462938a00 i: 0
[    7.928435] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928436] Allocated blk-mq req: ffff88046250a240, req->tag: 59
[    7.928437] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928439] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928441] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928441] Allocated blk-mq req: ffff88046250a240, req->tag: 59
[    7.928443] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928444] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928446] sd 0:0:1:0: [sdb] Sector size 0 reported, assuming 512.
[    7.928448] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928448] Allocated blk-mq req: ffff88046250a240, req->tag: 59
[    7.928450] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928451] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928452] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928452] Allocated blk-mq req: ffff88046250a240, req->tag: 59
[    7.928457] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928458] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928459] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928460] Allocated blk-mq req: ffff88046250a240, req->tag: 59
[    7.928461] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928462] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928463] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928464] Allocated blk-mq req: ffff88046250a240, req->tag: 59
[    7.928465] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928466] scsi_execute(): Calling blk_mq_free_request >>>
[    7.928468] sd 0:0:1:0: [sdb] Asking for cache data failed
[    7.928469] sd 0:0:1:0: [sdb] Assuming drive cache: write through
[    7.928487] ------------[ cut here ]------------
[    7.928492] WARNING: at drivers/ata/libata-core.c:5038 ata_qc_issue+0x266/0x380()

Here is the code in question:

void ata_qc_issue(struct ata_queued_cmd *qc)
{
        struct ata_port *ap = qc->ap;
        struct ata_link *link = qc->dev->link;
        u8 prot = qc->tf.protocol;

        /* Make sure only one non-NCQ command is outstanding.  The
         * check is skipped for old EH because it reuses active qc to
         * request ATAPI sense.
         */
        WARN_ON_ONCE(ap->ops->error_handler && ata_tag_valid(link->active_tag));

	....
}

So I think that ata_tag_valid() is triggering this WARN_ON_ONCE due to
the default queue_depth setting in scsi-mq.c:scsi_mq_alloc_queue(),
which is queueing more than a single outstanding struct scsi_cmnd at a
time into the underlying LLD.  Note in this value is currently
hard-coded to:

        sdev->sdev_mq_reg.queue_depth = 64;

This value should actually be coming from what the hardware is
advertising, eg:

   min(scsi_host_template->cmd_per_lun, scsi_host_template->can_queue)

but I'd recommend to try to hardcode this value to 1 for the moment in
order to match what ata_piix is reporting to SCSI.

Also, just to be safe, please also disable scsi-generic
(CONFIG_CHR_DEV_SG) in your kernel config, as it's not hooked up to
scsi-mq just yet, and may be causing problems elsewhere.

Thanks!

--nab

[    7.928494] Modules linked in:
[    7.928496] CPU: 9 PID: 153 Comm: kworker/u50:5 Not tainted 3.10.0-rc5.nab+ #11
[    7.928497] Hardware name: Cisco Systems Inc R210-2121605W/R210-2121605W, BIOS C200.1.4.3j.0.020720132258 02/07/2013
[    7.928502] Workqueue: events_unbound async_run_entry_fn
[    7.928505]  0000000000000009 ffff88046241f588 ffffffff8162cc58 ffff88046241f5c8
[    7.928507]  ffffffff8104a200 ffff88046241f5d8 ffff880462b90000 0000000000000003
[    7.928509]  ffff880462b91c68 ffffffff81415450 ffff880462b90230 ffff88046241f5d8
[    7.928510] Call Trace:
[    7.928514]  [<ffffffff8162cc58>] dump_stack+0x19/0x1b
[    7.928518]  [<ffffffff8104a200>] warn_slowpath_common+0x70/0xa0
[    7.928521]  [<ffffffff81415450>] ? ata_scsi_set_sense.constprop.26+0x30/0x30
[    7.928523]  [<ffffffff8104a24a>] warn_slowpath_null+0x1a/0x20
[    7.928525]  [<ffffffff8140f586>] ata_qc_issue+0x266/0x380
[    7.928526]  [<ffffffff814155b3>] ? ata_scsi_rw_xlat+0x163/0x210
[    7.928528]  [<ffffffff81415450>] ? ata_scsi_set_sense.constprop.26+0x30/0x30
[    7.928530]  [<ffffffff814140d7>] ata_scsi_translate+0xa7/0x180
[    7.928531] scsi_mq_alloc_queue() complete !! >>>>>>>>>>>>>>>>>>>>>>>>>>>
[    7.928533]  [<ffffffff814181f9>] ata_scsi_queuecmd+0xa9/0x2b0
[    7.928534] Entering scsi_execute with q->mq_ops: ffffffff81ca13e0
[    7.928537]  [<ffffffff813ed956>] scsi_dispatch_cmd+0x1c6/0x310
[    7.928537] Allocated blk-mq req: ffff88046259ad40, req->tag: 63
[    7.928540]  [<ffffffff813f63bb>] scsi_mq_queue_rq+0x17b/0x280
[    7.928541] Calling blk_mq_insert_request from blk_execute_rq_nowait >>>>>>>>>>>>>>>>
[    7.928546]  [<ffffffff812c15c5>] __blk_mq_run_hw_queue+0x1b5/0x3a0
[    7.928549]  [<ffffffff812c1c65>] blk_mq_run_hw_queue+0x35/0x40
[    7.928550]  [<ffffffff812c1fbb>] blk_mq_make_request+0x34b/0x4a0
[    7.928554]  [<ffffffff812b74f2>] generic_make_request+0xc2/0x110
[    7.928556]  [<ffffffff812b7a1b>] submit_bio+0x7b/0x160
[    7.928560]  [<ffffffff811baa8d>] ? bio_alloc_bioset+0x9d/0x1b0
[    7.928562]  [<ffffffff811b576e>] _submit_bh+0x13e/0x200
[    7.928564]  [<ffffffff811b5840>] submit_bh+0x10/0x20
[    7.928566]  [<ffffffff811b754d>] block_read_full_page+0x21d/0x350
[    7.928568]  [<ffffffff811bbff0>] ? I_BDEV+0x10/0x10
[    7.928571]  [<ffffffff8113be73>] ? __inc_zone_page_state+0x33/0x40
[    7.928573]  [<ffffffff8111f4bf>] ? add_to_page_cache_locked+0xdf/0x190
[    7.928575]  [<ffffffff811bc4a0>] ? blkdev_write_begin+0x30/0x30
[    7.928577]  [<ffffffff811bc4b8>] blkdev_readpage+0x18/0x20
[    7.928579]  [<ffffffff8111fcfa>] do_read_cache_page+0x7a/0x170
[    7.928581]  [<ffffffff8112837a>] ? __alloc_pages_nodemask+0x17a/0xad0
[    7.928583]  [<ffffffff8111fe09>] read_cache_page_async+0x19/0x20
[    7.928585]  [<ffffffff8112015e>] read_cache_page+0xe/0x20
[    7.928588]  [<ffffffff812c80cd>] read_dev_sector+0x2d/0x90
[    7.928590]  [<ffffffff812cdc4c>] read_lba+0xec/0x190
[    7.928592]  [<ffffffff812ce255>] ? efi_partition+0xe5/0x5f0
[    7.928594]  [<ffffffff812ce26f>] efi_partition+0xff/0x5f0
[    7.928596]  [<ffffffff812e9c34>] ? snprintf+0x34/0x40
[    7.928598]  [<ffffffff812ce170>] ? is_gpt_valid+0x480/0x480
[    7.928600]  [<ffffffff812c9138>] check_partition+0x108/0x220
[    7.928602]  [<ffffffff812c8d44>] rescan_partitions+0xb4/0x2c0
[    7.928604]  [<ffffffff811bda35>] __blkdev_get+0x375/0x4b0
[    7.928606]  [<ffffffff8119e447>] ? inode_init_always+0x107/0x1c0
[    7.928608]  [<ffffffff811bc010>] ? blkdev_get_block+0x20/0x20
[    7.928610]  [<ffffffff811bdd05>] blkdev_get+0x195/0x2e0
[    7.928612]  [<ffffffff8119f0c7>] ? unlock_new_inode+0x47/0x70
[    7.928613]  [<ffffffff811bcff0>] ? bdget+0x120/0x140
[    7.928615]  [<ffffffff812c6531>] add_disk+0x391/0x490
[    7.928618]  [<ffffffff81402c4a>] sd_probe_async+0x13a/0x230
[    7.928620]  [<ffffffff81075de6>] async_run_entry_fn+0x46/0x140
[    7.928623]  [<ffffffff810683c4>] process_one_work+0x174/0x400
[    7.928624]  [<ffffffff81068acc>] worker_thread+0x11c/0x370
[    7.928626]  [<ffffffff810689b0>] ? rescuer_thread+0x320/0x320
[    7.928629]  [<ffffffff8106f410>] kthread+0xc0/0xd0
[    7.928631]  [<ffffffff8106f350>] ? flush_kthread_worker+0x80/0x80
[    7.928633]  [<ffffffff8163b2dc>] ret_from_fork+0x7c/0xb0
[    7.928635]  [<ffffffff8106f350>] ? flush_kthread_worker+0x80/0x80
[    7.928638] ---[ end trace 9f4b3fe3fb787a07 ]---

> > I *very* much recommend doing the same if at all possible for ata_piix
> > scsi-mq development + testing, as you'll want to be very careful when
> > using a real file-system on top of this early alpha code.
> 
> Thank you for the warning.
> Getting to writing CDBs would be of success :)
> 
> > --nab
> > 
> 

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

On Tue, Jul 16, 2013 at 02:38:03PM -0700, Nicholas A. Bellinger wrote:
> [    7.927818] scsi_execute(): Calling blk_mq_free_request >>>
> [    7.927826] scsi 0:0:0:0: Direct-Access     ATA      ST9500530NS      CC03 PQ: 0 ANSI: 5
> 
> OK, so INQUIRY response payload is looking as expected here.

Yep. It is not on the top of my head, but I remember something like INQUIRYs
are emulated and thus do not have payload.

> [    7.927960] sd 0:0:0:0: [sda] Sector size 0 reported, assuming 512.
> [    7.927964] sd 0:0:0:0: [sda] 1 512-byte logical blocks: (512 B/512 B)
> [    7.927965] sd 0:0:0:0: [sda] 0-byte physical blocks
> 
> Strange..  READ_CAPACITY appears to be returning a payload as zeros..?

Yep. Because blk_execute_rq() does not put the proper callback and data do
not get copied from sg's to bounce buffer. That is why I tried to use
blk_mq_execute_rq() instead. Once I do that, data start getting read and
booting stops elsewhere.

Of course, I was suspecting that change alone is not valid and wondered
about the status of scsi-mq in the first place, and if more changes are
coming.

So I it turns out "req->errors + req->resid_len" issue (you described
earlier) needs to be addressed before going forward with libata (only?).

> Not sure why yet some control CDBs are getting back the expected
> payload, while others are returning zeros..

Bio buffers do not get updated from callback.

> Also, looking at the included stack back-trace:

[...]

Thanks a lot for these and other your comments, Nicholas!

-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Nicholas A. Bellinger]

On Wed, 2013-07-17 at 18:19 +0200, Alexander Gordeev wrote:
> On Tue, Jul 16, 2013 at 02:38:03PM -0700, Nicholas A. Bellinger wrote:
> > [    7.927818] scsi_execute(): Calling blk_mq_free_request >>>
> > [    7.927826] scsi 0:0:0:0: Direct-Access     ATA      ST9500530NS      CC03 PQ: 0 ANSI: 5
> > 
> > OK, so INQUIRY response payload is looking as expected here.
> 
> Yep. It is not on the top of my head, but I remember something like INQUIRYs
> are emulated and thus do not have payload.
> 
> > [    7.927960] sd 0:0:0:0: [sda] Sector size 0 reported, assuming 512.
> > [    7.927964] sd 0:0:0:0: [sda] 1 512-byte logical blocks: (512 B/512 B)
> > [    7.927965] sd 0:0:0:0: [sda] 0-byte physical blocks
> > 
> > Strange..  READ_CAPACITY appears to be returning a payload as zeros..?
> 
> Yep. Because blk_execute_rq() does not put the proper callback and data do
> not get copied from sg's to bounce buffer. That is why I tried to use
> blk_mq_execute_rq() instead. Once I do that, data start getting read and
> booting stops elsewhere.

Mmmmmm.

The call to blk_queue_bounce() exists within blk_mq_make_request(), but
AFAICT this should still be getting invoked regardless of if the struct
request is dispatched into blk-mq via the modified blk_execute_rq() ->
blk_execute_rq_nowait() -> blk_mq_insert_request() codepath, or directly
via blk_mq_execute_rq()..

Jens..?

> 
> Of course, I was suspecting that change alone is not valid and wondered
> about the status of scsi-mq in the first place, and if more changes are
> coming.

Most certainly.  ;)

> 
> So I it turns out "req->errors + req->resid_len" issue (you described
> earlier) needs to be addressed before going forward with libata (only?).

AFAICT, getting an initial conversion of libata up does not depend upon
this specific issue being addressed first.  I could be wrong however..

> 
> > Not sure why yet some control CDBs are getting back the expected
> > payload, while others are returning zeros..
> 
> Bio buffers do not get updated from callback.
> 
> > Also, looking at the included stack back-trace:
> 
> [...]
> 
> Thanks a lot for these and other your comments, Nicholas!
> 

Sure.  I should have a few extra cycles to hack on this over the
weekend.

Also, thinking about this some more, trying to convert ahci to scsi-mq
first (using QEMU emulation), while keeping a rootfs on PIIX_IDE might
make debugging slightly easier..

--nab

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Mike Christie]

On 07/18/2013 12:51 PM, Nicholas A. Bellinger wrote:
> On Wed, 2013-07-17 at 18:19 +0200, Alexander Gordeev wrote:
>> On Tue, Jul 16, 2013 at 02:38:03PM -0700, Nicholas A. Bellinger wrote:
>>> [    7.927818] scsi_execute(): Calling blk_mq_free_request >>>
>>> [    7.927826] scsi 0:0:0:0: Direct-Access     ATA      ST9500530NS      CC03 PQ: 0 ANSI: 5
>>>
>>> OK, so INQUIRY response payload is looking as expected here.
>>
>> Yep. It is not on the top of my head, but I remember something like INQUIRYs
>> are emulated and thus do not have payload.
>>
>>> [    7.927960] sd 0:0:0:0: [sda] Sector size 0 reported, assuming 512.
>>> [    7.927964] sd 0:0:0:0: [sda] 1 512-byte logical blocks: (512 B/512 B)
>>> [    7.927965] sd 0:0:0:0: [sda] 0-byte physical blocks
>>>
>>> Strange..  READ_CAPACITY appears to be returning a payload as zeros..?
>>
>> Yep. Because blk_execute_rq() does not put the proper callback and data do
>> not get copied from sg's to bounce buffer. That is why I tried to use
>> blk_mq_execute_rq() instead. Once I do that, data start getting read and
>> booting stops elsewhere.
> 
> Mmmmmm.
> 
> The call to blk_queue_bounce() exists within blk_mq_make_request(), but
> AFAICT this should still be getting invoked regardless of if the struct
> request is dispatched into blk-mq via the modified blk_execute_rq() ->
> blk_execute_rq_nowait() -> blk_mq_insert_request() codepath, or directly
> via blk_mq_execute_rq()..
> 

blk_mq_make_request is not called from the blk insert/execute paths.
blk_mq_make_request takes a bio and tries to merge it with a request and
adds it to the queue. It is only called when the make_request_fn is
called like when generic_make_request is called.

blk_mq_insert_request adds a already formed request to the queue. It is
already formed so that is why that path does not bounce bios. The
bios/pages should already be added within the drivers restrictions. So
for the read_cap path, the call to blk_rq_map_kern in scsi_execute does
the blk_queue_bounce call.

Just saw this while trying out iscsi with the scsi-mq stuff :)

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Nicholas A. Bellinger]

On Thu, 2013-07-18 at 11:51 -0700, Nicholas A. Bellinger wrote:
> On Wed, 2013-07-17 at 18:19 +0200, Alexander Gordeev wrote:
> > On Tue, Jul 16, 2013 at 02:38:03PM -0700, Nicholas A. Bellinger wrote:
> > > [    7.927818] scsi_execute(): Calling blk_mq_free_request >>>
> > > [    7.927826] scsi 0:0:0:0: Direct-Access     ATA      ST9500530NS      CC03 PQ: 0 ANSI: 5
> > > 
> > > OK, so INQUIRY response payload is looking as expected here.
> > 
> > Yep. It is not on the top of my head, but I remember something like INQUIRYs
> > are emulated and thus do not have payload.
> > 
> > > [    7.927960] sd 0:0:0:0: [sda] Sector size 0 reported, assuming 512.
> > > [    7.927964] sd 0:0:0:0: [sda] 1 512-byte logical blocks: (512 B/512 B)
> > > [    7.927965] sd 0:0:0:0: [sda] 0-byte physical blocks
> > > 
> > > Strange..  READ_CAPACITY appears to be returning a payload as zeros..?
> > 
> > Yep. Because blk_execute_rq() does not put the proper callback and data do
> > not get copied from sg's to bounce buffer. That is why I tried to use
> > blk_mq_execute_rq() instead. Once I do that, data start getting read and
> > booting stops elsewhere.
> 
> Mmmmmm.
> 
> The call to blk_queue_bounce() exists within blk_mq_make_request(), but
> AFAICT this should still be getting invoked regardless of if the struct
> request is dispatched into blk-mq via the modified blk_execute_rq() ->
> blk_execute_rq_nowait() -> blk_mq_insert_request() codepath, or directly
> via blk_mq_execute_rq()..
> 
> Jens..?
> 

Actually sorry, your right.  A call to blk_mq_insert_request() for
REQ_TYPE_BLOCK_PC will not invoke blk_queue_bounce() located near the
top of blk_mq_execute_rq(), which means that only REQ_TYPE_FS is
currently using bounce buffers, if required.

Need to think a bit more about what to do here for REQ_TYPE_BLOCK_PC
bounce buffer special case with blk_execute_rq(), but I'm thinking that
blk_mq_execute_rq() should really not be used here..

Jens..?

--nab

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Jens Axboe]

On 07/18/2013 01:14 PM, Nicholas A. Bellinger wrote:
> On Thu, 2013-07-18 at 11:51 -0700, Nicholas A. Bellinger wrote:
>> On Wed, 2013-07-17 at 18:19 +0200, Alexander Gordeev wrote:
>>> On Tue, Jul 16, 2013 at 02:38:03PM -0700, Nicholas A. Bellinger wrote:
>>>> [    7.927818] scsi_execute(): Calling blk_mq_free_request >>>
>>>> [    7.927826] scsi 0:0:0:0: Direct-Access     ATA      ST9500530NS      CC03 PQ: 0 ANSI: 5
>>>>
>>>> OK, so INQUIRY response payload is looking as expected here.
>>>
>>> Yep. It is not on the top of my head, but I remember something like INQUIRYs
>>> are emulated and thus do not have payload.
>>>
>>>> [    7.927960] sd 0:0:0:0: [sda] Sector size 0 reported, assuming 512.
>>>> [    7.927964] sd 0:0:0:0: [sda] 1 512-byte logical blocks: (512 B/512 B)
>>>> [    7.927965] sd 0:0:0:0: [sda] 0-byte physical blocks
>>>>
>>>> Strange..  READ_CAPACITY appears to be returning a payload as zeros..?
>>>
>>> Yep. Because blk_execute_rq() does not put the proper callback and data do
>>> not get copied from sg's to bounce buffer. That is why I tried to use
>>> blk_mq_execute_rq() instead. Once I do that, data start getting read and
>>> booting stops elsewhere.
>>
>> Mmmmmm.
>>
>> The call to blk_queue_bounce() exists within blk_mq_make_request(), but
>> AFAICT this should still be getting invoked regardless of if the struct
>> request is dispatched into blk-mq via the modified blk_execute_rq() ->
>> blk_execute_rq_nowait() -> blk_mq_insert_request() codepath, or directly
>> via blk_mq_execute_rq()..
>>
>> Jens..?
>>
> 
> Actually sorry, your right.  A call to blk_mq_insert_request() for
> REQ_TYPE_BLOCK_PC will not invoke blk_queue_bounce() located near the
> top of blk_mq_execute_rq(), which means that only REQ_TYPE_FS is
> currently using bounce buffers, if required.
> 
> Need to think a bit more about what to do here for REQ_TYPE_BLOCK_PC
> bounce buffer special case with blk_execute_rq(), but I'm thinking that
> blk_mq_execute_rq() should really not be used here..
> 
> Jens..?

It needs to be pre-bounced, blk-mq will only bounce incoming bios and
not requests merely added to the queue(s). Might be useful to add an
equiv blk_mq_make_request() for this.

-- 
Jens Axboe

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Nicholas A. Bellinger]

On Thu, 2013-07-18 at 13:12 -0600, Mike Christie wrote:
> On 07/18/2013 12:51 PM, Nicholas A. Bellinger wrote:
> > On Wed, 2013-07-17 at 18:19 +0200, Alexander Gordeev wrote:
> >> On Tue, Jul 16, 2013 at 02:38:03PM -0700, Nicholas A. Bellinger wrote:
> >>> [    7.927818] scsi_execute(): Calling blk_mq_free_request >>>
> >>> [    7.927826] scsi 0:0:0:0: Direct-Access     ATA      ST9500530NS      CC03 PQ: 0 ANSI: 5
> >>>
> >>> OK, so INQUIRY response payload is looking as expected here.
> >>
> >> Yep. It is not on the top of my head, but I remember something like INQUIRYs
> >> are emulated and thus do not have payload.
> >>
> >>> [    7.927960] sd 0:0:0:0: [sda] Sector size 0 reported, assuming 512.
> >>> [    7.927964] sd 0:0:0:0: [sda] 1 512-byte logical blocks: (512 B/512 B)
> >>> [    7.927965] sd 0:0:0:0: [sda] 0-byte physical blocks
> >>>
> >>> Strange..  READ_CAPACITY appears to be returning a payload as zeros..?
> >>
> >> Yep. Because blk_execute_rq() does not put the proper callback and data do
> >> not get copied from sg's to bounce buffer. That is why I tried to use
> >> blk_mq_execute_rq() instead. Once I do that, data start getting read and
> >> booting stops elsewhere.
> > 
> > Mmmmmm.
> > 
> > The call to blk_queue_bounce() exists within blk_mq_make_request(), but
> > AFAICT this should still be getting invoked regardless of if the struct
> > request is dispatched into blk-mq via the modified blk_execute_rq() ->
> > blk_execute_rq_nowait() -> blk_mq_insert_request() codepath, or directly
> > via blk_mq_execute_rq()..
> > 
> 
> blk_mq_make_request is not called from the blk insert/execute paths.
> blk_mq_make_request takes a bio and tries to merge it with a request and
> adds it to the queue. It is only called when the make_request_fn is
> called like when generic_make_request is called.
> 
> blk_mq_insert_request adds a already formed request to the queue. It is
> already formed so that is why that path does not bounce bios. The
> bios/pages should already be added within the drivers restrictions. So
> for the read_cap path, the call to blk_rq_map_kern in scsi_execute does
> the blk_queue_bounce call.
> 

<nod>, just noticed the blk_queue_bounce() in blk_rq_map_kern().  

Not sure why this doesn't seem to be doing what it's supposed to for
libata just yet..

> Just saw this while trying out iscsi with the scsi-mq stuff :)
> 

Took at stab at this a while back, but ended getting distracted on other
items.  Do you have an initial conversion running yet..?

--nab

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Jens Axboe]

On Thu, Jul 18 2013, Nicholas A. Bellinger wrote:
> On Thu, 2013-07-18 at 13:12 -0600, Mike Christie wrote:
> > On 07/18/2013 12:51 PM, Nicholas A. Bellinger wrote:
> > > On Wed, 2013-07-17 at 18:19 +0200, Alexander Gordeev wrote:
> > >> On Tue, Jul 16, 2013 at 02:38:03PM -0700, Nicholas A. Bellinger wrote:
> > >>> [    7.927818] scsi_execute(): Calling blk_mq_free_request >>>
> > >>> [    7.927826] scsi 0:0:0:0: Direct-Access     ATA      ST9500530NS      CC03 PQ: 0 ANSI: 5
> > >>>
> > >>> OK, so INQUIRY response payload is looking as expected here.
> > >>
> > >> Yep. It is not on the top of my head, but I remember something like INQUIRYs
> > >> are emulated and thus do not have payload.
> > >>
> > >>> [    7.927960] sd 0:0:0:0: [sda] Sector size 0 reported, assuming 512.
> > >>> [    7.927964] sd 0:0:0:0: [sda] 1 512-byte logical blocks: (512 B/512 B)
> > >>> [    7.927965] sd 0:0:0:0: [sda] 0-byte physical blocks
> > >>>
> > >>> Strange..  READ_CAPACITY appears to be returning a payload as zeros..?
> > >>
> > >> Yep. Because blk_execute_rq() does not put the proper callback and data do
> > >> not get copied from sg's to bounce buffer. That is why I tried to use
> > >> blk_mq_execute_rq() instead. Once I do that, data start getting read and
> > >> booting stops elsewhere.
> > > 
> > > Mmmmmm.
> > > 
> > > The call to blk_queue_bounce() exists within blk_mq_make_request(), but
> > > AFAICT this should still be getting invoked regardless of if the struct
> > > request is dispatched into blk-mq via the modified blk_execute_rq() ->
> > > blk_execute_rq_nowait() -> blk_mq_insert_request() codepath, or directly
> > > via blk_mq_execute_rq()..
> > > 
> > 
> > blk_mq_make_request is not called from the blk insert/execute paths.
> > blk_mq_make_request takes a bio and tries to merge it with a request and
> > adds it to the queue. It is only called when the make_request_fn is
> > called like when generic_make_request is called.
> > 
> > blk_mq_insert_request adds a already formed request to the queue. It is
> > already formed so that is why that path does not bounce bios. The
> > bios/pages should already be added within the drivers restrictions. So
> > for the read_cap path, the call to blk_rq_map_kern in scsi_execute does
> > the blk_queue_bounce call.
> > 
> 
> <nod>, just noticed the blk_queue_bounce() in blk_rq_map_kern().  
> 
> Not sure why this doesn't seem to be doing what it's supposed to for
> libata just yet..

How are you make the request from the bio? It'd be pretty trivial to
ensure that it gets bounced properly... blk_mq_execute_rq() assumes a
fully complete request, so it wont bounce anything.

-- 
Jens Axboe

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Nicholas A. Bellinger]

On Thu, 2013-07-18 at 18:30 -0600, Jens Axboe wrote:
> On Thu, Jul 18 2013, Nicholas A. Bellinger wrote:
> > On Thu, 2013-07-18 at 13:12 -0600, Mike Christie wrote:
> > > On 07/18/2013 12:51 PM, Nicholas A. Bellinger wrote:
> > > > On Wed, 2013-07-17 at 18:19 +0200, Alexander Gordeev wrote:
> > > >> On Tue, Jul 16, 2013 at 02:38:03PM -0700, Nicholas A. Bellinger wrote:
> > > >>> [    7.927818] scsi_execute(): Calling blk_mq_free_request >>>
> > > >>> [    7.927826] scsi 0:0:0:0: Direct-Access     ATA      ST9500530NS      CC03 PQ: 0 ANSI: 5
> > > >>>
> > > >>> OK, so INQUIRY response payload is looking as expected here.
> > > >>
> > > >> Yep. It is not on the top of my head, but I remember something like INQUIRYs
> > > >> are emulated and thus do not have payload.
> > > >>
> > > >>> [    7.927960] sd 0:0:0:0: [sda] Sector size 0 reported, assuming 512.
> > > >>> [    7.927964] sd 0:0:0:0: [sda] 1 512-byte logical blocks: (512 B/512 B)
> > > >>> [    7.927965] sd 0:0:0:0: [sda] 0-byte physical blocks
> > > >>>
> > > >>> Strange..  READ_CAPACITY appears to be returning a payload as zeros..?
> > > >>
> > > >> Yep. Because blk_execute_rq() does not put the proper callback and data do
> > > >> not get copied from sg's to bounce buffer. That is why I tried to use
> > > >> blk_mq_execute_rq() instead. Once I do that, data start getting read and
> > > >> booting stops elsewhere.
> > > > 
> > > > Mmmmmm.
> > > > 
> > > > The call to blk_queue_bounce() exists within blk_mq_make_request(), but
> > > > AFAICT this should still be getting invoked regardless of if the struct
> > > > request is dispatched into blk-mq via the modified blk_execute_rq() ->
> > > > blk_execute_rq_nowait() -> blk_mq_insert_request() codepath, or directly
> > > > via blk_mq_execute_rq()..
> > > > 
> > > 
> > > blk_mq_make_request is not called from the blk insert/execute paths.
> > > blk_mq_make_request takes a bio and tries to merge it with a request and
> > > adds it to the queue. It is only called when the make_request_fn is
> > > called like when generic_make_request is called.
> > > 
> > > blk_mq_insert_request adds a already formed request to the queue. It is
> > > already formed so that is why that path does not bounce bios. The
> > > bios/pages should already be added within the drivers restrictions. So
> > > for the read_cap path, the call to blk_rq_map_kern in scsi_execute does
> > > the blk_queue_bounce call.
> > > 
> > 
> > <nod>, just noticed the blk_queue_bounce() in blk_rq_map_kern().  
> > 
> > Not sure why this doesn't seem to be doing what it's supposed to for
> > libata just yet..
> 
> How are you make the request from the bio? It'd be pretty trivial to
> ensure that it gets bounced properly... blk_mq_execute_rq() assumes a
> fully complete request, so it wont bounce anything.
> 

>From what I gather for REQ_TYPE_BLOCK_PC, scsi_execute() ->
blk_rq_map_kern() -> blk_rq_append_bio() -> blk_rq_bio_prep() is what
does the request setup from the bios returned by bio_[copy,map]_kern()
in blk_rq_map_kern() code.

blk_queue_bounce() is called immediately after blk_rq_append_bio() here,
which AFAICT looks like it's doing the correct thing for scsi-mq..

What is strange here is that libata-scsi.c CDB emulation code is doing
the same stuff for both INQUIRY (that seems to be OK) and READ_CAPACITY
(that is returning zeros), which makes me think that something else is
going on..

Alexander, where you able to re-test using sdev->sdev_mq_reg.queue_depth
= 1 in scsi_mq_alloc_queue()..?

--nab

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Nicholas A. Bellinger]

On Thu, 2013-07-18 at 18:03 -0700, Nicholas A. Bellinger wrote:
> On Thu, 2013-07-18 at 18:30 -0600, Jens Axboe wrote:
> > On Thu, Jul 18 2013, Nicholas A. Bellinger wrote:
> > > On Thu, 2013-07-18 at 13:12 -0600, Mike Christie wrote:
> > > > On 07/18/2013 12:51 PM, Nicholas A. Bellinger wrote:

<SNIP>

> > > <nod>, just noticed the blk_queue_bounce() in blk_rq_map_kern().  
> > > 
> > > Not sure why this doesn't seem to be doing what it's supposed to for
> > > libata just yet..
> > 
> > How are you make the request from the bio? It'd be pretty trivial to
> > ensure that it gets bounced properly... blk_mq_execute_rq() assumes a
> > fully complete request, so it wont bounce anything.
> > 
> 
> From what I gather for REQ_TYPE_BLOCK_PC, scsi_execute() ->
> blk_rq_map_kern() -> blk_rq_append_bio() -> blk_rq_bio_prep() is what
> does the request setup from the bios returned by bio_[copy,map]_kern()
> in blk_rq_map_kern() code.
> 
> blk_queue_bounce() is called immediately after blk_rq_append_bio() here,
> which AFAICT looks like it's doing the correct thing for scsi-mq..
> 
> What is strange here is that libata-scsi.c CDB emulation code is doing
> the same stuff for both INQUIRY (that seems to be OK) and READ_CAPACITY
> (that is returning zeros), which makes me think that something else is
> going on..
> 
> Alexander, where you able to re-test using sdev->sdev_mq_reg.queue_depth
> = 1 in scsi_mq_alloc_queue()..?
> 

So after a bit more hacking tonight it appears the explicit setting of
dma_alignment by libata (sdev->sector_size - 1) is what is making
blk_rq_map_kern() invoke blk_copy_kern() instead of blk_map_kern(), and
triggering this scsi-mq specific bug with libata.  I'm able to confirm
with QEMU IDE emulation the bug is occurring only after INQUIRY and
before READ_CAPACITY, as reported by Alexander.

Below is a quick hack to skip this setting in ata_scsi_dev_config() for
blk-mq, and leaves the default dma_alignment=0x03 for REQ_TYPE_BLOCK_PC
requests as initially set by scsi-core in scsi_init_request_queue().

Also included is the change for using queue_depth = min(SHT->can_queue,
SHT->cmd_per_lun) during scsi-mq request_queue initialization, along
with a very basic ata_piix conversion for testing purposes.

With these three changes in place, I'm able to register a single 1GB
ata_piix LUN using QEMU IDE emulation, and successfully run simple fio
writeverify tests with blocksize=4k @ queue_depth=1.

Obviously this is not a proper bugfix for unaligned blk_copy_kern() with
scsi-mq + REQ_TYPE_BLOCK_PC case, but should be enough to at least get
libata LUN scanning to work.  Need to take a deeper look at the problem
here..

Alexander, care to give this work-around a shot on your bare-metal setup
using ata_piix..?

--nab

diff --git a/drivers/ata/ata_piix.c b/drivers/ata/ata_piix.c
index 9a8a674..ac05cd6 100644
--- a/drivers/ata/ata_piix.c
+++ b/drivers/ata/ata_piix.c
@@ -1066,6 +1066,8 @@ static u8 piix_vmw_bmdma_status(struct ata_port *ap)
 
 static struct scsi_host_template piix_sht = {
        ATA_BMDMA_SHT(DRV_NAME),
+       .scsi_mq        = true,
+       .queuecommand_mq = ata_scsi_queuecmd,
 };
 
 static struct ata_port_operations piix_sata_ops = {
diff --git a/drivers/ata/libata-scsi.c b/drivers/ata/libata-scsi.c
index 0101af5..191bc15 100644
--- a/drivers/ata/libata-scsi.c
+++ b/drivers/ata/libata-scsi.c
@@ -1144,7 +1144,11 @@ static int ata_scsi_dev_config(struct scsi_device *sdev,
                        "sector_size=%u > PAGE_SIZE, PIO may malfunction\n",
                        sdev->sector_size);
 
-       blk_queue_update_dma_alignment(q, sdev->sector_size - 1);
+       if (!q->mq_ops) {
+               blk_queue_update_dma_alignment(q, sdev->sector_size - 1);
+       } else {
+               printk("Skipping dma_alignment for libata w/ scsi-mq\n");
+       }
 
        if (dev->flags & ATA_DFLAG_AN)
                set_bit(SDEV_EVT_MEDIA_CHANGE, sdev->supported_events);
diff --git a/drivers/scsi/scsi-mq.c b/drivers/scsi/scsi-mq.c
index ca6ff67..81b2633 100644
--- a/drivers/scsi/scsi-mq.c
+++ b/drivers/scsi/scsi-mq.c
@@ -199,11 +199,11 @@ int scsi_mq_alloc_queue(struct Scsi_Host *sh, struct scsi_device *sdev)
        int i, j;
 
        sdev->sdev_mq_reg.ops = &scsi_mq_ops;
-       sdev->sdev_mq_reg.queue_depth = sdev->queue_depth;
+       sdev->sdev_mq_reg.queue_depth = min((short)sh->hostt->can_queue,
+                                           sh->hostt->cmd_per_lun);
        sdev->sdev_mq_reg.cmd_size = sizeof(struct scsi_cmnd) + sh->hostt->cmd_size;
        sdev->sdev_mq_reg.numa_node = NUMA_NO_NODE;
        sdev->sdev_mq_reg.nr_hw_queues = 1;
-       sdev->sdev_mq_reg.queue_depth = 64;
        sdev->sdev_mq_reg.flags = BLK_MQ_F_SHOULD_MERGE;
 
        printk("Calling blk_mq_init_queue: scsi_mq_ops: %p, queue_depth: %d,"

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Nicholas A. Bellinger]

On Thu, 2013-07-18 at 18:03 -0700, Nicholas A. Bellinger wrote:
> On Thu, 2013-07-18 at 18:30 -0600, Jens Axboe wrote:
> > On Thu, Jul 18 2013, Nicholas A. Bellinger wrote:
> > > On Thu, 2013-07-18 at 13:12 -0600, Mike Christie wrote:
> > > > On 07/18/2013 12:51 PM, Nicholas A. Bellinger wrote:

<SNIP>

> > > <nod>, just noticed the blk_queue_bounce() in blk_rq_map_kern().  
> > > 
> > > Not sure why this doesn't seem to be doing what it's supposed to for
> > > libata just yet..
> > 
> > How are you make the request from the bio? It'd be pretty trivial to
> > ensure that it gets bounced properly... blk_mq_execute_rq() assumes a
> > fully complete request, so it wont bounce anything.
> > 
> 
> From what I gather for REQ_TYPE_BLOCK_PC, scsi_execute() ->
> blk_rq_map_kern() -> blk_rq_append_bio() -> blk_rq_bio_prep() is what
> does the request setup from the bios returned by bio_[copy,map]_kern()
> in blk_rq_map_kern() code.
> 
> blk_queue_bounce() is called immediately after blk_rq_append_bio() here,
> which AFAICT looks like it's doing the correct thing for scsi-mq..
> 
> What is strange here is that libata-scsi.c CDB emulation code is doing
> the same stuff for both INQUIRY (that seems to be OK) and READ_CAPACITY
> (that is returning zeros), which makes me think that something else is
> going on..
> 
> Alexander, where you able to re-test using sdev->sdev_mq_reg.queue_depth
> = 1 in scsi_mq_alloc_queue()..?
> 

So after a bit more hacking tonight it appears the explicit setting of
dma_alignment by libata (sdev->sector_size - 1) is what is making
blk_rq_map_kern() invoke blk_copy_kern() instead of blk_map_kern(), and
triggering this scsi-mq specific bug with libata.  I'm able to confirm
with QEMU IDE emulation the bug is occurring only after INQUIRY and
before READ_CAPACITY, as reported by Alexander.

Below is a quick hack to skip this setting in ata_scsi_dev_config() for
blk-mq, and leaves the default dma_alignment=0x03 for REQ_TYPE_BLOCK_PC
requests as initially set by scsi-core in scsi_init_request_queue().

Also included is the change for using queue_depth = min(SHT->can_queue,
SHT->cmd_per_lun) during scsi-mq request_queue initialization, along
with a very basic ata_piix conversion for testing purposes.

With these three changes in place, I'm able to register a single 1GB
ata_piix LUN using QEMU IDE emulation, and successfully run simple fio
writeverify tests with blocksize=4k @ queue_depth=1.

Obviously this is not a proper bugfix for unaligned blk_copy_kern() with
scsi-mq + REQ_TYPE_BLOCK_PC case, but should be enough to at least get
libata LUN scanning to work.  Need to take a deeper look at the problem
here..

Alexander, care to give this work-around a shot on your bare-metal setup
using ata_piix..?

--nab

diff --git a/drivers/ata/ata_piix.c b/drivers/ata/ata_piix.c
index 9a8a674..ac05cd6 100644
--- a/drivers/ata/ata_piix.c
+++ b/drivers/ata/ata_piix.c
@@ -1066,6 +1066,8 @@ static u8 piix_vmw_bmdma_status(struct ata_port *ap)
 
 static struct scsi_host_template piix_sht = {
        ATA_BMDMA_SHT(DRV_NAME),
+       .scsi_mq        = true,
+       .queuecommand_mq = ata_scsi_queuecmd,
 };
 
 static struct ata_port_operations piix_sata_ops = {
diff --git a/drivers/ata/libata-scsi.c b/drivers/ata/libata-scsi.c
index 0101af5..191bc15 100644
--- a/drivers/ata/libata-scsi.c
+++ b/drivers/ata/libata-scsi.c
@@ -1144,7 +1144,11 @@ static int ata_scsi_dev_config(struct scsi_device *sdev,
                        "sector_size=%u > PAGE_SIZE, PIO may malfunction\n",
                        sdev->sector_size);
 
-       blk_queue_update_dma_alignment(q, sdev->sector_size - 1);
+       if (!q->mq_ops) {
+               blk_queue_update_dma_alignment(q, sdev->sector_size - 1);
+       } else {
+               printk("Skipping dma_alignment for libata w/ scsi-mq\n");
+       }
 
        if (dev->flags & ATA_DFLAG_AN)
                set_bit(SDEV_EVT_MEDIA_CHANGE, sdev->supported_events);
diff --git a/drivers/scsi/scsi-mq.c b/drivers/scsi/scsi-mq.c
index ca6ff67..81b2633 100644
--- a/drivers/scsi/scsi-mq.c
+++ b/drivers/scsi/scsi-mq.c
@@ -199,11 +199,11 @@ int scsi_mq_alloc_queue(struct Scsi_Host *sh, struct scsi_device *sdev)
        int i, j;
 
        sdev->sdev_mq_reg.ops = &scsi_mq_ops;
-       sdev->sdev_mq_reg.queue_depth = sdev->queue_depth;
+       sdev->sdev_mq_reg.queue_depth = min((short)sh->hostt->can_queue,
+                                           sh->hostt->cmd_per_lun);
        sdev->sdev_mq_reg.cmd_size = sizeof(struct scsi_cmnd) + sh->hostt->cmd_size;
        sdev->sdev_mq_reg.numa_node = NUMA_NO_NODE;
        sdev->sdev_mq_reg.nr_hw_queues = 1;
-       sdev->sdev_mq_reg.queue_depth = 64;
        sdev->sdev_mq_reg.flags = BLK_MQ_F_SHOULD_MERGE;
 
        printk("Calling blk_mq_init_queue: scsi_mq_ops: %p, queue_depth: %d,"

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[James Bottomley]

On Thu, 2013-07-18 at 23:34 -0700, Nicholas A. Bellinger wrote:
> diff --git a/drivers/ata/libata-scsi.c b/drivers/ata/libata-scsi.c
> index 0101af5..191bc15 100644
> --- a/drivers/ata/libata-scsi.c
> +++ b/drivers/ata/libata-scsi.c
> @@ -1144,7 +1144,11 @@ static int ata_scsi_dev_config(struct scsi_device *sdev,
>                         "sector_size=%u > PAGE_SIZE, PIO may malfunction\n",
>                         sdev->sector_size);
>  
> -       blk_queue_update_dma_alignment(q, sdev->sector_size - 1);
> +       if (!q->mq_ops) {
> +               blk_queue_update_dma_alignment(q, sdev->sector_size - 1);
> +       } else {
> +               printk("Skipping dma_alignment for libata w/ scsi-mq\n");
> +       }

Amazingly enough there is a reason for the dma alignment, and it wasn't
just to annoy you, so you can't blindly do this.

The email thread is probably lost in the mists of time, but if I
remember correctly the problem is that some ahci DMA controllers barf if
the sector they're doing DMA on crosses a page boundary.  Some are
annoying enough to actually cause silent data corruption.  You won't
find every ahci DMA controller doing this, so the change will work for
some, but it will be hard to identify those it won't work for until
people start losing data.

The correct fix, obviously, is to do the bio copy on the kernel path for
unaligned data.  It is OK to assume that REQ_TYPE_FS data is correctly
aligned (because of the block to page alignment).

James

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Mike Christie]

On 07/18/2013 06:23 PM, Nicholas A. Bellinger wrote:
>> Just saw this while trying out iscsi with the scsi-mq stuff :)
>> > 
> Took at stab at this a while back, but ended getting distracted on other
> items.  Do you have an initial conversion running yet..?

Not running well :) Have a patch but I am debugging it now. I messed up
something with the cmd_size stuff.

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Nicholas A. Bellinger]

On Fri, 2013-07-19 at 08:33 -0700, James Bottomley wrote:
> On Thu, 2013-07-18 at 23:34 -0700, Nicholas A. Bellinger wrote:
> > diff --git a/drivers/ata/libata-scsi.c b/drivers/ata/libata-scsi.c
> > index 0101af5..191bc15 100644
> > --- a/drivers/ata/libata-scsi.c
> > +++ b/drivers/ata/libata-scsi.c
> > @@ -1144,7 +1144,11 @@ static int ata_scsi_dev_config(struct scsi_device *sdev,
> >                         "sector_size=%u > PAGE_SIZE, PIO may malfunction\n",
> >                         sdev->sector_size);
> >  
> > -       blk_queue_update_dma_alignment(q, sdev->sector_size - 1);
> > +       if (!q->mq_ops) {
> > +               blk_queue_update_dma_alignment(q, sdev->sector_size - 1);
> > +       } else {
> > +               printk("Skipping dma_alignment for libata w/ scsi-mq\n");
> > +       }
> 
> Amazingly enough there is a reason for the dma alignment, and it wasn't
> just to annoy you, so you can't blindly do this.
> 
> The email thread is probably lost in the mists of time, but if I
> remember correctly the problem is that some ahci DMA controllers barf if
> the sector they're doing DMA on crosses a page boundary.  Some are
> annoying enough to actually cause silent data corruption.  You won't
> find every ahci DMA controller doing this, so the change will work for
> some, but it will be hard to identify those it won't work for until
> people start losing data.

Thanks for the extra background.

So at least from what I gather thus far this shouldn't be an issue for
initial testing with scsi-mq <-> libata w/ ata_piix.

> 
> The correct fix, obviously, is to do the bio copy on the kernel path for
> unaligned data.  It is OK to assume that REQ_TYPE_FS data is correctly
> aligned (because of the block to page alignment).
> 

Indeed.  Looking into the bio_copy_kern() breakage next..

--nab

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Nicholas A. Bellinger]

On Fri, 2013-07-19 at 09:58 -0600, Mike Christie wrote:
> On 07/18/2013 06:23 PM, Nicholas A. Bellinger wrote:
> >> Just saw this while trying out iscsi with the scsi-mq stuff :)
> >> > 
> > Took at stab at this a while back, but ended getting distracted on other
> > items.  Do you have an initial conversion running yet..?
> 
> Not running well :) Have a patch but I am debugging it now. I messed up
> something with the cmd_size stuff.

<nod>, looking forward to see ib_iser move beyond the existing
scsi_request_fn() bottleneck.   ;)

Feel free to send me the WIP off-list if you'd like another pair of
eyes.

--nab

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Nicholas A. Bellinger]

On Fri, 2013-07-19 at 14:01 -0700, Nicholas A. Bellinger wrote:
> On Fri, 2013-07-19 at 08:33 -0700, James Bottomley wrote:
> > On Thu, 2013-07-18 at 23:34 -0700, Nicholas A. Bellinger wrote:
> > > diff --git a/drivers/ata/libata-scsi.c b/drivers/ata/libata-scsi.c
> > > index 0101af5..191bc15 100644
> > > --- a/drivers/ata/libata-scsi.c
> > > +++ b/drivers/ata/libata-scsi.c
> > > @@ -1144,7 +1144,11 @@ static int ata_scsi_dev_config(struct scsi_device *sdev,
> > >                         "sector_size=%u > PAGE_SIZE, PIO may malfunction\n",
> > >                         sdev->sector_size);
> > >  
> > > -       blk_queue_update_dma_alignment(q, sdev->sector_size - 1);
> > > +       if (!q->mq_ops) {
> > > +               blk_queue_update_dma_alignment(q, sdev->sector_size - 1);
> > > +       } else {
> > > +               printk("Skipping dma_alignment for libata w/ scsi-mq\n");
> > > +       }
> > 
> > Amazingly enough there is a reason for the dma alignment, and it wasn't
> > just to annoy you, so you can't blindly do this.
> > 
> > The email thread is probably lost in the mists of time, but if I
> > remember correctly the problem is that some ahci DMA controllers barf if
> > the sector they're doing DMA on crosses a page boundary.  Some are
> > annoying enough to actually cause silent data corruption.  You won't
> > find every ahci DMA controller doing this, so the change will work for
> > some, but it will be hard to identify those it won't work for until
> > people start losing data.
> 
> Thanks for the extra background.
> 
> So at least from what I gather thus far this shouldn't be an issue for
> initial testing with scsi-mq <-> libata w/ ata_piix.
> 
> > 
> > The correct fix, obviously, is to do the bio copy on the kernel path for
> > unaligned data.  It is OK to assume that REQ_TYPE_FS data is correctly
> > aligned (because of the block to page alignment).
> > 
> 
> Indeed.  Looking into the bio_copy_kern() breakage next..
> 

OK, after further investigation the root cause is a actually a missing
bio->bio_end_io() -> bio_copy_kern_endio() -> bio_put() from the
blk_end_sync_rq() callback path that scsi-mq REQ_TYPE_BLOCK_PC is
currently using.

Including the following patch into the scsi-mq working branch now, and
reverting the libata dma_alignment=0x03 hack.

Alexander, care to give this a try..?

--nab

diff --git a/block/blk-exec.c b/block/blk-exec.c
index 0761c89..70303d2 100644
--- a/block/blk-exec.c
+++ b/block/blk-exec.c
@@ -25,7 +25,10 @@ static void blk_end_sync_rq(struct request *rq, int error)
        struct completion *waiting = rq->end_io_data;
 
        rq->end_io_data = NULL;
-       if (!rq->q->mq_ops) {
+       if (rq->q->mq_ops) {
+               if (rq->bio)
+                       bio_endio(rq->bio, error);
+       } else {
                __blk_put_request(rq->q, rq);
        }

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Mike Christie]

[-- Attachment #1: Type: text/plain, Size: 4062 bytes --]

On 07/19/2013 11:56 PM, Nicholas A. Bellinger wrote:
> On Fri, 2013-07-19 at 14:01 -0700, Nicholas A. Bellinger wrote:
>> On Fri, 2013-07-19 at 08:33 -0700, James Bottomley wrote:
>>> On Thu, 2013-07-18 at 23:34 -0700, Nicholas A. Bellinger wrote:
>>>> diff --git a/drivers/ata/libata-scsi.c b/drivers/ata/libata-scsi.c
>>>> index 0101af5..191bc15 100644
>>>> --- a/drivers/ata/libata-scsi.c
>>>> +++ b/drivers/ata/libata-scsi.c
>>>> @@ -1144,7 +1144,11 @@ static int ata_scsi_dev_config(struct scsi_device *sdev,
>>>>                         "sector_size=%u > PAGE_SIZE, PIO may malfunction\n",
>>>>                         sdev->sector_size);
>>>>  
>>>> -       blk_queue_update_dma_alignment(q, sdev->sector_size - 1);
>>>> +       if (!q->mq_ops) {
>>>> +               blk_queue_update_dma_alignment(q, sdev->sector_size - 1);
>>>> +       } else {
>>>> +               printk("Skipping dma_alignment for libata w/ scsi-mq\n");
>>>> +       }
>>>
>>> Amazingly enough there is a reason for the dma alignment, and it wasn't
>>> just to annoy you, so you can't blindly do this.
>>>
>>> The email thread is probably lost in the mists of time, but if I
>>> remember correctly the problem is that some ahci DMA controllers barf if
>>> the sector they're doing DMA on crosses a page boundary.  Some are
>>> annoying enough to actually cause silent data corruption.  You won't
>>> find every ahci DMA controller doing this, so the change will work for
>>> some, but it will be hard to identify those it won't work for until
>>> people start losing data.
>>
>> Thanks for the extra background.
>>
>> So at least from what I gather thus far this shouldn't be an issue for
>> initial testing with scsi-mq <-> libata w/ ata_piix.
>>
>>>
>>> The correct fix, obviously, is to do the bio copy on the kernel path for
>>> unaligned data.  It is OK to assume that REQ_TYPE_FS data is correctly
>>> aligned (because of the block to page alignment).
>>>
>>
>> Indeed.  Looking into the bio_copy_kern() breakage next..
>>
> 
> OK, after further investigation the root cause is a actually a missing
> bio->bio_end_io() -> bio_copy_kern_endio() -> bio_put() from the
> blk_end_sync_rq() callback path that scsi-mq REQ_TYPE_BLOCK_PC is
> currently using.
> 
> Including the following patch into the scsi-mq working branch now, and
> reverting the libata dma_alignment=0x03 hack.
> 
> Alexander, care to give this a try..?
> 
> --nab
> 
> diff --git a/block/blk-exec.c b/block/blk-exec.c
> index 0761c89..70303d2 100644
> --- a/block/blk-exec.c
> +++ b/block/blk-exec.c
> @@ -25,7 +25,10 @@ static void blk_end_sync_rq(struct request *rq, int error)
>         struct completion *waiting = rq->end_io_data;
>  
>         rq->end_io_data = NULL;
> -       if (!rq->q->mq_ops) {
> +       if (rq->q->mq_ops) {
> +               if (rq->bio)
> +                       bio_endio(rq->bio, error);
> +       } else {
>                 __blk_put_request(rq->q, rq);
>         }
> 


This does not handle requests with multiple bios, and for the mq stye
passthrough insertion completions you actually want to call
blk_mq_finish_request in scsi_execute. Same for all the other
passthrough code in your scsi mq tree. That is your root bug. Instead of
doing that though I think we want to have the block layer free the bios
like before.

For the non mq calls, blk_end_request type of calls will complete the
bios when blk_finish_request is called from that path. It will then call
the rq end_io callback.

I think the blk mq code assumes if the end_io callack is set that the
end_io function will do the bio cleanup. See __blk_mq_end_io. Also see
how blk_mq_execute_rq calls blk_mq_finish_request for an example of how
rq passthrough execution and cleanup is being done in the mq paths.

Now with the scsi mq changes, when blk_execute_rq_nowait calls
blk_mq_insert_request it calls it with a old non mq style of end io
function that does not complete the bios.

What about the attached only compile tested patch. The patch has the mq
block code work like the non mq code for bio cleanups.



[-- Attachment #2: blk-mq-free-bio.patch --]
[-- Type: text/plain, Size: 2513 bytes --]

blk-mq: blk-mq should free bios in pass through case

For non mq calls, the block layer will free the bios when
blk_finish_request is called.

For mq calls, the blk mq code wants the caller to do this.

This patch has the blk mq code work like the non mq code
and has the block layer free the bios.

Signed-off-by: Mike Christie <michaelc@cs.wisc.edu>

diff --git a/block/blk-flush.c b/block/blk-flush.c
index c56c37d..3e4cc9c 100644
--- a/block/blk-flush.c
+++ b/block/blk-flush.c
@@ -231,7 +231,7 @@ static void flush_end_io(struct request *flush_rq, int error)
 	unsigned long flags = 0;
 
 	if (q->mq_ops) {
-		blk_mq_finish_request(flush_rq, error);
+		blk_mq_free_request(flush_rq);
 		spin_lock_irqsave(&q->mq_flush_lock, flags);
 	}
 	running = &q->flush_queue[q->flush_running_idx];
diff --git a/block/blk-mq.c b/block/blk-mq.c
index 799d305..5489b5a 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -270,7 +270,7 @@ void blk_mq_free_request(struct request *rq)
 }
 EXPORT_SYMBOL(blk_mq_free_request);
 
-void blk_mq_finish_request(struct request *rq, int error)
+static void blk_mq_finish_request(struct request *rq, int error)
 {
 	struct bio *bio = rq->bio;
 	unsigned int bytes = 0;
@@ -286,22 +286,17 @@ void blk_mq_finish_request(struct request *rq, int error)
 
 	blk_account_io_completion(rq, bytes);
 	blk_account_io_done(rq);
-	blk_mq_free_request(rq);
 }
 
 void blk_mq_complete_request(struct request *rq, int error)
 {
 	trace_block_rq_complete(rq->q, rq);
+	blk_mq_finish_request(rq, error);
 
-	/*
-	 * If ->end_io is set, it's responsible for doing the rest of the
-	 * completion.
-	 */
 	if (rq->end_io)
 		rq->end_io(rq, error);
 	else
-		blk_mq_finish_request(rq, error);
-
+		blk_mq_free_request(rq);
 }
 
 void __blk_mq_end_io(struct request *rq, int error)
@@ -973,8 +968,7 @@ int blk_mq_execute_rq(struct request_queue *q, struct request *rq)
 	if (rq->errors)
 		err = -EIO;
 
-	blk_mq_finish_request(rq, rq->errors);
-
+	blk_mq_free_request(rq);
 	return err;
 }
 EXPORT_SYMBOL(blk_mq_execute_rq);
diff --git a/block/blk-mq.h b/block/blk-mq.h
index 42d0110..52bf1f9 100644
--- a/block/blk-mq.h
+++ b/block/blk-mq.h
@@ -27,7 +27,6 @@ void blk_mq_complete_request(struct request *rq, int error);
 void blk_mq_run_request(struct request *rq, bool run_queue, bool async);
 void blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async);
 void blk_mq_init_flush(struct request_queue *q);
-void blk_mq_finish_request(struct request *rq, int error);
 
 /*
  * CPU hotplug helpers

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Mike Christie]

[-- Attachment #1: Type: text/plain, Size: 4062 bytes --]

On 07/19/2013 11:56 PM, Nicholas A. Bellinger wrote:
> On Fri, 2013-07-19 at 14:01 -0700, Nicholas A. Bellinger wrote:
>> On Fri, 2013-07-19 at 08:33 -0700, James Bottomley wrote:
>>> On Thu, 2013-07-18 at 23:34 -0700, Nicholas A. Bellinger wrote:
>>>> diff --git a/drivers/ata/libata-scsi.c b/drivers/ata/libata-scsi.c
>>>> index 0101af5..191bc15 100644
>>>> --- a/drivers/ata/libata-scsi.c
>>>> +++ b/drivers/ata/libata-scsi.c
>>>> @@ -1144,7 +1144,11 @@ static int ata_scsi_dev_config(struct scsi_device *sdev,
>>>>                         "sector_size=%u > PAGE_SIZE, PIO may malfunction\n",
>>>>                         sdev->sector_size);
>>>>  
>>>> -       blk_queue_update_dma_alignment(q, sdev->sector_size - 1);
>>>> +       if (!q->mq_ops) {
>>>> +               blk_queue_update_dma_alignment(q, sdev->sector_size - 1);
>>>> +       } else {
>>>> +               printk("Skipping dma_alignment for libata w/ scsi-mq\n");
>>>> +       }
>>>
>>> Amazingly enough there is a reason for the dma alignment, and it wasn't
>>> just to annoy you, so you can't blindly do this.
>>>
>>> The email thread is probably lost in the mists of time, but if I
>>> remember correctly the problem is that some ahci DMA controllers barf if
>>> the sector they're doing DMA on crosses a page boundary.  Some are
>>> annoying enough to actually cause silent data corruption.  You won't
>>> find every ahci DMA controller doing this, so the change will work for
>>> some, but it will be hard to identify those it won't work for until
>>> people start losing data.
>>
>> Thanks for the extra background.
>>
>> So at least from what I gather thus far this shouldn't be an issue for
>> initial testing with scsi-mq <-> libata w/ ata_piix.
>>
>>>
>>> The correct fix, obviously, is to do the bio copy on the kernel path for
>>> unaligned data.  It is OK to assume that REQ_TYPE_FS data is correctly
>>> aligned (because of the block to page alignment).
>>>
>>
>> Indeed.  Looking into the bio_copy_kern() breakage next..
>>
> 
> OK, after further investigation the root cause is a actually a missing
> bio->bio_end_io() -> bio_copy_kern_endio() -> bio_put() from the
> blk_end_sync_rq() callback path that scsi-mq REQ_TYPE_BLOCK_PC is
> currently using.
> 
> Including the following patch into the scsi-mq working branch now, and
> reverting the libata dma_alignment=0x03 hack.
> 
> Alexander, care to give this a try..?
> 
> --nab
> 
> diff --git a/block/blk-exec.c b/block/blk-exec.c
> index 0761c89..70303d2 100644
> --- a/block/blk-exec.c
> +++ b/block/blk-exec.c
> @@ -25,7 +25,10 @@ static void blk_end_sync_rq(struct request *rq, int error)
>         struct completion *waiting = rq->end_io_data;
>  
>         rq->end_io_data = NULL;
> -       if (!rq->q->mq_ops) {
> +       if (rq->q->mq_ops) {
> +               if (rq->bio)
> +                       bio_endio(rq->bio, error);
> +       } else {
>                 __blk_put_request(rq->q, rq);
>         }
> 


This does not handle requests with multiple bios, and for the mq stye
passthrough insertion completions you actually want to call
blk_mq_finish_request in scsi_execute. Same for all the other
passthrough code in your scsi mq tree. That is your root bug. Instead of
doing that though I think we want to have the block layer free the bios
like before.

For the non mq calls, blk_end_request type of calls will complete the
bios when blk_finish_request is called from that path. It will then call
the rq end_io callback.

I think the blk mq code assumes if the end_io callack is set that the
end_io function will do the bio cleanup. See __blk_mq_end_io. Also see
how blk_mq_execute_rq calls blk_mq_finish_request for an example of how
rq passthrough execution and cleanup is being done in the mq paths.

Now with the scsi mq changes, when blk_execute_rq_nowait calls
blk_mq_insert_request it calls it with a old non mq style of end io
function that does not complete the bios.

What about the attached only compile tested patch. The patch has the mq
block code work like the non mq code for bio cleanups.



[-- Attachment #2: blk-mq-free-bio.patch --]
[-- Type: text/plain, Size: 2513 bytes --]

blk-mq: blk-mq should free bios in pass through case

For non mq calls, the block layer will free the bios when
blk_finish_request is called.

For mq calls, the blk mq code wants the caller to do this.

This patch has the blk mq code work like the non mq code
and has the block layer free the bios.

Signed-off-by: Mike Christie <michaelc@cs.wisc.edu>

diff --git a/block/blk-flush.c b/block/blk-flush.c
index c56c37d..3e4cc9c 100644
--- a/block/blk-flush.c
+++ b/block/blk-flush.c
@@ -231,7 +231,7 @@ static void flush_end_io(struct request *flush_rq, int error)
 	unsigned long flags = 0;
 
 	if (q->mq_ops) {
-		blk_mq_finish_request(flush_rq, error);
+		blk_mq_free_request(flush_rq);
 		spin_lock_irqsave(&q->mq_flush_lock, flags);
 	}
 	running = &q->flush_queue[q->flush_running_idx];
diff --git a/block/blk-mq.c b/block/blk-mq.c
index 799d305..5489b5a 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -270,7 +270,7 @@ void blk_mq_free_request(struct request *rq)
 }
 EXPORT_SYMBOL(blk_mq_free_request);
 
-void blk_mq_finish_request(struct request *rq, int error)
+static void blk_mq_finish_request(struct request *rq, int error)
 {
 	struct bio *bio = rq->bio;
 	unsigned int bytes = 0;
@@ -286,22 +286,17 @@ void blk_mq_finish_request(struct request *rq, int error)
 
 	blk_account_io_completion(rq, bytes);
 	blk_account_io_done(rq);
-	blk_mq_free_request(rq);
 }
 
 void blk_mq_complete_request(struct request *rq, int error)
 {
 	trace_block_rq_complete(rq->q, rq);
+	blk_mq_finish_request(rq, error);
 
-	/*
-	 * If ->end_io is set, it's responsible for doing the rest of the
-	 * completion.
-	 */
 	if (rq->end_io)
 		rq->end_io(rq, error);
 	else
-		blk_mq_finish_request(rq, error);
-
+		blk_mq_free_request(rq);
 }
 
 void __blk_mq_end_io(struct request *rq, int error)
@@ -973,8 +968,7 @@ int blk_mq_execute_rq(struct request_queue *q, struct request *rq)
 	if (rq->errors)
 		err = -EIO;
 
-	blk_mq_finish_request(rq, rq->errors);
-
+	blk_mq_free_request(rq);
 	return err;
 }
 EXPORT_SYMBOL(blk_mq_execute_rq);
diff --git a/block/blk-mq.h b/block/blk-mq.h
index 42d0110..52bf1f9 100644
--- a/block/blk-mq.h
+++ b/block/blk-mq.h
@@ -27,7 +27,6 @@ void blk_mq_complete_request(struct request *rq, int error);
 void blk_mq_run_request(struct request *rq, bool run_queue, bool async);
 void blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async);
 void blk_mq_init_flush(struct request_queue *q);
-void blk_mq_finish_request(struct request *rq, int error);
 
 /*
  * CPU hotplug helpers

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Nicholas A. Bellinger]

On Sat, 2013-07-20 at 09:48 -0500, Mike Christie wrote:
> On 07/19/2013 11:56 PM, Nicholas A. Bellinger wrote:
> > On Fri, 2013-07-19 at 14:01 -0700, Nicholas A. Bellinger wrote:
> >> On Fri, 2013-07-19 at 08:33 -0700, James Bottomley wrote:

<SNIP>

> >>
> >> Indeed.  Looking into the bio_copy_kern() breakage next..
> >>
> > 
> > OK, after further investigation the root cause is a actually a missing
> > bio->bio_end_io() -> bio_copy_kern_endio() -> bio_put() from the
> > blk_end_sync_rq() callback path that scsi-mq REQ_TYPE_BLOCK_PC is
> > currently using.
> > 
> > Including the following patch into the scsi-mq working branch now, and
> > reverting the libata dma_alignment=0x03 hack.
> > 
> > Alexander, care to give this a try..?
> > 
> > --nab
> > 
> > diff --git a/block/blk-exec.c b/block/blk-exec.c
> > index 0761c89..70303d2 100644
> > --- a/block/blk-exec.c
> > +++ b/block/blk-exec.c
> > @@ -25,7 +25,10 @@ static void blk_end_sync_rq(struct request *rq, int error)
> >         struct completion *waiting = rq->end_io_data;
> >  
> >         rq->end_io_data = NULL;
> > -       if (!rq->q->mq_ops) {
> > +       if (rq->q->mq_ops) {
> > +               if (rq->bio)
> > +                       bio_endio(rq->bio, error);
> > +       } else {
> >                 __blk_put_request(rq->q, rq);
> >         }
> > 
> 
> 
> This does not handle requests with multiple bios, and for the mq stye
> passthrough insertion completions you actually want to call
> blk_mq_finish_request in scsi_execute. Same for all the other
> passthrough code in your scsi mq tree. That is your root bug. Instead of
> doing that though I think we want to have the block layer free the bios
> like before.
> 
> For the non mq calls, blk_end_request type of calls will complete the
> bios when blk_finish_request is called from that path. It will then call
> the rq end_io callback.
> 
> I think the blk mq code assumes if the end_io callack is set that the
> end_io function will do the bio cleanup. See __blk_mq_end_io. Also see
> how blk_mq_execute_rq calls blk_mq_finish_request for an example of how
> rq passthrough execution and cleanup is being done in the mq paths.
> 
> Now with the scsi mq changes, when blk_execute_rq_nowait calls
> blk_mq_insert_request it calls it with a old non mq style of end io
> function that does not complete the bios.
> 
> What about the attached only compile tested patch. The patch has the mq
> block code work like the non mq code for bio cleanups.
> 
> 

OK, so with your blk-mq patch in place to always call
blk_mq_finish_request() in blk_mq_complete_request() regardless of
rq->end_io, the preceding scsi_mq_end_request() can now simply call
blk_mq_end_io() for both BLOCK_RQ and FS request types.

diff --git a/drivers/scsi/scsi-mq.c b/drivers/scsi/scsi-mq.c
index 81b2633..f1d4789 100644
--- a/drivers/scsi/scsi-mq.c
+++ b/drivers/scsi/scsi-mq.c
@@ -93,19 +93,7 @@ struct scsi_cmnd *scsi_mq_end_request(struct scsi_cmnd *sc, int error,
 #endif
 
 //FIXME: Add proper blk_mq_end_io residual bytes + requeue
-       if (rq->end_io) {
-#if 0
-               printk("scsi_mq_end_request: Calling rq->end_io BLOCK_PC for"
-                       " CDB: 0x%02x\n", sc->cmnd[0]);
-#endif
-               rq->end_io(rq, error);
-       } else {
-#if 0
-               printk("scsi_mq_end_request: Calling blk_mq_end_io for CDB: 0x%02x\n",
-                               sc->cmnd[0]);
-#endif
-               blk_mq_end_io(rq, error);
-       }
+       blk_mq_end_io(rq, error);
 //FIXME: Need to do equiv of scsi_next_command to kick hctx..?
 
        return NULL;

Thanks for fixing up that bit of ugliness.  ;)

Jens, care to review+include Mike's change into your working branch..?

--nab

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Jens Axboe]

On Sat, Jul 20 2013, Mike Christie wrote:
> blk-mq: blk-mq should free bios in pass through case
> 
> For non mq calls, the block layer will free the bios when
> blk_finish_request is called.
> 
> For mq calls, the blk mq code wants the caller to do this.
> 
> This patch has the blk mq code work like the non mq code
> and has the block layer free the bios.

Thanks Mike, looks good, applied.

-- 
Jens Axboe

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

On Fri, Jul 19, 2013 at 09:56:02PM -0700, Nicholas A. Bellinger wrote:
> On Fri, 2013-07-19 at 14:01 -0700, Nicholas A. Bellinger wrote:
> OK, after further investigation the root cause is a actually a missing
> bio->bio_end_io() -> bio_copy_kern_endio() -> bio_put() from the
> blk_end_sync_rq() callback path that scsi-mq REQ_TYPE_BLOCK_PC is
> currently using.

Yes, missing bio_copy_kern_endio() callback is exactly the reason I
turned to blk_mq_execute_rq() initially. I should have been more
specific on this :|

I will try Mike's and your other change, hopefully soon (sorry,
constantly getting distracted).

> Including the following patch into the scsi-mq working branch now, and
> reverting the libata dma_alignment=0x03 hack.

-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Nicholas A. Bellinger]

On Mon, 2013-07-22 at 17:03 +0200, Alexander Gordeev wrote:
> On Fri, Jul 19, 2013 at 09:56:02PM -0700, Nicholas A. Bellinger wrote:
> > On Fri, 2013-07-19 at 14:01 -0700, Nicholas A. Bellinger wrote:
> > OK, after further investigation the root cause is a actually a missing
> > bio->bio_end_io() -> bio_copy_kern_endio() -> bio_put() from the
> > blk_end_sync_rq() callback path that scsi-mq REQ_TYPE_BLOCK_PC is
> > currently using.
> 
> Yes, missing bio_copy_kern_endio() callback is exactly the reason I
> turned to blk_mq_execute_rq() initially. I should have been more
> specific on this :|
> 
> I will try Mike's and your other change, hopefully soon (sorry,
> constantly getting distracted).
> 

Np.  FYI, you'll want to use the latest commit e7827b351 HEAD from
target-pending/scsi-mq, which now has functioning scsi-generic support.

Also, your scsi_times_out patch from earlier has not been included just
yet, but that should be the only extra patch you need to apply in order
to get scsi-mq enabled libata/ata_piix running.

--nab

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

On Mon, Jul 22, 2013 at 02:10:36PM -0700, Nicholas A. Bellinger wrote:
> Np.  FYI, you'll want to use the latest commit e7827b351 HEAD from
> target-pending/scsi-mq, which now has functioning scsi-generic support.

Survives a boot, a kernel build and the build's result :)

-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Nicholas A. Bellinger]

On Thu, 2013-07-25 at 12:16 +0200, Alexander Gordeev wrote:
> On Mon, Jul 22, 2013 at 02:10:36PM -0700, Nicholas A. Bellinger wrote:
> > Np.  FYI, you'll want to use the latest commit e7827b351 HEAD from
> > target-pending/scsi-mq, which now has functioning scsi-generic support.
> 
> Survives a boot, a kernel build and the build's result :)

Great.  Thanks for the feedback Alexander!

So the next step on my end is to enable -mq for ahci, and verify initial
correctness using QEMU/KVM hardware emulation.

Btw, I've been looking at enabling the SHT->cmd_size for struct
ata_queued_cmd descriptor pre-allocation, but AFAICT these descriptors
are already all pre-allocated by libata and obtained via ata_qc_new() ->
__ata_qc_from_tag() during ata_scsi_queuecmd().

So that said, with the struct request + struct scsi_cmnd pre-allocations
already provided by blk-mq -> scsi-mq code, all memory allocations
should have already been eliminated from I/O fast path.

--nab

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Jens Axboe]

On Thu, Jul 25 2013, Nicholas A. Bellinger wrote:
> On Thu, 2013-07-25 at 12:16 +0200, Alexander Gordeev wrote:
> > On Mon, Jul 22, 2013 at 02:10:36PM -0700, Nicholas A. Bellinger wrote:
> > > Np.  FYI, you'll want to use the latest commit e7827b351 HEAD from
> > > target-pending/scsi-mq, which now has functioning scsi-generic support.
> > 
> > Survives a boot, a kernel build and the build's result :)
> 
> Great.  Thanks for the feedback Alexander!
> 
> So the next step on my end is to enable -mq for ahci, and verify initial
> correctness using QEMU/KVM hardware emulation.
> 
> Btw, I've been looking at enabling the SHT->cmd_size for struct
> ata_queued_cmd descriptor pre-allocation, but AFAICT these descriptors
> are already all pre-allocated by libata and obtained via ata_qc_new() ->
> __ata_qc_from_tag() during ata_scsi_queuecmd().

Might still not be a bad idea to do it:

- Cleans up a driver, getting rid of the need to alloc, maintain, and
  free those structures.

- Should be some cache locality benefits to having it all sequential.

> So that said, with the struct request + struct scsi_cmnd pre-allocations
> already provided by blk-mq -> scsi-mq code, all memory allocations
> should have already been eliminated from I/O fast path.

Nice!

-- 
Jens Axboe

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Nicholas A. Bellinger]

On Thu, 2013-07-25 at 20:09 -0600, Jens Axboe wrote:
> On Thu, Jul 25 2013, Nicholas A. Bellinger wrote:
> > On Thu, 2013-07-25 at 12:16 +0200, Alexander Gordeev wrote:
> > > On Mon, Jul 22, 2013 at 02:10:36PM -0700, Nicholas A. Bellinger wrote:
> > > > Np.  FYI, you'll want to use the latest commit e7827b351 HEAD from
> > > > target-pending/scsi-mq, which now has functioning scsi-generic support.
> > > 
> > > Survives a boot, a kernel build and the build's result :)
> > 
> > Great.  Thanks for the feedback Alexander!
> > 
> > So the next step on my end is to enable -mq for ahci, and verify initial
> > correctness using QEMU/KVM hardware emulation.
> > 
> > Btw, I've been looking at enabling the SHT->cmd_size for struct
> > ata_queued_cmd descriptor pre-allocation, but AFAICT these descriptors
> > are already all pre-allocated by libata and obtained via ata_qc_new() ->
> > __ata_qc_from_tag() during ata_scsi_queuecmd().
> 
> Might still not be a bad idea to do it:
> 
> - Cleans up a driver, getting rid of the need to alloc, maintain, and
>   free those structures.
> 
> - Should be some cache locality benefits to having it all sequential.
> 

Looking at this some more, there are a number of locations outside of
the main blk_mq_ops->queue_rq() -> SHT->queuecommand_mq() dispatch that
use *ata_qc_from_tag() to obtain *ata_queued_cmd, and a few without a
associated struct scsi_cmnd like libata-core.c:ata_exec_internal_sg()
for example..

So I don't think (completely) getting rid of ata_port->qcmds[] will be
possible, and just converting the ata_scsi_queuecmd() path to use the
extra SHT->cmd_size pre-allocation for *ata_queued_cmd might end up
being more trouble that it's worth.  Still undecided on that part..

Tejun, do you have any thoughts + input here..?

--nab

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Nicholas A. Bellinger]

On Fri, 2013-07-26 at 14:14 -0700, Nicholas A. Bellinger wrote:
> On Thu, 2013-07-25 at 20:09 -0600, Jens Axboe wrote:
> > On Thu, Jul 25 2013, Nicholas A. Bellinger wrote:
> > > On Thu, 2013-07-25 at 12:16 +0200, Alexander Gordeev wrote:
> > > > On Mon, Jul 22, 2013 at 02:10:36PM -0700, Nicholas A. Bellinger wrote:
> > > > > Np.  FYI, you'll want to use the latest commit e7827b351 HEAD from
> > > > > target-pending/scsi-mq, which now has functioning scsi-generic support.
> > > > 
> > > > Survives a boot, a kernel build and the build's result :)
> > > 
> > > Great.  Thanks for the feedback Alexander!
> > > 
> > > So the next step on my end is to enable -mq for ahci, and verify initial
> > > correctness using QEMU/KVM hardware emulation.
> > > 
> > > Btw, I've been looking at enabling the SHT->cmd_size for struct
> > > ata_queued_cmd descriptor pre-allocation, but AFAICT these descriptors
> > > are already all pre-allocated by libata and obtained via ata_qc_new() ->
> > > __ata_qc_from_tag() during ata_scsi_queuecmd().
> > 
> > Might still not be a bad idea to do it:
> > 
> > - Cleans up a driver, getting rid of the need to alloc, maintain, and
> >   free those structures.
> > 
> > - Should be some cache locality benefits to having it all sequential.
> > 
> 
> Looking at this some more, there are a number of locations outside of
> the main blk_mq_ops->queue_rq() -> SHT->queuecommand_mq() dispatch that
> use *ata_qc_from_tag() to obtain *ata_queued_cmd, and a few without a
> associated struct scsi_cmnd like libata-core.c:ata_exec_internal_sg()
> for example..
> 
> So I don't think (completely) getting rid of ata_port->qcmds[] will be
> possible, and just converting the ata_scsi_queuecmd() path to use the
> extra SHT->cmd_size pre-allocation for *ata_queued_cmd might end up
> being more trouble that it's worth.  Still undecided on that part..
> 
> Tejun, do you have any thoughts + input here..?
> 

OK, so I decided to give this a shot anyways..  Here is a quick
conversion for libata + AHCI to use blk-mq -> scsi-mq pre-allocation for
ata_queued_cmd descriptors:

diff --git a/drivers/ata/ahci.c b/drivers/ata/ahci.c
index 2b50dfd..61b3db8 100644
--- a/drivers/ata/ahci.c
+++ b/drivers/ata/ahci.c
@@ -92,6 +92,9 @@ static int ahci_pci_device_resume(struct pci_dev *pdev);
 
 static struct scsi_host_template ahci_sht = {
        AHCI_SHT("ahci"),
+       .scsi_mq = true,
+       .cmd_size = sizeof(struct ata_queued_cmd),
+       .queuecommand_mq = ata_scsi_queuecmd,
 };
diff --git a/drivers/ata/libata-core.c b/drivers/ata/libata-core.c
index f218427..e21814d 100644
--- a/drivers/ata/libata-core.c
+++ b/drivers/ata/libata-core.c
@@ -4725,29 +4725,25 @@ void swap_buf_le16(u16 *buf, unsigned int buf_words)
 /**
  *     ata_qc_new - Request an available ATA command, for queueing
  *     @ap: target port
+ *     @sc: incoming scsi_cmnd descriptor
  *
  *     LOCKING:
  *     None.
  */
 
-static struct ata_queued_cmd *ata_qc_new(struct ata_port *ap)
+static struct ata_queued_cmd *ata_qc_new(struct ata_port *ap,
+                                        struct scsi_cmnd *sc)
 {
        struct ata_queued_cmd *qc = NULL;
-       unsigned int i;
+       struct request *rq = sc->request;
 
        /* no command while frozen */
        if (unlikely(ap->pflags & ATA_PFLAG_FROZEN))
                return NULL;
 
-       /* the last tag is reserved for internal command. */
-       for (i = 0; i < ATA_MAX_QUEUE - 1; i++)
-               if (!test_and_set_bit(i, &ap->qc_allocated)) {
-                       qc = __ata_qc_from_tag(ap, i);
-                       break;
-               }
-
-       if (qc)
-               qc->tag = i;
+       qc = (struct ata_queued_cmd *)sc->SCp.ptr;
+       qc->scsicmd = sc;
+       qc->tag = rq->tag;
 
        return qc;
 }
@@ -4755,19 +4751,20 @@ static struct ata_queued_cmd *ata_qc_new(struct ata_port *ap)
 /**
  *     ata_qc_new_init - Request an available ATA command, and initialize it
  *     @dev: Device from whom we request an available command structure
+ *     @sc: incoming scsi_cmnd descriptor
  *
  *     LOCKING:
  *     None.
  */
 
-struct ata_queued_cmd *ata_qc_new_init(struct ata_device *dev)
+struct ata_queued_cmd *ata_qc_new_init(struct ata_device *dev,
+                                      struct scsi_cmnd *sc)
 {
        struct ata_port *ap = dev->link->ap;
        struct ata_queued_cmd *qc;
 
-       qc = ata_qc_new(ap);
+       qc = ata_qc_new(ap, sc);
        if (qc) {
-               qc->scsicmd = NULL;
                qc->ap = ap;
                qc->dev = dev;
 
@@ -4797,10 +4794,9 @@ void ata_qc_free(struct ata_queued_cmd *qc)
 
        qc->flags = 0;
        tag = qc->tag;
-       if (likely(ata_tag_valid(tag))) {
+
+       if (likely(ata_tag_valid(tag)))
                qc->tag = ATA_TAG_POISON;
-               clear_bit(tag, &ap->qc_allocated);
-       }
 }
 
diff --git a/drivers/ata/libata-scsi.c b/drivers/ata/libata-scsi.c
index 0101af5..e5ab880 100644
--- a/drivers/ata/libata-scsi.c
+++ b/drivers/ata/libata-scsi.c
@@ -742,9 +742,8 @@ static struct ata_queued_cmd *ata_scsi_qc_new(struct ata_device *dev,
 {
        struct ata_queued_cmd *qc;
 
-       qc = ata_qc_new_init(dev);
+       qc = ata_qc_new_init(dev, cmd);
        if (qc) {
-               qc->scsicmd = cmd;
                qc->scsidone = cmd->scsi_done;
 
                qc->sg = scsi_sglist(cmd);
diff --git a/drivers/ata/libata.h b/drivers/ata/libata.h
index c949dd3..4cd88af 100644
--- a/drivers/ata/libata.h
+++ b/drivers/ata/libata.h
@@ -63,7 +63,8 @@ extern struct ata_link *ata_dev_phys_link(struct ata_device *dev);
 extern void ata_force_cbl(struct ata_port *ap);
 extern u64 ata_tf_to_lba(const struct ata_taskfile *tf);
 extern u64 ata_tf_to_lba48(const struct ata_taskfile *tf);
-extern struct ata_queued_cmd *ata_qc_new_init(struct ata_device *dev);
+extern struct ata_queued_cmd *ata_qc_new_init(struct ata_device *dev,
+                                             struct scsi_cmnd *sc);
 extern int ata_build_rw_tf(struct ata_taskfile *tf, struct ata_device *dev,
                           u64 block, u32 n_block, unsigned int tf_flags,
                           unsigned int tag);
diff --git a/include/linux/libata.h b/include/linux/libata.h
index eae7a05..52e9e9e 100644
--- a/include/linux/libata.h
+++ b/include/linux/libata.h
@@ -35,9 +35,13 @@
 #include <linux/ata.h>
 #include <linux/workqueue.h>
 #include <scsi/scsi_host.h>
+#include <scsi/scsi_device.h>
+#include <scsi/scsi_cmnd.h>
 #include <linux/acpi.h>
 #include <linux/cdrom.h>
 #include <linux/sched.h>
+#include <linux/blk-mq.h>
+#include <../../block/blk-mq.h>
 
 /*
  * Define if arch has non-standard setup.  This is a _PCI_ standard
@@ -1500,9 +1504,26 @@ static inline void ata_qc_set_polling(struct ata_queued_cmd *qc)
 static inline struct ata_queued_cmd *__ata_qc_from_tag(struct ata_port *ap,
                                                       unsigned int tag)
 {
-       if (likely(ata_tag_valid(tag)))
-               return &ap->qcmd[tag];
-       return NULL;
+       struct scsi_device *sdev = ap->link.device[0].sdev;
+       struct request_queue *q = sdev->request_queue;
+
+       if (unlikely(!ata_tag_valid(tag)))
+               return NULL;
+
+       if (likely(sdev->host->hostt->scsi_mq)) {
+               struct blk_mq_ctx *ctx = blk_mq_get_ctx(q);
+               struct blk_mq_hw_ctx *hctx = q->mq_ops->map_queue(q, ctx->cpu);
+               struct request *rq;
+               struct ata_queued_cmd *qc;
+
+               BUG_ON(tag > hctx->queue_depth);
+
+               rq = hctx->rqs[tag];
+               qc = blk_mq_rq_to_pdu(rq) + sizeof(struct scsi_cmnd);
+               blk_mq_put_ctx(ctx);
+               return qc;
+       }
+       return &ap->qcmd[tag];
 }

The thing that I'm hung up on now for existing __ata_qc_from_tag() usage
outside of the main blk_mq_ops->queue_rq -> SHT->queuecommand_mq()
dispatch path, is how to actually locate the underlying scsi_device ->
request_queue -> blk_mq_ctx -> blk_mq_hw_hctx from the passed
ata_port..?

Considering there can be more than a single ata_device hanging off each
ata_port, the '*sdev = ap->link.device[0].sdev' in __ata_qc_from_tag()
is definitely bogus, but I'm not sure how else to correlate
blk-mq/scsi-mq per device descriptors to existing code expecting
ata_port->qcmd[] descriptors to be shared across multiple devices..

Tejun..?

--nab

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Hannes Reinecke]

On 07/26/2013 04:09 AM, Jens Axboe wrote:
> On Thu, Jul 25 2013, Nicholas A. Bellinger wrote:
>> On Thu, 2013-07-25 at 12:16 +0200, Alexander Gordeev wrote:
>>> On Mon, Jul 22, 2013 at 02:10:36PM -0700, Nicholas A. Bellinger wrote:
>>>> Np.  FYI, you'll want to use the latest commit e7827b351 HEAD from
>>>> target-pending/scsi-mq, which now has functioning scsi-generic support.
>>>
>>> Survives a boot, a kernel build and the build's result :)
>>
>> Great.  Thanks for the feedback Alexander!
>>
>> So the next step on my end is to enable -mq for ahci, and verify initial
>> correctness using QEMU/KVM hardware emulation.
>>
>> Btw, I've been looking at enabling the SHT->cmd_size for struct
>> ata_queued_cmd descriptor pre-allocation, but AFAICT these descriptors
>> are already all pre-allocated by libata and obtained via ata_qc_new() ->
>> __ata_qc_from_tag() during ata_scsi_queuecmd().
> 
> Might still not be a bad idea to do it:
> 
> - Cleans up a driver, getting rid of the need to alloc, maintain, and
>   free those structures.
> 
> - Should be some cache locality benefits to having it all sequential.
> 
>> So that said, with the struct request + struct scsi_cmnd pre-allocations
>> already provided by blk-mq -> scsi-mq code, all memory allocations
>> should have already been eliminated from I/O fast path.
> 
> Nice!
> 
Hmm.

I'm trying to work out if it would be possible to move multipath
handling over to scsi-mq.
However, when doing so I would need to reconfigure 'nr_hw_queues' on
the fly. Now with all the static cmd preallocation going on this is
going to be tricky.

This leaves me with two choices:
- Tear down the command pool altogether whenever I need to
  reconfigure the device (which is going to be painful)
- Allocate some max nr_hw_queues, and mark the superfluous
  ones as 'unused' or something. Seeing the a sane max nr_hw_queues
  will be possibly the number of cpus this might end up hogging
  quite some memory.

Would you accept patches moving the static command allocation over
to pools or is this a desired feature?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

On Fri, Jul 26, 2013 at 05:43:13PM -0700, Nicholas A. Bellinger wrote:
> On Fri, 2013-07-26 at 14:14 -0700, Nicholas A. Bellinger wrote:
> > On Thu, 2013-07-25 at 20:09 -0600, Jens Axboe wrote:
> > > On Thu, Jul 25 2013, Nicholas A. Bellinger wrote:
> > > > On Thu, 2013-07-25 at 12:16 +0200, Alexander Gordeev wrote:
> > > > > On Mon, Jul 22, 2013 at 02:10:36PM -0700, Nicholas A. Bellinger wrote:
> > > > > > Np.  FYI, you'll want to use the latest commit e7827b351 HEAD from
> > > > > > target-pending/scsi-mq, which now has functioning scsi-generic support.
> > > > > 
> > > > > Survives a boot, a kernel build and the build's result :)
> > > > 
> > > > Great.  Thanks for the feedback Alexander!
> > > > 
> > > > So the next step on my end is to enable -mq for ahci, and verify initial
> > > > correctness using QEMU/KVM hardware emulation.
> > > > 
> > > > Btw, I've been looking at enabling the SHT->cmd_size for struct
> > > > ata_queued_cmd descriptor pre-allocation, but AFAICT these descriptors
> > > > are already all pre-allocated by libata and obtained via ata_qc_new() ->
> > > > __ata_qc_from_tag() during ata_scsi_queuecmd().
> > > 
> > > Might still not be a bad idea to do it:
> > > 
> > > - Cleans up a driver, getting rid of the need to alloc, maintain, and
> > >   free those structures.
> > > 
> > > - Should be some cache locality benefits to having it all sequential.
> > > 
> > 
> > Looking at this some more, there are a number of locations outside of
> > the main blk_mq_ops->queue_rq() -> SHT->queuecommand_mq() dispatch that
> > use *ata_qc_from_tag() to obtain *ata_queued_cmd, and a few without a
> > associated struct scsi_cmnd like libata-core.c:ata_exec_internal_sg()
> > for example..
> > 
> > So I don't think (completely) getting rid of ata_port->qcmds[] will be
> > possible, and just converting the ata_scsi_queuecmd() path to use the
> > extra SHT->cmd_size pre-allocation for *ata_queued_cmd might end up
> > being more trouble that it's worth.  Still undecided on that part..
> > 
> > Tejun, do you have any thoughts + input here..?
> > 
> 
> OK, so I decided to give this a shot anyways..  Here is a quick
> conversion for libata + AHCI to use blk-mq -> scsi-mq pre-allocation for
> ata_queued_cmd descriptors:
> 
> diff --git a/drivers/ata/ahci.c b/drivers/ata/ahci.c
> index 2b50dfd..61b3db8 100644
> --- a/drivers/ata/ahci.c
> +++ b/drivers/ata/ahci.c
> @@ -92,6 +92,9 @@ static int ahci_pci_device_resume(struct pci_dev *pdev);
>  
>  static struct scsi_host_template ahci_sht = {
>         AHCI_SHT("ahci"),
> +       .scsi_mq = true,
> +       .cmd_size = sizeof(struct ata_queued_cmd),
> +       .queuecommand_mq = ata_scsi_queuecmd,
>  };
> diff --git a/drivers/ata/libata-core.c b/drivers/ata/libata-core.c
> index f218427..e21814d 100644
> --- a/drivers/ata/libata-core.c
> +++ b/drivers/ata/libata-core.c
> @@ -4725,29 +4725,25 @@ void swap_buf_le16(u16 *buf, unsigned int buf_words)
>  /**
>   *     ata_qc_new - Request an available ATA command, for queueing
>   *     @ap: target port
> + *     @sc: incoming scsi_cmnd descriptor
>   *
>   *     LOCKING:
>   *     None.
>   */
>  
> -static struct ata_queued_cmd *ata_qc_new(struct ata_port *ap)
> +static struct ata_queued_cmd *ata_qc_new(struct ata_port *ap,
> +                                        struct scsi_cmnd *sc)
>  {
>         struct ata_queued_cmd *qc = NULL;
> -       unsigned int i;
> +       struct request *rq = sc->request;
>  
>         /* no command while frozen */
>         if (unlikely(ap->pflags & ATA_PFLAG_FROZEN))
>                 return NULL;
>  
> -       /* the last tag is reserved for internal command. */
> -       for (i = 0; i < ATA_MAX_QUEUE - 1; i++)

blk-mq does not prevent tag ATA_TAG_INTERNAL from being using. Would it make
sense to promote queue depth of length (ATA_MAX_QUEUE - 1) while always
pointing ATA_TAG_INTERNAL to qcmd (see below)?

> -               if (!test_and_set_bit(i, &ap->qc_allocated)) {

ata_port::qc_allocated becomes redundant.

> -                       qc = __ata_qc_from_tag(ap, i);
> -                       break;
> -               }
> -
> -       if (qc)
> -               qc->tag = i;
> +       qc = (struct ata_queued_cmd *)sc->SCp.ptr;
> +       qc->scsicmd = sc;
> +       qc->tag = rq->tag;
>  
>         return qc;
>  }
> @@ -4755,19 +4751,20 @@ static struct ata_queued_cmd *ata_qc_new(struct ata_port *ap)
>  /**
>   *     ata_qc_new_init - Request an available ATA command, and initialize it
>   *     @dev: Device from whom we request an available command structure
> + *     @sc: incoming scsi_cmnd descriptor
>   *
>   *     LOCKING:
>   *     None.
>   */
>  
> -struct ata_queued_cmd *ata_qc_new_init(struct ata_device *dev)
> +struct ata_queued_cmd *ata_qc_new_init(struct ata_device *dev,
> +                                      struct scsi_cmnd *sc)
>  {
>         struct ata_port *ap = dev->link->ap;
>         struct ata_queued_cmd *qc;
>  
> -       qc = ata_qc_new(ap);
> +       qc = ata_qc_new(ap, sc);
>         if (qc) {
> -               qc->scsicmd = NULL;
>                 qc->ap = ap;
>                 qc->dev = dev;
>  
> @@ -4797,10 +4794,9 @@ void ata_qc_free(struct ata_queued_cmd *qc)
>  
>         qc->flags = 0;
>         tag = qc->tag;
> -       if (likely(ata_tag_valid(tag))) {
> +
> +       if (likely(ata_tag_valid(tag)))
>                 qc->tag = ATA_TAG_POISON;
> -               clear_bit(tag, &ap->qc_allocated);
> -       }
>  }
>  
> diff --git a/drivers/ata/libata-scsi.c b/drivers/ata/libata-scsi.c
> index 0101af5..e5ab880 100644
> --- a/drivers/ata/libata-scsi.c
> +++ b/drivers/ata/libata-scsi.c
> @@ -742,9 +742,8 @@ static struct ata_queued_cmd *ata_scsi_qc_new(struct ata_device *dev,
>  {
>         struct ata_queued_cmd *qc;
>  
> -       qc = ata_qc_new_init(dev);
> +       qc = ata_qc_new_init(dev, cmd);
>         if (qc) {
> -               qc->scsicmd = cmd;
>                 qc->scsidone = cmd->scsi_done;
>  
>                 qc->sg = scsi_sglist(cmd);
> diff --git a/drivers/ata/libata.h b/drivers/ata/libata.h
> index c949dd3..4cd88af 100644
> --- a/drivers/ata/libata.h
> +++ b/drivers/ata/libata.h
> @@ -63,7 +63,8 @@ extern struct ata_link *ata_dev_phys_link(struct ata_device *dev);
>  extern void ata_force_cbl(struct ata_port *ap);
>  extern u64 ata_tf_to_lba(const struct ata_taskfile *tf);
>  extern u64 ata_tf_to_lba48(const struct ata_taskfile *tf);
> -extern struct ata_queued_cmd *ata_qc_new_init(struct ata_device *dev);
> +extern struct ata_queued_cmd *ata_qc_new_init(struct ata_device *dev,
> +                                             struct scsi_cmnd *sc);
>  extern int ata_build_rw_tf(struct ata_taskfile *tf, struct ata_device *dev,
>                            u64 block, u32 n_block, unsigned int tf_flags,
>                            unsigned int tag);
> diff --git a/include/linux/libata.h b/include/linux/libata.h
> index eae7a05..52e9e9e 100644
> --- a/include/linux/libata.h
> +++ b/include/linux/libata.h
> @@ -35,9 +35,13 @@
>  #include <linux/ata.h>
>  #include <linux/workqueue.h>
>  #include <scsi/scsi_host.h>
> +#include <scsi/scsi_device.h>
> +#include <scsi/scsi_cmnd.h>
>  #include <linux/acpi.h>
>  #include <linux/cdrom.h>
>  #include <linux/sched.h>
> +#include <linux/blk-mq.h>
> +#include <../../block/blk-mq.h>
>  
>  /*
>   * Define if arch has non-standard setup.  This is a _PCI_ standard
> @@ -1500,9 +1504,26 @@ static inline void ata_qc_set_polling(struct ata_queued_cmd *qc)
>  static inline struct ata_queued_cmd *__ata_qc_from_tag(struct ata_port *ap,
>                                                        unsigned int tag)
>  {
> -       if (likely(ata_tag_valid(tag)))
> -               return &ap->qcmd[tag];
> -       return NULL;
> +       struct scsi_device *sdev = ap->link.device[0].sdev;
> +       struct request_queue *q = sdev->request_queue;
> +
> +       if (unlikely(!ata_tag_valid(tag)))
> +               return NULL;
> +
> +       if (likely(sdev->host->hostt->scsi_mq)) {

Together with the comment above:

          if (likely(sdev->host->hostt->scsi_mq && (tag != ATA_TAG_INTERNAL))) {
          	...

> +               struct blk_mq_ctx *ctx = blk_mq_get_ctx(q);
> +               struct blk_mq_hw_ctx *hctx = q->mq_ops->map_queue(q, ctx->cpu);
> +               struct request *rq;
> +               struct ata_queued_cmd *qc;
> +
> +               BUG_ON(tag > hctx->queue_depth);
> +
> +               rq = hctx->rqs[tag];
> +               qc = blk_mq_rq_to_pdu(rq) + sizeof(struct scsi_cmnd);
> +               blk_mq_put_ctx(ctx);
> +               return qc;
> +       }
> +       return &ap->qcmd[tag];
>  }

I also tried to make a "quick" conversion and hit the same issue(s) as you.
Generally, I am concerned with these assumptions in such approach:

1. While libata concept of tags matches nicely with blk-mq (blk_mq_hw_ctx::
rqs[] vs ata_port::qcmd[]) right now, it is too exposed to changes in blk-mq
in the long run. I.e. ata_link::sactive limits tags to indices, while tags
might become hashes. Easily fixable, but still.

2. Unallocated requests in blk-mq are accessed/analized from libata-eh.c as
result of such iterations:

        for (tag = 0; tag < ATA_MAX_QUEUE; tag++) {
                qc = __ata_qc_from_tag(ap, tag);

                if (!(qc->flags & ATA_QCFLAG_FAILED))
                        continue;

		...
	}

While it is probably okay right now, it is still based on a premise that
blk-mq will not change the contents/concept of "payload", i.e. from embedded
to (re-)allocated memory.

> The thing that I'm hung up on now for existing __ata_qc_from_tag() usage
> outside of the main blk_mq_ops->queue_rq -> SHT->queuecommand_mq()
> dispatch path, is how to actually locate the underlying scsi_device ->
> request_queue -> blk_mq_ctx -> blk_mq_hw_hctx from the passed
> ata_port..?

I am actually in favor of getting rid of ata_queued_cmd::tag. Converting
ata_link::sactive to a list, making ata_link::active_tag as struct
ata_queued_cmd *ata_link::active_qc and converting ata_port::qc_allocated to a
list seems solves it all, including [2]. Have not checked it though.

Anyway, if we need a blk-mq tag (why?), we have qc->scsicmd->request->tag.

> Considering there can be more than a single ata_device hanging off each
> ata_port, the '*sdev = ap->link.device[0].sdev' in __ata_qc_from_tag()
> is definitely bogus, but I'm not sure how else to correlate
> blk-mq/scsi-mq per device descriptors to existing code expecting
> ata_port->qcmd[] descriptors to be shared across multiple devices..
> 
> Tejun..?
> 
> --nab
> 

-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Tejun Heo]

Hello,

On Fri, Jul 26, 2013 at 02:14:36PM -0700, Nicholas A. Bellinger wrote:
> So I don't think (completely) getting rid of ata_port->qcmds[] will be
> possible, and just converting the ata_scsi_queuecmd() path to use the
> extra SHT->cmd_size pre-allocation for *ata_queued_cmd might end up
> being more trouble that it's worth.  Still undecided on that part..
> 
> Tejun, do you have any thoughts + input here..?

libata exception handling which includes probing doesn't go through
SCSI at all.  It all works inside libata proper using ata_queuecmds
and only the result is exposed to SCSI.  Most of those SCSI semantics
need to be emulated anyway, so this makes things a lot easier than
going through SCSI for each command.  As it currently stands, it'd be
a lot of effort to try to embed ata_qc's into higher layer construct.
Given how it's used, I don't think it's a high priority task.

One thing which would probably be worthwhile tho is getting rid of the
bitmap based qc tag allocator in libata.  That one is just borderline
stupid to keep around on any setup which is supposed to be scalable.

Thanks.

-- 
tejun

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Tejun Heo]

Yo,

On Fri, Jul 26, 2013 at 05:43:13PM -0700, Nicholas A. Bellinger wrote:
> Considering there can be more than a single ata_device hanging off each
> ata_port, the '*sdev = ap->link.device[0].sdev' in __ata_qc_from_tag()
> is definitely bogus, but I'm not sure how else to correlate
> blk-mq/scsi-mq per device descriptors to existing code expecting
> ata_port->qcmd[] descriptors to be shared across multiple devices..
> 
> Tejun..?

I have no idea.  Let's please just do simpler conversion and worry
about embedding qc's into scsi_cmnds later.  libata isn't a normal
SCSI driver and has a rather its own thick midlayer doing the
impedance matching inbetween && I really don't think there is too much
benefit to be reaped from embedding qc's into scsi_cmnds.

Thanks.

-- 
tejun

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Jens Axboe]

On 07/29/2013 05:46 AM, Tejun Heo wrote:
> Hello,
> 
> On Fri, Jul 26, 2013 at 02:14:36PM -0700, Nicholas A. Bellinger wrote:
>> So I don't think (completely) getting rid of ata_port->qcmds[] will be
>> possible, and just converting the ata_scsi_queuecmd() path to use the
>> extra SHT->cmd_size pre-allocation for *ata_queued_cmd might end up
>> being more trouble that it's worth.  Still undecided on that part..
>>
>> Tejun, do you have any thoughts + input here..?
> 
> libata exception handling which includes probing doesn't go through
> SCSI at all.  It all works inside libata proper using ata_queuecmds
> and only the result is exposed to SCSI.  Most of those SCSI semantics
> need to be emulated anyway, so this makes things a lot easier than
> going through SCSI for each command.  As it currently stands, it'd be
> a lot of effort to try to embed ata_qc's into higher layer construct.
> Given how it's used, I don't think it's a high priority task.
> 
> One thing which would probably be worthwhile tho is getting rid of the
> bitmap based qc tag allocator in libata.  That one is just borderline
> stupid to keep around on any setup which is supposed to be scalable.

Your border might be wider than mine :-). Yes, the bitmap should
definitely go.

-- 
Jens Axboe

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Jens Axboe]

On 07/29/2013 05:18 AM, Alexander Gordeev wrote:
>> -static struct ata_queued_cmd *ata_qc_new(struct ata_port *ap)
>> +static struct ata_queued_cmd *ata_qc_new(struct ata_port *ap,
>> +                                        struct scsi_cmnd *sc)
>>  {
>>         struct ata_queued_cmd *qc = NULL;
>> -       unsigned int i;
>> +       struct request *rq = sc->request;
>>  
>>         /* no command while frozen */
>>         if (unlikely(ap->pflags & ATA_PFLAG_FROZEN))
>>                 return NULL;
>>  
>> -       /* the last tag is reserved for internal command. */
>> -       for (i = 0; i < ATA_MAX_QUEUE - 1; i++)
> 
> blk-mq does not prevent tag ATA_TAG_INTERNAL from being using. Would it make
> sense to promote queue depth of length (ATA_MAX_QUEUE - 1) while always
> pointing ATA_TAG_INTERNAL to qcmd (see below)?

blk-mq does support a number of reserved tags, information just needs to
be passed in appropriately. So there is support for reserving X number
of error handling / emergency tags.

-- 
Jens Axboe

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Nicholas A. Bellinger]

On Mon, 2013-07-29 at 07:50 -0400, Tejun Heo wrote:
> Yo,
> 
> On Fri, Jul 26, 2013 at 05:43:13PM -0700, Nicholas A. Bellinger wrote:
> > Considering there can be more than a single ata_device hanging off each
> > ata_port, the '*sdev = ap->link.device[0].sdev' in __ata_qc_from_tag()
> > is definitely bogus, but I'm not sure how else to correlate
> > blk-mq/scsi-mq per device descriptors to existing code expecting
> > ata_port->qcmd[] descriptors to be shared across multiple devices..
> > 
> > Tejun..?
> 
> I have no idea.  Let's please just do simpler conversion and worry
> about embedding qc's into scsi_cmnds later.  libata isn't a normal
> SCSI driver and has a rather its own thick midlayer doing the
> impedance matching inbetween && I really don't think there is too much
> benefit to be reaped from embedding qc's into scsi_cmnds.
> 

That is essentially the same conclusion that I came to, but wanted to at
least give you a chance to comment here.  ;)

So that said, I'll include a simple conversion for libata into the
scsi-mq WIP branch, and folks who are interested in more detailed
conversions can pursue them as separate items.

--nab 

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Nicholas A. Bellinger]

On Mon, 2013-07-29 at 13:18 +0200, Alexander Gordeev wrote:
> On Fri, Jul 26, 2013 at 05:43:13PM -0700, Nicholas A. Bellinger wrote:
> > On Fri, 2013-07-26 at 14:14 -0700, Nicholas A. Bellinger wrote:

<SNIP>

> I also tried to make a "quick" conversion and hit the same issue(s) as you.
> Generally, I am concerned with these assumptions in such approach:
> 
> 1. While libata concept of tags matches nicely with blk-mq (blk_mq_hw_ctx::
> rqs[] vs ata_port::qcmd[]) right now, it is too exposed to changes in blk-mq
> in the long run. I.e. ata_link::sactive limits tags to indices, while tags
> might become hashes. Easily fixable, but still.
> 
> 2. Unallocated requests in blk-mq are accessed/analized from libata-eh.c as
> result of such iterations:
> 
>         for (tag = 0; tag < ATA_MAX_QUEUE; tag++) {
>                 qc = __ata_qc_from_tag(ap, tag);
> 
>                 if (!(qc->flags & ATA_QCFLAG_FAILED))
>                         continue;
> 
> 		...
> 	}
> 
> While it is probably okay right now, it is still based on a premise that
> blk-mq will not change the contents/concept of "payload", i.e. from embedded
> to (re-)allocated memory.
> 
> > The thing that I'm hung up on now for existing __ata_qc_from_tag() usage
> > outside of the main blk_mq_ops->queue_rq -> SHT->queuecommand_mq()
> > dispatch path, is how to actually locate the underlying scsi_device ->
> > request_queue -> blk_mq_ctx -> blk_mq_hw_hctx from the passed
> > ata_port..?
> 
> I am actually in favor of getting rid of ata_queued_cmd::tag. Converting
> ata_link::sactive to a list, making ata_link::active_tag as struct
> ata_queued_cmd *ata_link::active_qc and converting ata_port::qc_allocated to a
> list seems solves it all, including [2]. Have not checked it though.
> 
> Anyway, if we need a blk-mq tag (why?), we have qc->scsicmd->request->tag.
> 

Hi Alexander,

So given the feedback from Tejun, I'm going to setup back for the moment
from a larger conversion, and keep the SHT->cmd_size=0 setting for
libata in the scsi-mq WIP branch.

I'm happy to accept patches to drop the bitmap piece that Tejun
mentioned if your interested, but at least on my end right now there are
bigger fish to fry for scsi-mq.  ;)

Thanks,

--nab

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Marc C]

>> One thing which would probably be worthwhile tho is getting rid of the
>> bitmap based qc tag allocator in libata.  That one is just borderline
>> stupid to keep around on any setup which is supposed to be scalable.
> Your border might be wider than mine :-). Yes, the bitmap should
> definitely go.
A naive implementation is obviously less-than-efficient. However, what
other problems exist with the libata QC tag allocator? I highly doubt
SATA will change to beyond 32 queue tags, primarily because it would
be a pain to change SDB FIS (it's likely to break the dozens of AHCI
controller implementations out there). Further, it seems like the
industry stopped caring about SATA and is pushing NVMe for future
offerings.

In any event, most modern systems should have instructions to count
leading zeroes and modify bits atomically.

-MC

On Mon, Jul 29, 2013 at 12:19 PM, Nicholas A. Bellinger
<nab@linux-iscsi.org> wrote:
> On Mon, 2013-07-29 at 13:18 +0200, Alexander Gordeev wrote:
>> On Fri, Jul 26, 2013 at 05:43:13PM -0700, Nicholas A. Bellinger wrote:
>> > On Fri, 2013-07-26 at 14:14 -0700, Nicholas A. Bellinger wrote:
>
> <SNIP>
>
>> I also tried to make a "quick" conversion and hit the same issue(s) as you.
>> Generally, I am concerned with these assumptions in such approach:
>>
>> 1. While libata concept of tags matches nicely with blk-mq (blk_mq_hw_ctx::
>> rqs[] vs ata_port::qcmd[]) right now, it is too exposed to changes in blk-mq
>> in the long run. I.e. ata_link::sactive limits tags to indices, while tags
>> might become hashes. Easily fixable, but still.
>>
>> 2. Unallocated requests in blk-mq are accessed/analized from libata-eh.c as
>> result of such iterations:
>>
>>         for (tag = 0; tag < ATA_MAX_QUEUE; tag++) {
>>                 qc = __ata_qc_from_tag(ap, tag);
>>
>>                 if (!(qc->flags & ATA_QCFLAG_FAILED))
>>                         continue;
>>
>>               ...
>>       }
>>
>> While it is probably okay right now, it is still based on a premise that
>> blk-mq will not change the contents/concept of "payload", i.e. from embedded
>> to (re-)allocated memory.
>>
>> > The thing that I'm hung up on now for existing __ata_qc_from_tag() usage
>> > outside of the main blk_mq_ops->queue_rq -> SHT->queuecommand_mq()
>> > dispatch path, is how to actually locate the underlying scsi_device ->
>> > request_queue -> blk_mq_ctx -> blk_mq_hw_hctx from the passed
>> > ata_port..?
>>
>> I am actually in favor of getting rid of ata_queued_cmd::tag. Converting
>> ata_link::sactive to a list, making ata_link::active_tag as struct
>> ata_queued_cmd *ata_link::active_qc and converting ata_port::qc_allocated to a
>> list seems solves it all, including [2]. Have not checked it though.
>>
>> Anyway, if we need a blk-mq tag (why?), we have qc->scsicmd->request->tag.
>>
>
> Hi Alexander,
>
> So given the feedback from Tejun, I'm going to setup back for the moment
> from a larger conversion, and keep the SHT->cmd_size=0 setting for
> libata in the scsi-mq WIP branch.
>
> I'm happy to accept patches to drop the bitmap piece that Tejun
> mentioned if your interested, but at least on my end right now there are
> bigger fish to fry for scsi-mq.  ;)
>
> Thanks,
>
> --nab
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ide" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Tejun Heo]

Hello,

On Tue, Jul 30, 2013 at 09:16:02PM -0700, Marc C wrote:
> >> One thing which would probably be worthwhile tho is getting rid of the
> >> bitmap based qc tag allocator in libata.  That one is just borderline
> >> stupid to keep around on any setup which is supposed to be scalable.
> > Your border might be wider than mine :-). Yes, the bitmap should
> > definitely go.
>
> A naive implementation is obviously less-than-efficient. However, what
> other problems exist with the libata QC tag allocator? I highly doubt
> SATA will change to beyond 32 queue tags, primarily because it would
> be a pain to change SDB FIS (it's likely to break the dozens of AHCI
> controller implementations out there). Further, it seems like the
> industry stopped caring about SATA and is pushing NVMe for future
> offerings.
> 
> In any event, most modern systems should have instructions to count
> leading zeroes and modify bits atomically.

It's inefficient not because scanning is expensive but because it
makes all CPUs in the system to hit on the exact same cacheline over
and over and over again.

Thanks.

-- 
tejun

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

On Thu, Jul 25, 2013 at 12:16:41PM +0200, Alexander Gordeev wrote:
> On Mon, Jul 22, 2013 at 02:10:36PM -0700, Nicholas A. Bellinger wrote:
> > Np.  FYI, you'll want to use the latest commit e7827b351 HEAD from
> > target-pending/scsi-mq, which now has functioning scsi-generic support.
> 
> Survives a boot, a kernel build and the build's result :)

Not that rosy. Turned out the old code is called. Hangs with this hunk..

diff --git a/drivers/ata/ata_piix.c b/drivers/ata/ata_piix.c
index ac05cd6..a75fd41 100644
--- a/drivers/ata/ata_piix.c
+++ b/drivers/ata/ata_piix.c
@@ -1103,6 +1103,8 @@ static struct device_attribute *piix_sidpr_shost_attrs[] = {
 static struct scsi_host_template piix_sidpr_sht = {
 	ATA_BMDMA_SHT(DRV_NAME),
 	.shost_attrs		= piix_sidpr_shost_attrs,
+	.scsi_mq		= true,
+	.queuecommand_mq	= ata_scsi_queuecmd,
 };
 
 static struct ata_port_operations piix_sidpr_sata_ops = {


-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

On Mon, Jul 29, 2013 at 07:46:53AM -0400, Tejun Heo wrote:
> One thing which would probably be worthwhile tho is getting rid of the
> bitmap based qc tag allocator in libata.  That one is just borderline
> stupid to keep around on any setup which is supposed to be scalable.

Hi Tejun,

How about this approach?

diff --git a/drivers/ata/libata-core.c b/drivers/ata/libata-core.c
index f218427..5c2a236 100644
--- a/drivers/ata/libata-core.c
+++ b/drivers/ata/libata-core.c
@@ -72,6 +72,8 @@
 #include "libata.h"
 #include "libata-transport.h"
 
+#include "../../block/blk-mq-tag.h"
+
 /* debounce timing parameters in msecs { interval, duration, timeout } */
 const unsigned long sata_deb_timing_normal[]		= {   5,  100, 2000 };
 const unsigned long sata_deb_timing_hotplug[]		= {  25,  500, 2000 };
@@ -1569,18 +1571,8 @@ unsigned ata_exec_internal_sg(struct ata_device *dev,
 
 	/* initialize internal qc */
 
-	/* XXX: Tag 0 is used for drivers with legacy EH as some
-	 * drivers choke if any other tag is given.  This breaks
-	 * ata_tag_internal() test for those drivers.  Don't use new
-	 * EH stuff without converting to it.
-	 */
-	if (ap->ops->error_handler)
-		tag = ATA_TAG_INTERNAL;
-	else
-		tag = 0;
-
-	if (test_and_set_bit(tag, &ap->qc_allocated))
-		BUG();
+	tag = blk_mq_get_tag(ap->qc_tags, GFP_ATOMIC, true);
+	BUG_ON(!ata_tag_internal(tag));
 	qc = __ata_qc_from_tag(ap, tag);
 
 	qc->tag = tag;
@@ -4733,21 +4725,17 @@ void swap_buf_le16(u16 *buf, unsigned int buf_words)
 static struct ata_queued_cmd *ata_qc_new(struct ata_port *ap)
 {
 	struct ata_queued_cmd *qc = NULL;
-	unsigned int i;
+	unsigned int tag;
 
 	/* no command while frozen */
 	if (unlikely(ap->pflags & ATA_PFLAG_FROZEN))
 		return NULL;
 
-	/* the last tag is reserved for internal command. */
-	for (i = 0; i < ATA_MAX_QUEUE - 1; i++)
-		if (!test_and_set_bit(i, &ap->qc_allocated)) {
-			qc = __ata_qc_from_tag(ap, i);
-			break;
-		}
+	tag = blk_mq_get_tag(ap->qc_tags, GFP_ATOMIC, false);
+	qc = __ata_qc_from_tag(ap, tag);
 
 	if (qc)
-		qc->tag = i;
+		qc->tag = tag;
 
 	return qc;
 }
@@ -4799,7 +4787,7 @@ void ata_qc_free(struct ata_queued_cmd *qc)
 	tag = qc->tag;
 	if (likely(ata_tag_valid(tag))) {
 		qc->tag = ATA_TAG_POISON;
-		clear_bit(tag, &ap->qc_allocated);
+		blk_mq_put_tag(ap->qc_tags, tag);
 	}
 }
 
@@ -5639,6 +5627,12 @@ struct ata_port *ata_port_alloc(struct ata_host *host)
 	if (!ap)
 		return NULL;
 
+	ap->qc_tags = blk_mq_init_tags(ATA_MAX_QUEUE, 1, NUMA_NO_NODE);
+	if (!ap->qc_tags) {
+		kfree(ap);
+		return NULL;
+	}
+
 	ap->pflags |= ATA_PFLAG_INITIALIZING | ATA_PFLAG_FROZEN;
 	ap->lock = &host->lock;
 	ap->print_id = -1;
@@ -5677,6 +5671,14 @@ struct ata_port *ata_port_alloc(struct ata_host *host)
 	return ap;
 }
 
+static void ata_port_free(struct ata_port *ap)
+{
+	kfree(ap->pmp_link);
+	kfree(ap->slave_link);
+	blk_mq_free_tags(ap->qc_tags);
+	kfree(ap);
+}
+
 static void ata_host_release(struct device *gendev, void *res)
 {
 	struct ata_host *host = dev_get_drvdata(gendev);
@@ -5691,9 +5693,7 @@ static void ata_host_release(struct device *gendev, void *res)
 		if (ap->scsi_host)
 			scsi_host_put(ap->scsi_host);
 
-		kfree(ap->pmp_link);
-		kfree(ap->slave_link);
-		kfree(ap);
+		ata_port_free(ap);
 		host->ports[i] = NULL;
 	}
 
diff --git a/include/linux/libata.h b/include/linux/libata.h
index eae7a05..4ff9494 100644
--- a/include/linux/libata.h
+++ b/include/linux/libata.h
@@ -126,7 +126,7 @@ enum {
 	ATA_DEF_QUEUE		= 1,
 	/* tag ATA_MAX_QUEUE - 1 is reserved for internal commands */
 	ATA_MAX_QUEUE		= 32,
-	ATA_TAG_INTERNAL	= ATA_MAX_QUEUE - 1,
+	ATA_TAG_INTERNAL	= 0,
 	ATA_SHORT_PAUSE		= 16,
 
 	ATAPI_MAX_DRAIN		= 16 << 10,
@@ -766,7 +766,7 @@ struct ata_port {
 	unsigned int		cbl;	/* cable type; ATA_CBL_xxx */
 
 	struct ata_queued_cmd	qcmd[ATA_MAX_QUEUE];
-	unsigned long		qc_allocated;
+	struct blk_mq_tags	*qc_tags;
 	unsigned int		qc_active;
 	int			nr_active_links; /* #links with active qcs */
 

> Thanks.
> 
> -- 
> tejun

-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Tejun Heo]

On Fri, Aug 09, 2013 at 10:23:35AM +0200, Alexander Gordeev wrote:
> On Mon, Jul 29, 2013 at 07:46:53AM -0400, Tejun Heo wrote:
> > One thing which would probably be worthwhile tho is getting rid of the
> > bitmap based qc tag allocator in libata.  That one is just borderline
> > stupid to keep around on any setup which is supposed to be scalable.
> 
> Hi Tejun,
> 
> How about this approach?

Haven't looked at it thoroughly and I still don't know anything about
blk-mq but it looks good on the first glance.

Thanks.

-- 
tejun

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Jens Axboe]

On 08/09/2013 02:23 AM, Alexander Gordeev wrote:
> On Mon, Jul 29, 2013 at 07:46:53AM -0400, Tejun Heo wrote:
>> One thing which would probably be worthwhile tho is getting rid of the
>> bitmap based qc tag allocator in libata.  That one is just borderline
>> stupid to keep around on any setup which is supposed to be scalable.
> 
> Hi Tejun,
> 
> How about this approach?
> 
> @@ -5639,6 +5627,12 @@ struct ata_port *ata_port_alloc(struct ata_host *host)
>  	if (!ap)
>  		return NULL;
>  
> +	ap->qc_tags = blk_mq_init_tags(ATA_MAX_QUEUE, 1, NUMA_NO_NODE);
> +	if (!ap->qc_tags) {
> +		kfree(ap);
> +		return NULL;
> +	}

This should be blk_mq_init_tags(ATA_MAX_QUEUE - 1, 1, ...) since the
total depth is normal_tags + reserved_tags. Apart from that, I think it
looks alright based on a cursory look.

-- 
Jens Axboe

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

On Fri, Aug 09, 2013 at 08:24:38AM -0600, Jens Axboe wrote:
> On 08/09/2013 02:23 AM, Alexander Gordeev wrote:
> > +	ap->qc_tags = blk_mq_init_tags(ATA_MAX_QUEUE, 1, NUMA_NO_NODE);
> > +	if (!ap->qc_tags) {
> > +		kfree(ap);
> > +		return NULL;
> > +	}
> 
> This should be blk_mq_init_tags(ATA_MAX_QUEUE - 1, 1, ...) since the
> total depth is normal_tags + reserved_tags.

Aha.. If blk_mq_init_tags() should be like this then?

diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c
index dcbc2a4..b131a48 100644
--- a/block/blk-mq-tag.c
+++ b/block/blk-mq-tag.c
@@ -468,10 +468,9 @@ struct blk_mq_tags *blk_mq_init_tags(unsigned int nr_tags,
 	 * Rest of the tags start at the queue list
 	 */
 	tags->nr_free = 0;
-	while (nr_tags - tags->nr_reserved) {
+	while (nr_tags--) {
 		tags->freelist[tags->nr_free] = tags->nr_free +
 							tags->nr_reserved;
-		nr_tags--;
 		tags->nr_free++;
 	}

> -- 
> Jens Axboe

-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Jens Axboe]

On 08/09/2013 09:07 AM, Alexander Gordeev wrote:
> On Fri, Aug 09, 2013 at 08:24:38AM -0600, Jens Axboe wrote:
>> On 08/09/2013 02:23 AM, Alexander Gordeev wrote:
>>> +	ap->qc_tags = blk_mq_init_tags(ATA_MAX_QUEUE, 1, NUMA_NO_NODE);
>>> +	if (!ap->qc_tags) {
>>> +		kfree(ap);
>>> +		return NULL;
>>> +	}
>>
>> This should be blk_mq_init_tags(ATA_MAX_QUEUE - 1, 1, ...) since the
>> total depth is normal_tags + reserved_tags.
> 
> Aha.. If blk_mq_init_tags() should be like this then?
> 
> diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c
> index dcbc2a4..b131a48 100644
> --- a/block/blk-mq-tag.c
> +++ b/block/blk-mq-tag.c
> @@ -468,10 +468,9 @@ struct blk_mq_tags *blk_mq_init_tags(unsigned int nr_tags,
>  	 * Rest of the tags start at the queue list
>  	 */
>  	tags->nr_free = 0;
> -	while (nr_tags - tags->nr_reserved) {
> +	while (nr_tags--) {
>  		tags->freelist[tags->nr_free] = tags->nr_free +
>  							tags->nr_reserved;
> -		nr_tags--;
>  		tags->nr_free++;
>  	}

I misremembered, just checked the code. I think I used to have it like I
described, but changed it since I thought it would be more logical to
pass in full depth, and then what part of that is reserved. Looking at
the current code, your patch looks correct as-is.

-- 
Jens Axboe

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

On Fri, Aug 09, 2013 at 09:52:19AM -0600, Jens Axboe wrote:
> On 08/09/2013 09:07 AM, Alexander Gordeev wrote:
> > diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c
> > index dcbc2a4..b131a48 100644
> > --- a/block/blk-mq-tag.c
> > +++ b/block/blk-mq-tag.c
> > @@ -468,10 +468,9 @@ struct blk_mq_tags *blk_mq_init_tags(unsigned int nr_tags,
> >  	 * Rest of the tags start at the queue list
> >  	 */
> >  	tags->nr_free = 0;
> > -	while (nr_tags - tags->nr_reserved) {
> > +	while (nr_tags--) {
> >  		tags->freelist[tags->nr_free] = tags->nr_free +
> >  							tags->nr_reserved;
> > -		nr_tags--;
> >  		tags->nr_free++;
> >  	}
> 
> I misremembered, just checked the code. I think I used to have it like I
> described, but changed it since I thought it would be more logical to
> pass in full depth, and then what part of that is reserved. Looking at
> the current code, your patch looks correct as-is.

Ok, then a whole series "[PATCH 0/3] blk-mq: Avoid effects of a weird queue
depth" (I posted earlier in a separate thread) should make sense. Besides
the hunk above it limits the per-cpu cache size and sanity-checks total vs
reserved length. I can resubmit if you want.

> 
> -- 
> Jens Axboe
> 

-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Jens Axboe]

On 08/09/2013 10:46 AM, Alexander Gordeev wrote:
> On Fri, Aug 09, 2013 at 09:52:19AM -0600, Jens Axboe wrote:
>> On 08/09/2013 09:07 AM, Alexander Gordeev wrote:
>>> diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c
>>> index dcbc2a4..b131a48 100644
>>> --- a/block/blk-mq-tag.c
>>> +++ b/block/blk-mq-tag.c
>>> @@ -468,10 +468,9 @@ struct blk_mq_tags *blk_mq_init_tags(unsigned int nr_tags,
>>>  	 * Rest of the tags start at the queue list
>>>  	 */
>>>  	tags->nr_free = 0;
>>> -	while (nr_tags - tags->nr_reserved) {
>>> +	while (nr_tags--) {
>>>  		tags->freelist[tags->nr_free] = tags->nr_free +
>>>  							tags->nr_reserved;
>>> -		nr_tags--;
>>>  		tags->nr_free++;
>>>  	}
>>
>> I misremembered, just checked the code. I think I used to have it like I
>> described, but changed it since I thought it would be more logical to
>> pass in full depth, and then what part of that is reserved. Looking at
>> the current code, your patch looks correct as-is.
> 
> Ok, then a whole series "[PATCH 0/3] blk-mq: Avoid effects of a weird queue
> depth" (I posted earlier in a separate thread) should make sense. Besides
> the hunk above it limits the per-cpu cache size and sanity-checks total vs
> reserved length. I can resubmit if you want.

You don't have to resubmit, I'll get it reviewed and applied today.

-- 
Jens Axboe

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

On Sat, Jul 20, 2013 at 09:48:28AM -0500, Mike Christie wrote:
> What about the attached only compile tested patch. The patch has the mq
> block code work like the non mq code for bio cleanups.

Not sure if it is related to the patch or not, but it never returns from
wait_for_completion_io(&wait) in blkdev_issue_flush():

# ps axl | awk '$10 ~ /D\+/'
4     0   938   879  20   0 111216   656 blkdev D+   pts/1      0:00 fdisk/dev/sda
#
# cat /proc/938/stack
[<ffffffff812a8a5c>] blkdev_issue_flush+0xfc/0x160
[<ffffffff811ac606>] blkdev_fsync+0x96/0xc0
[<ffffffff811a2f4d>] do_fsync+0x5d/0x90
[<ffffffff811a3330>] SyS_fsync+0x10/0x20
[<ffffffff81611582>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff

Any ideas?

Thanks!

-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Nicholas A. Bellinger]

On Fri, 2013-08-09 at 21:15 +0200, Alexander Gordeev wrote:
> On Sat, Jul 20, 2013 at 09:48:28AM -0500, Mike Christie wrote:
> > What about the attached only compile tested patch. The patch has the mq
> > block code work like the non mq code for bio cleanups.
> 
> Not sure if it is related to the patch or not, but it never returns from
> wait_for_completion_io(&wait) in blkdev_issue_flush():
> 
> # ps axl | awk '$10 ~ /D\+/'
> 4     0   938   879  20   0 111216   656 blkdev D+   pts/1      0:00 fdisk/dev/sda
> #
> # cat /proc/938/stack
> [<ffffffff812a8a5c>] blkdev_issue_flush+0xfc/0x160
> [<ffffffff811ac606>] blkdev_fsync+0x96/0xc0
> [<ffffffff811a2f4d>] do_fsync+0x5d/0x90
> [<ffffffff811a3330>] SyS_fsync+0x10/0x20
> [<ffffffff81611582>] system_call_fastpath+0x16/0x1b
> [<ffffffffffffffff>] 0xffffffffffffffff
> 
> Any ideas?
> 

Mmmm, I'm able to reproduce over here with ahci + scsi-mq, and it
appears to be a bug related with using sdev->sdev_md_req.queue_depth=1,
that ends up causing the blkdev_issue_flush() to wait forever because
blk_mq_wait_for_tags() never ends up getting the single tag back for the
WRITE_FLUSH bio -> SYNCHRONIZE_CACHE cdb.

Here's the echo w > /proc/sysrq-trigger output:

[  282.620140] SysRq : Show Blocked State
[  282.620958]   task                        PC stack   pid father
[  282.622228] kworker/2:1H    D 0000000000000002     0   532      2 0x00000000
[  282.623607] Workqueue: kblockd mq_flush_work
[  282.624027]  ffff880037869c98 0000000000000046 ffff880037868010 0000000000011380
[  282.624027]  ffff88007d255910 0000000000011380 ffff880037869fd8 0000000000011380
[  282.624027]  ffff880037869fd8 0000000000011380 ffff88007d06f0d0 ffff88007d255910
[  282.624027] Call Trace:
[  282.624027]  [<ffffffff8125b4fd>] ? do_rw_taskfile+0x2ab/0x2bf
[  282.624027]  [<ffffffff810235c4>] ? kvm_clock_read+0x1f/0x21
[  282.624027]  [<ffffffff81054fbc>] ? update_curr+0x4f/0xcd
[  282.624027]  [<ffffffff810235c4>] ? kvm_clock_read+0x1f/0x21
[  282.624027]  [<ffffffff810235cf>] ? kvm_clock_get_cycles+0x9/0xb
[  282.624027]  [<ffffffff81383946>] schedule+0x5f/0x61
[  282.624027]  [<ffffffff813839cf>] io_schedule+0x87/0xca
[  282.624027]  [<ffffffff81192402>] wait_on_tags+0x10f/0x146
[  282.624027]  [<ffffffff81192462>] blk_mq_wait_for_tags+0x29/0x3b
[  282.624027]  [<ffffffff8119132d>] blk_mq_alloc_request_pinned+0xcf/0xe5
[  282.624027]  [<ffffffff811913a6>] blk_mq_alloc_request+0x2d/0x34
[  282.624027]  [<ffffffff8118c60f>] mq_flush_work+0x1a/0x3d
[  282.624027]  [<ffffffff8104474b>] process_one_work+0x257/0x368
[  282.624027]  [<ffffffff81044a4a>] worker_thread+0x1ee/0x34b
[  282.624027]  [<ffffffff8104485c>] ? process_one_work+0x368/0x368
[  282.624027]  [<ffffffff81049771>] kthread+0xb0/0xb8
[  282.624027]  [<ffffffff810496c1>] ? kthread_freezable_should_stop+0x60/0x60
[  282.624027]  [<ffffffff8138a07c>] ret_from_fork+0x7c/0xb0
[  282.624027]  [<ffffffff810496c1>] ? kthread_freezable_should_stop+0x60/0x60
[  282.624027] fdisk           D 0000000000000002     0  1947   1930 0x00000000
[  282.624027]  ffff880037bd9d48 0000000000000082 ffff880037bd8010 0000000000011380
[  282.624027]  ffff88007ca223a0 0000000000011380 ffff880037bd9fd8 0000000000011380
[  282.624027]  ffff880037bd9fd8 0000000000011380 ffff88007d06bb60 ffff88007ca223a0
[  282.624027] Call Trace:
[  282.624027]  [<ffffffff813835e8>] ? __schedule+0x687/0x726
[  282.624027]  [<ffffffff81383946>] schedule+0x5f/0x61
[  282.624027]  [<ffffffff81381d18>] schedule_timeout+0x24/0x183
[  282.624027]  [<ffffffff810235c4>] ? kvm_clock_read+0x1f/0x21
[  282.624027]  [<ffffffff810235cf>] ? kvm_clock_get_cycles+0x9/0xb
[  282.624027]  [<ffffffff8105d2bd>] ? ktime_get_ts+0x53/0xc7
[  282.624027]  [<ffffffff81382e01>] io_schedule_timeout+0x93/0xe4
[  282.624027]  [<ffffffff81051fc7>] ? __cond_resched+0x25/0x31
[  282.624027]  [<ffffffff81383c46>] T.1554+0x8e/0xfc
[  282.624027]  [<ffffffff81054159>] ? try_to_wake_up+0x222/0x222
[  282.624027]  [<ffffffff81383cc7>] wait_for_completion_io+0x13/0x15
[  282.624027]  [<ffffffff8118cbde>] blkdev_issue_flush+0xfb/0x145
[  282.624027]  [<ffffffff810f067a>] blkdev_fsync+0x30/0x3d
[  282.624027]  [<ffffffff810e9259>] vfs_fsync_range+0x18/0x21
[  282.624027]  [<ffffffff810e9279>] vfs_fsync+0x17/0x19
[  282.624027]  [<ffffffff810e942e>] do_fsync+0x35/0x53
[  282.624027]  [<ffffffff810d5574>] ? SyS_ioctl+0x47/0x69
[  282.624027]  [<ffffffff810e9469>] SyS_fsync+0xb/0xf
[  282.624027]  [<ffffffff8138a129>] system_call_fastpath+0x16/0x1b
[  282.624027] blkid           D 0000000000000001     0  1952      1 0x00000000
[  282.679428]  ffff8800371638a8 0000000000000082 ffff880037162010 0000000000011380
[  282.679428]  ffff88007ca205f0 0000000000011380 ffff880037163fd8 0000000000011380
[  282.679428]  ffff880037163fd8 0000000000011380 ffff88007d06b570 ffff88007ca205f0
[  282.679428] Call Trace:
[  282.679428]  [<ffffffff810677a9>] ? generic_exec_single+0x75/0x93
[  282.679428]  [<ffffffff8119212a>] ? blk_mq_tag_busy_iter+0x116/0x116
[  282.679428]  [<ffffffff8106797f>] ? smp_call_function_single+0xf9/0x111
[  282.679428]  [<ffffffff81383946>] schedule+0x5f/0x61
[  282.679428]  [<ffffffff813839cf>] io_schedule+0x87/0xca
[  282.679428]  [<ffffffff81192402>] wait_on_tags+0x10f/0x146
[  282.679428]  [<ffffffff81192462>] blk_mq_wait_for_tags+0x29/0x3b
[  282.679428]  [<ffffffff8119132d>] blk_mq_alloc_request_pinned+0xcf/0xe5
[  282.679428]  [<ffffffff811916b9>] blk_mq_make_request+0x14d/0x2dc
[  282.679428]  [<ffffffff810978c4>] ? mempool_alloc_slab+0x10/0x12
[  282.679428]  [<ffffffff8118951e>] generic_make_request+0x9c/0xdf
[  282.679428]  [<ffffffff81189648>] submit_bio+0xe7/0xf2
[  282.679428]  [<ffffffff810eaaeb>] _submit_bh+0x1b0/0x1d3
[  282.679428]  [<ffffffff810eab19>] submit_bh+0xb/0xd
[  282.679428]  [<ffffffff810ed6e5>] block_read_full_page+0x24d/0x26d
[  282.679428]  [<ffffffff810ef905>] ? I_BDEV+0xd/0xd
[  282.679428]  [<ffffffff810a7624>] ? __inc_zone_page_state+0x1e/0x20
[  282.679428]  [<ffffffff81096188>] ? add_to_page_cache_locked+0x78/0xb0
[  282.679428]  [<ffffffff810f04a5>] blkdev_readpage+0x13/0x15
[  282.679428]  [<ffffffff8109de8d>] __do_page_cache_readahead+0x194/0x1d0
[  282.679428]  [<ffffffff81381f41>] ? __wait_on_bit_lock+0x79/0x8a
[  282.679428]  [<ffffffff8109df50>] force_page_cache_readahead+0x67/0x8d
[  282.679428]  [<ffffffff8109e29a>] page_cache_sync_readahead+0x26/0x3a
[  282.679428]  [<ffffffff81097510>] generic_file_aio_read+0x265/0x5cd
[  282.679428]  [<ffffffff810efaae>] blkdev_aio_read+0x57/0x5e
[  282.679428]  [<ffffffff810c6b8d>] do_sync_read+0x79/0x9f
[  282.679428]  [<ffffffff810c7db7>] vfs_read+0xab/0x130
[  282.679428]  [<ffffffff810c7f06>] SyS_read+0x4f/0x79
[  282.679428]  [<ffffffff8138a129>] system_call_fastpath+0x16/0x1b
[  282.679428] blkid           D 0000000000000003     0  1992    927 0x00000000
[  282.679428]  ffff88007848f8a8 0000000000000086 ffff88007848e010 0000000000011380
[  282.679428]  ffff88007d250000 0000000000011380 ffff88007848ffd8 0000000000011380
[  282.679428]  ffff88007848ffd8 0000000000011380 ffff88007d06c150 ffff88007d250000
[  282.679428] Call Trace:
[  282.679428]  [<ffffffff8109b9b6>] ? __alloc_pages_nodemask+0xf7/0x5eb
[  282.679428]  [<ffffffff81383946>] schedule+0x5f/0x61
[  282.679428]  [<ffffffff813839cf>] io_schedule+0x87/0xca
[  282.679428]  [<ffffffff81192402>] wait_on_tags+0x10f/0x146
[  282.679428]  [<ffffffff81192462>] blk_mq_wait_for_tags+0x29/0x3b
[  282.679428]  [<ffffffff8119132d>] blk_mq_alloc_request_pinned+0xcf/0xe5
[  282.679428]  [<ffffffff811916b9>] blk_mq_make_request+0x14d/0x2dc
[  282.679428]  [<ffffffff8118d81f>] ? create_task_io_context+0xa6/0xf5
[  282.679428]  [<ffffffff8118951e>] generic_make_request+0x9c/0xdf
[  282.679428]  [<ffffffff81189648>] submit_bio+0xe7/0xf2
[  282.679428]  [<ffffffff810eaaeb>] _submit_bh+0x1b0/0x1d3
[  282.679428]  [<ffffffff810eab19>] submit_bh+0xb/0xd
[  282.679428]  [<ffffffff810ed6e5>] block_read_full_page+0x24d/0x26d
[  282.679428]  [<ffffffff810ef905>] ? I_BDEV+0xd/0xd
[  282.679428]  [<ffffffff810f04a5>] blkdev_readpage+0x13/0x15
[  282.679428]  [<ffffffff8109de8d>] __do_page_cache_readahead+0x194/0x1d0
[  282.679428]  [<ffffffff8109df50>] force_page_cache_readahead+0x67/0x8d
[  282.679428]  [<ffffffff8109e29a>] page_cache_sync_readahead+0x26/0x3a
[  282.679428]  [<ffffffff81097510>] generic_file_aio_read+0x265/0x5cd
[  282.679428]  [<ffffffff810efaae>] blkdev_aio_read+0x57/0x5e
[  282.679428]  [<ffffffff810c6b8d>] do_sync_read+0x79/0x9f
[  282.679428]  [<ffffffff810c7db7>] vfs_read+0xab/0x130
[  282.679428]  [<ffffffff810c7f06>] SyS_read+0x4f/0x79
[  282.679428]  [<ffffffff8138a129>] system_call_fastpath+0x16/0x1b

Bumping queue_depth=2 seems to work-around the issue, but AFAICT it's a
genuine tag starvation bug with queue_depth=1 and WRITE_FLUSH..

--nab

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

On Fri, Aug 09, 2013 at 11:07:37AM -0600, Jens Axboe wrote:
> You don't have to resubmit, I'll get it reviewed and applied today.

Hi Jens,

I limited the minimal queue depth to 4, which is apparently wrong
in case of libata. I will post a new series.

> -- 
> Jens Axboe
> 

-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

On Fri, Aug 09, 2013 at 01:17:37PM -0700, Nicholas A. Bellinger wrote:
> On Fri, 2013-08-09 at 21:15 +0200, Alexander Gordeev wrote:
> Mmmm, I'm able to reproduce over here with ahci + scsi-mq, and it
> appears to be a bug related with using sdev->sdev_md_req.queue_depth=1,
> that ends up causing the blkdev_issue_flush() to wait forever because
> blk_mq_wait_for_tags() never ends up getting the single tag back for the
> WRITE_FLUSH bio -> SYNCHRONIZE_CACHE cdb.

It turns out this way - blkdev_issue_flush() claims the only tag, submits
the bio and waits for the completion. But because blk_mq_make_request()
does not mark any context in blk_mq_hw_ctx::ctx_map (nor enslists the request
into blk_mq_ctx::rq_list) it never gets processed from blk_mq_work_fn->
__blk_mq_run_hw_queue() and blkdev_issue_flush() waits endlessly. All other
requests are just waiting for the tag availability as result.

[...]

> Bumping queue_depth=2 seems to work-around the issue, but AFAICT it's a
> genuine tag starvation bug with queue_depth=1 and WRITE_FLUSH..

If I try to hack and force __blk_mq_run_hw_queue() to process the request...

diff --git a/block/blk-mq.c b/block/blk-mq.c
index 6fc1df3..c22b6f66 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -889,9 +962,12 @@ static void blk_mq_make_request(struct request_queue *q, struct bio *bio)
 	hctx->queued++;
 
 	if (unlikely(is_flush_fua)) {
+		list_add(&rq->queuelist, &hctx->dispatch);
 		blk_mq_bio_to_request(q, rq, bio);
 		blk_mq_put_ctx(ctx);
 		blk_insert_flush(rq);
 		goto run_queue;
 	}

... I get a kernel BUG at drivers/scsi/scsi_lib.c:1233

	BUG_ON(!req->nr_phys_segments);

IOW I am not sure how to proceed.

> --nab
> 

-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Nicholas A. Bellinger]

On Thu, 2013-08-15 at 18:23 +0200, Alexander Gordeev wrote:
> On Fri, Aug 09, 2013 at 01:17:37PM -0700, Nicholas A. Bellinger wrote:
> > On Fri, 2013-08-09 at 21:15 +0200, Alexander Gordeev wrote:
> > Mmmm, I'm able to reproduce over here with ahci + scsi-mq, and it
> > appears to be a bug related with using sdev->sdev_md_req.queue_depth=1,
> > that ends up causing the blkdev_issue_flush() to wait forever because
> > blk_mq_wait_for_tags() never ends up getting the single tag back for the
> > WRITE_FLUSH bio -> SYNCHRONIZE_CACHE cdb.
> 
> It turns out this way - blkdev_issue_flush() claims the only tag, submits
> the bio and waits for the completion. But because blk_mq_make_request()
> does not mark any context in blk_mq_hw_ctx::ctx_map (nor enslists the request
> into blk_mq_ctx::rq_list) it never gets processed from blk_mq_work_fn->
> __blk_mq_run_hw_queue() and blkdev_issue_flush() waits endlessly. All other
> requests are just waiting for the tag availability as result.
> 

Ok, here's a bit better idea of what is going on now..

The problem is that blkdev_issue_flush() -> blk_mq_make_request() ->
__blk_mq_alloc_request() allocates the first tag, which calls
blk_insert_flush() -> blk_flush_complete_seq() -> blk_flush_kick() ->
mq_flush_work() -> blk_mq_alloc_request() to allocate a second tag for
the struct request that actually gets dispatched into scsi-mq as a
SYCHRONIZE_CACHE command..

I'm not exactly sure why this double tag usage of struct request is
occurring, but AFAICT it does happen for every flush, and is not
specific to the blkdev_issue_flush() codepath..  I'm sure that Jens can
fill us in on that bit.  ;)

So, assuming that this double tag usage is necessary and not a bug,
perhaps using a reserved tag for the first tag (eg: the one that's not
dispatched into scsi_mq_queue_rq) makes sense..?

I'm playing with a patch to do this, but am currently getting hung-up on
what appear to be some separate blk-mq reserved_tags > 0 bugs, the first
of which is passing queue_depth=1 + reserved_tags=1 is broken, and
results in tags->nr_free = 0.

Here's the quick fix:

diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c
index 6718007..ffdf686 100644
--- a/block/blk-mq-tag.c
+++ b/block/blk-mq-tag.c
@@ -470,7 +470,7 @@ struct blk_mq_tags *blk_mq_init_tags(unsigned int nr_tags,
 	 * Rest of the tags start at the queue list
 	 */
 	tags->nr_free = 0;
-	while (nr_tags - tags->nr_reserved) {
+	while (nr_tags) {
 		tags->freelist[tags->nr_free] = tags->nr_free +
 							tags->nr_reserved;
 		nr_tags--;

Anyways, before digging further into reserved tags logic, Jens, what are
your thoughts for addressing this special queue_depth=1 case for libata
+ the like..?

--nab

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

On Thu, Aug 15, 2013 at 07:19:29PM -0700, Nicholas A. Bellinger wrote:
> I'm playing with a patch to do this, but am currently getting hung-up on
> what appear to be some separate blk-mq reserved_tags > 0 bugs, the first
> of which is passing queue_depth=1 + reserved_tags=1 is broken, and
> results in tags->nr_free = 0.

That is not a bug - please look at Jens replies in this thread some week ago.
In short, queue_depth=1 means 1 tags in total and reserved_tags=1 results
in zero normal tags. You need to request the depth=2 and reserved_tags=1.

But yes, this is a separate topic and I am looking forward to hear from Jens
wrt flushes.

-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Nicholas A. Bellinger]

On Fri, 2013-08-16 at 18:41 +0200, Alexander Gordeev wrote:
> On Thu, Aug 15, 2013 at 07:19:29PM -0700, Nicholas A. Bellinger wrote:
> > I'm playing with a patch to do this, but am currently getting hung-up on
> > what appear to be some separate blk-mq reserved_tags > 0 bugs, the first
> > of which is passing queue_depth=1 + reserved_tags=1 is broken, and
> > results in tags->nr_free = 0.
> 
> That is not a bug - please look at Jens replies in this thread some week ago.
> In short, queue_depth=1 means 1 tags in total and reserved_tags=1 results
> in zero normal tags. You need to request the depth=2 and reserved_tags=1.
> 

Ahhh, yes of course.  I'll re-work a proposed patch this afternoon with
this in mind..

> But yes, this is a separate topic and I am looking forward to hear from Jens
> wrt flushes.
> 

Indeed.  ;)

--nab

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

On Thu, Aug 15, 2013 at 07:19:29PM -0700, Nicholas A. Bellinger wrote:
> Anyways, before digging further into reserved tags logic, Jens, what are
> your thoughts for addressing this special queue_depth=1 case for libata
> + the like..?

Hi Jens,

Have some comments?

-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

On Thu, Aug 15, 2013 at 07:19:29PM -0700, Nicholas A. Bellinger wrote:
> Anyways, before digging further into reserved tags logic, Jens, what are
> your thoughts for addressing this special queue_depth=1 case for libata
> + the like..?

Hi Jens,

Have some comments?

-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

On Thu, Aug 15, 2013 at 07:19:29PM -0700, Nicholas A. Bellinger wrote:
> Ok, here's a bit better idea of what is going on now..
> 
> The problem is that blkdev_issue_flush() -> blk_mq_make_request() ->
> __blk_mq_alloc_request() allocates the first tag, which calls
> blk_insert_flush() -> blk_flush_complete_seq() -> blk_flush_kick() ->
> mq_flush_work() -> blk_mq_alloc_request() to allocate a second tag for
> the struct request that actually gets dispatched into scsi-mq as a
> SYCHRONIZE_CACHE command..
> 
> I'm not exactly sure why this double tag usage of struct request is
> occurring, but AFAICT it does happen for every flush, and is not
> specific to the blkdev_issue_flush() codepath..  I'm sure that Jens can
> fill us in on that bit.  ;)

I also played with the double tag using a reserved tag (below).

While it fixes 'fdisk /dev/sda' issue when trying to 'mount /dev/sda1 /mnt'
what appears to be a call to bio->bi_end_io() from the free'd bio hits in.

Not sure if I should pursue the root cause until the whole double-tag
thingy is confirmed.

Jens?


diff --git a/block/blk-mq.c b/block/blk-mq.c
index 6fc1df3..81794dc 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -874,14 +874,14 @@ static void blk_mq_make_request(struct request_queue *q, struct bio *bio)
 	hctx = q->mq_ops->map_queue(q, ctx->cpu);
 
 	trace_block_getrq(q, bio, rw);
-	rq = __blk_mq_alloc_request(hctx, GFP_ATOMIC, false);
+	rq = __blk_mq_alloc_request(hctx, GFP_ATOMIC, is_flush_fua);
 	if (likely(rq))
 		blk_mq_rq_ctx_init(ctx, rq, rw);
 	else {
 		blk_mq_put_ctx(ctx);
 		trace_block_sleeprq(q, bio, rw);
 		rq = blk_mq_alloc_request_pinned(q, rw, __GFP_WAIT|GFP_ATOMIC,
-							false);
+							is_flush_fua);
 		ctx = rq->mq_ctx;
 		hctx = q->mq_ops->map_queue(q, ctx->cpu);
 	}
@@ -1317,6 +1317,9 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_reg *reg,
 		reg->queue_depth = BLK_MQ_MAX_DEPTH;
 	}
 
+	reg->queue_depth++;
+	reg->reserved_tags++;
+
 	ctx = alloc_percpu(struct blk_mq_ctx);
 	if (!ctx)
 		return ERR_PTR(-ENOMEM);

-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

On Thu, Aug 15, 2013 at 07:19:29PM -0700, Nicholas A. Bellinger wrote:
> Ok, here's a bit better idea of what is going on now..
> 
> The problem is that blkdev_issue_flush() -> blk_mq_make_request() ->
> __blk_mq_alloc_request() allocates the first tag, which calls
> blk_insert_flush() -> blk_flush_complete_seq() -> blk_flush_kick() ->
> mq_flush_work() -> blk_mq_alloc_request() to allocate a second tag for
> the struct request that actually gets dispatched into scsi-mq as a
> SYCHRONIZE_CACHE command..
> 
> I'm not exactly sure why this double tag usage of struct request is
> occurring, but AFAICT it does happen for every flush, and is not
> specific to the blkdev_issue_flush() codepath..  I'm sure that Jens can
> fill us in on that bit.  ;)

I also played with the double tag using a reserved tag (below).

While it fixes 'fdisk /dev/sda' issue when trying to 'mount /dev/sda1 /mnt'
what appears to be a call to bio->bi_end_io() from the free'd bio hits in.

Not sure if I should pursue the root cause until the whole double-tag
thingy is confirmed.

Jens?


diff --git a/block/blk-mq.c b/block/blk-mq.c
index 6fc1df3..81794dc 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -874,14 +874,14 @@ static void blk_mq_make_request(struct request_queue *q, struct bio *bio)
 	hctx = q->mq_ops->map_queue(q, ctx->cpu);
 
 	trace_block_getrq(q, bio, rw);
-	rq = __blk_mq_alloc_request(hctx, GFP_ATOMIC, false);
+	rq = __blk_mq_alloc_request(hctx, GFP_ATOMIC, is_flush_fua);
 	if (likely(rq))
 		blk_mq_rq_ctx_init(ctx, rq, rw);
 	else {
 		blk_mq_put_ctx(ctx);
 		trace_block_sleeprq(q, bio, rw);
 		rq = blk_mq_alloc_request_pinned(q, rw, __GFP_WAIT|GFP_ATOMIC,
-							false);
+							is_flush_fua);
 		ctx = rq->mq_ctx;
 		hctx = q->mq_ops->map_queue(q, ctx->cpu);
 	}
@@ -1317,6 +1317,9 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_reg *reg,
 		reg->queue_depth = BLK_MQ_MAX_DEPTH;
 	}
 
+	reg->queue_depth++;
+	reg->reserved_tags++;
+
 	ctx = alloc_percpu(struct blk_mq_ctx);
 	if (!ctx)
 		return ERR_PTR(-ENOMEM);

-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Nicholas A. Bellinger]

Hi Alexander!

Apologies for the long delay on this follow-up..  Comments below.

On Fri, 2013-09-20 at 17:19 +0200, Alexander Gordeev wrote:
> On Thu, Aug 15, 2013 at 07:19:29PM -0700, Nicholas A. Bellinger wrote:
> > Ok, here's a bit better idea of what is going on now..
> > 
> > The problem is that blkdev_issue_flush() -> blk_mq_make_request() ->
> > __blk_mq_alloc_request() allocates the first tag, which calls
> > blk_insert_flush() -> blk_flush_complete_seq() -> blk_flush_kick() ->
> > mq_flush_work() -> blk_mq_alloc_request() to allocate a second tag for
> > the struct request that actually gets dispatched into scsi-mq as a
> > SYCHRONIZE_CACHE command..
> > 
> > I'm not exactly sure why this double tag usage of struct request is
> > occurring, but AFAICT it does happen for every flush, and is not
> > specific to the blkdev_issue_flush() codepath..  I'm sure that Jens can
> > fill us in on that bit.  ;)
> 
> I also played with the double tag using a reserved tag (below).
> 
> While it fixes 'fdisk /dev/sda' issue when trying to 'mount /dev/sda1 /mnt'
> what appears to be a call to bio->bi_end_io() from the free'd bio hits in.
> 
> Not sure if I should pursue the root cause until the whole double-tag
> thingy is confirmed.
> 
> Jens?
> 
> 
> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index 6fc1df3..81794dc 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -874,14 +874,14 @@ static void blk_mq_make_request(struct request_queue *q, struct bio *bio)
>  	hctx = q->mq_ops->map_queue(q, ctx->cpu);
>  
>  	trace_block_getrq(q, bio, rw);
> -	rq = __blk_mq_alloc_request(hctx, GFP_ATOMIC, false);
> +	rq = __blk_mq_alloc_request(hctx, GFP_ATOMIC, is_flush_fua);
>  	if (likely(rq))
>  		blk_mq_rq_ctx_init(ctx, rq, rw);
>  	else {
>  		blk_mq_put_ctx(ctx);
>  		trace_block_sleeprq(q, bio, rw);
>  		rq = blk_mq_alloc_request_pinned(q, rw, __GFP_WAIT|GFP_ATOMIC,
> -							false);
> +							is_flush_fua);
>  		ctx = rq->mq_ctx;
>  		hctx = q->mq_ops->map_queue(q, ctx->cpu);
>  	}

So this is what I ended up doing as well, and does address the specific
bug with queue_depth=1.

> @@ -1317,6 +1317,9 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_reg *reg,
>  		reg->queue_depth = BLK_MQ_MAX_DEPTH;
>  	}
>  
> +	reg->queue_depth++;
> +	reg->reserved_tags++;
> +
>  	ctx = alloc_percpu(struct blk_mq_ctx);
>  	if (!ctx)
>  		return ERR_PTR(-ENOMEM);
> 

I was actually setting this within scsi_mq_alloc_queue(), but given that
the queue_depth=1 issue is independent of scsi-mq, this does make more
sense.

Also, these extra increments should probably happen only when the passed
queue_depth == 1 && reserved_tags == 0.

Other than that minor nit.

Reviewed-by: Nicholas Bellinger <nab@linux-iscsi.org>

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Christoph Hellwig]

On Sat, Jul 20, 2013 at 09:48:28AM -0500, Mike Christie wrote:
> What about the attached only compile tested patch. The patch has the mq
> block code work like the non mq code for bio cleanups.
> 
> 

> blk-mq: blk-mq should free bios in pass through case
> 
> For non mq calls, the block layer will free the bios when
> blk_finish_request is called.
e 
> For mq calls, the blk mq code wants the caller to do this.
> 
> This patch has the blk mq code work like the non mq code
> and has the block layer free the bios.
> 
> Signed-off-by: Mike Christie <michaelc@cs.wisc.edu>

This patch breaks booting for me in the current blk multiqueue tree,
with an apparent double free of a bio when using virtio-blk in writeback
mode (cache=writeback or cache=none in qemu):

[   15.253608] ------------[ cut here ]------------
[   15.256422] kernel BUG at /work/hch/linux/fs/bio.c:498!
[   15.256879] invalid opcode: 0000 [#1] SMP 
[   15.256879] Modules linked in:
[   15.256879] CPU: 3 PID: 353 Comm: kblockd Not tainted 3.11.0+ #25
[   15.256879] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[   15.256879] task: ffff88007d75e0c0 ti: ffff88007d676000 task.ti: ffff88007d676000
[   15.256879] RIP: 0010:[<ffffffff811b470a>]  [<ffffffff811b470a>] bio_put+0x8a/0x90
[   15.256879] RSP: 0018:ffff88007fd83b50  EFLAGS: 00010046
[   15.256879] RAX: 0000000000000000 RBX: ffff88007d713080 RCX: 0000000000000035
[   15.256879] RDX: 0000000000000002 RSI: ffff88007ad50338 RDI: ffff88007d713080
[   15.256879] RBP: ffff88007fd83b60 R08: 7010000000000000 R09: 007ad50338080000
[   15.256879] R10: ff672b1b7d38ce02 R11: 000000000000028b R12: 0000000000000000
[   15.256879] R13: 0000000000000000 R14: ffff88007b4c36c0 R15: ffff88007b40d608
[   15.256879] FS:  0000000000000000(0000) GS:ffff88007fd80000(0000) knlGS:0000000000000000
[   15.256879] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[   15.256879] CR2: 0000000000000138 CR3: 0000000002124000 CR4: 00000000000006e0
[   15.256879] Stack:
[   15.256879]  ffff88007d713080 0000000000000000 ffff88007fd83b80 ffffffff811ae8a3
[   15.256879]  ffff88007fd83bf0 0000000000001000 ffff88007fd83b90 ffffffff811b3268
[   15.256879]  ffff88007fd83bc0 ffffffff816ac847 ffff88007b4c36c0 ffff88007fd99d00
[   15.256879] Call Trace:
[   15.256879]  <IRQ> 
[   15.256879]  [<ffffffff811ae8a3>] end_bio_bh_io_sync+0x33/0x50
[   15.256879]  [<ffffffff811b3268>] bio_endio+0x18/0x30
[   15.256879]  [<ffffffff816ac847>] blk_mq_complete_request+0x47/0xd0
[   15.256879]  [<ffffffff816ac8e9>] __blk_mq_end_io+0x19/0x20
[   15.256879]  [<ffffffff816ac958>] blk_mq_end_io+0x68/0xd0
[   15.256879]  [<ffffffff816a6162>] blk_flush_complete_seq+0xe2/0x370
[   15.256879]  [<ffffffff816a653b>] flush_end_io+0x11b/0x200
[   15.256879]  [<ffffffff816ac875>] blk_mq_complete_request+0x75/0xd0
[   15.256879]  [<ffffffff816ac8e9>] __blk_mq_end_io+0x19/0x20
[   15.256879]  [<ffffffff816ac958>] blk_mq_end_io+0x68/0xd0
[   15.256879]  [<ffffffff81844c2f>] virtblk_done+0xef/0x260
[   15.256879]  [<ffffffff81753cc0>] vring_interrupt+0x30/0x60
[   15.256879]  [<ffffffff81103724>] handle_irq_event_percpu+0x54/0x1f0
[   15.256879]  [<ffffffff81103903>] handle_irq_event+0x43/0x70
[   15.256879]  [<ffffffff8110609f>] handle_edge_irq+0x6f/0x120
[   15.256879]  [<ffffffff810445b8>] handle_irq+0x58/0x140
[   15.256879]  [<ffffffff81094bbf>] ? irq_enter+0x4f/0x90
[   15.256879]  [<ffffffff810440b5>] do_IRQ+0x55/0xd0
[   15.256879]  [<ffffffff81bd3972>] common_interrupt+0x72/0x72
[   15.256879]  [<ffffffff810c5135>] ? sched_clock_local+0x25/0xa0
[   15.256879]  [<ffffffff81094960>] ? __do_softirq+0xb0/0x250
[   15.256879]  [<ffffffff81094959>] ? __do_softirq+0xa9/0x250
[   15.256879]  [<ffffffff81094cae>] irq_exit+0xae/0xd0
[   15.256879]  [<ffffffff8106dcd5>] smp_apic_timer_interrupt+0x45/0x60
[   15.256879]  [<ffffffff81bdc772>] apic_timer_interrupt+0x72/0x80
[   15.256879]  <EOI> 
[   15.256879]  [<ffffffff81bd3a33>] ? retint_restore_args+0x13/0x13
[   15.256879]  [<ffffffff81bd3502>] ? _raw_spin_unlock_irq+0x32/0x40
[   15.256879]  [<ffffffff81bd34fb>] ? _raw_spin_unlock_irq+0x2b/0x40
[   15.256879]  [<ffffffff810ac0c4>] rescuer_thread+0xe4/0x2f0
[   15.256879]  [<ffffffff810abfe0>] ? process_scheduled_works+0x40/0x40
[   15.256879]  [<ffffffff810b3916>] kthread+0xd6/0xe0
[   15.256879]  [<ffffffff81bd34fb>] ? _raw_spin_unlock_irq+0x2b/0x40
[   15.256879]  [<ffffffff810b3840>] ? __init_kthread_worker+0x70/0x70
[   15.256879]  [<ffffffff81bdbabc>] ret_from_fork+0x7c/0xb0
[   15.256879]  [<ffffffff810b3840>] ? __init_kthread_worker+0x70/0x70
[   15.256879] Code: ff 41 8b 44 24 08 48 89 df 49 8b 74 24 10 48 29 c7 e8 cb 88 f8 ff 48 8b 5d f0 4c 8b 65 f8 c9 c3 90 48 89 df e8 b8 5c fc ff eb 9b <0f> 0b 0f 1f 40 00 55 48 89 e5 41 57 45 31 ff 41 56 41 55 41 54 
[   15.256879] RIP  [<ffffffff811b470a>] bio_put+0x8a/0x90
[   15.256879]  RSP <ffff88007fd83b50>
[   15.256879] ---[ end trace 1f201608bfddfca7 ]---
[   15.256879] Kernel panic - not syncing: Fatal exception in interrupt
[   15.256879] Shutting down cpus with NMI

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

On Thu, Oct 03, 2013 at 04:06:51AM -0700, Christoph Hellwig wrote:
> On Sat, Jul 20, 2013 at 09:48:28AM -0500, Mike Christie wrote:
> > What about the attached only compile tested patch. The patch has the mq
> > block code work like the non mq code for bio cleanups.
> > 
> > 
> 
> > blk-mq: blk-mq should free bios in pass through case
> > 
> > For non mq calls, the block layer will free the bios when
> > blk_finish_request is called.
> e 
> > For mq calls, the blk mq code wants the caller to do this.
> > 
> > This patch has the blk mq code work like the non mq code
> > and has the block layer free the bios.
> > 
> > Signed-off-by: Mike Christie <michaelc@cs.wisc.edu>
> 
> This patch breaks booting for me in the current blk multiqueue tree,
> with an apparent double free of a bio when using virtio-blk in writeback
> mode (cache=writeback or cache=none in qemu):

I am not sure if the root cause the same, but the panic I experience with
mounting a ahci device (and those double-tag usage described in another
thread) is somehow similar:


[  181.184510] general protection fault: 0000 [#1] SMP 
[  181.184546] Modules linked in: lockd sunrpc snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_pcm mperf snd_page_alloc i5000_edac coretemp snd_timer edac_core iTCO_wdt snd kvm_intel iTCO_vendor_support lpc_ich mfd_core igb dca i5k_amb ppdev soundcore hp_wmi tg3 kvm sparse_keymap serio_raw ptp microcode pcspkr rfkill pps_core shpchp parport_pc parport mptsas scsi_transport_sas mptscsih mptbase floppy nouveau video mxm_wmi wmi i2c_algo_bit drm_kms_helper ttm drm i2c_core
[  181.184550] CPU: 6 PID: 0 Comm: swapper/6 Tainted: G        W    3.10.0-rc5.debug+ #180
[  181.184552] Hardware name: Hewlett-Packard HP xw6400 Workstation/0A04h, BIOS 786D4 v02.31 03/14/2008
[  181.184554] task: ffff88007b1a8000 ti: ffff88007b19c000 task.ti: ffff88007b19c000
[  181.184563] RIP: 0010:[<ffffffff811fa97b>]  [<ffffffff811fa97b>] bio_endio+0x1b/0x40
[  181.184565] RSP: 0018:ffff88007d203a28  EFLAGS: 00010002
[  181.184567] RAX: 6b6b6b6b6b6b6b6b RBX: 0000000000000000 RCX: dead000000200200
[  181.184568] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880068e29200
[  181.184570] RBP: ffff88007d203a28 R08: ffffe8ffff201240 R09: 0000000000000000
[  181.184571] R10: 0000000000000000 R11: 0000000000000001 R12: ffff880074d86000
[  181.184572] R13: 6b6b6b6b6b6b6b6b R14: 000000006b6b6b6b R15: 0000000000000001
[  181.184575] FS:  0000000000000000(0000) GS:ffff88007d200000(0000) knlGS:0000000000000000
[  181.184576] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  181.184578] CR2: 00007f8afac8c45c CR3: 000000005f431000 CR4: 00000000000007e0
[  181.184580] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  181.184581] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  181.184582] Stack:
[  181.184587]  ffff88007d203a78 ffffffff813202aa ffff88007d203a98 0000000000000046
[  181.184591]  0000000000000046 ffff880072d08000 0000000000000000 ffff880074d860d8
[  181.184594]  0000000000000000 0000000000000001 ffff88007d203a88 ffffffff81320445
[  181.184595] Call Trace:
[  181.184597]  <IRQ> 
[  181.184603]  [<ffffffff813202aa>] blk_mq_complete_request+0x5a/0x1d0
[  181.184607]  [<ffffffff81320445>] __blk_mq_end_io+0x25/0x30
[  181.184609]  [<ffffffff81320535>] blk_mq_end_io+0xe5/0xf0
[  181.184613]  [<ffffffff81319754>] blk_flush_complete_seq+0xf4/0x360
[  181.184616]  [<ffffffff81319a4b>] ? flush_end_io+0x4b/0x210
[  181.184619]  [<ffffffff81319b2a>] flush_end_io+0x12a/0x210
[  181.184622]  [<ffffffff813202da>] blk_mq_complete_request+0x8a/0x1d0
[  181.184626]  [<ffffffff8147b9fd>] ? scsi_device_unbusy+0x9d/0xd0
[  181.184629]  [<ffffffff81320445>] __blk_mq_end_io+0x25/0x30
[  181.184632]  [<ffffffff81320535>] blk_mq_end_io+0xe5/0xf0
[  181.184635]  [<ffffffff8147cae5>] scsi_mq_end_request+0x15/0x20
[  181.184638]  [<ffffffff8147bf20>] scsi_io_completion+0xa0/0x650
[  181.184643]  [<ffffffff810bbc3d>] ? trace_hardirqs_off+0xd/0x10
[  181.184648]  [<ffffffff814722f7>] scsi_finish_command+0x87/0xe0
[  181.184650]  [<ffffffff8147bccf>] scsi_softirq_done+0x13f/0x160
[  181.184653]  [<ffffffff8147cba5>] scsi_mq_done+0x15/0x20
[  181.184658]  [<ffffffff81495a73>] ata_scsi_qc_complete+0x63/0x470
[  181.184661]  [<ffffffff8148fad0>] __ata_qc_complete+0x90/0x140
[  181.184664]  [<ffffffff8148fc1d>] ata_qc_complete+0x9d/0x230
[  181.184667]  [<ffffffff8148fe51>] ata_qc_complete_multiple+0xa1/0xe0
[  181.184673]  [<ffffffff814aa449>] ahci_handle_port_interrupt+0x109/0x560
[  181.184676]  [<ffffffff814ab63f>] ahci_port_intr+0x2f/0x40
[  181.184678]  [<ffffffff814ab6f1>] ahci_interrupt+0xa1/0x100
[  181.184683]  [<ffffffff810ff7b5>] handle_irq_event_percpu+0x75/0x3d0
[  181.184686]  [<ffffffff810ffb58>] handle_irq_event+0x48/0x70
[  181.184689]  [<ffffffff81102d9e>] ? handle_fasteoi_irq+0x1e/0x100
[  181.184692]  [<ffffffff81102dda>] handle_fasteoi_irq+0x5a/0x100
[  181.184696]  [<ffffffff81004320>] handle_irq+0x60/0x150
[  181.184702]  [<ffffffff816ff846>] ? atomic_notifier_call_chain+0x16/0x20
[  181.184706]  [<ffffffff81705f7a>] do_IRQ+0x5a/0xe0
[  181.184710]  [<ffffffff816fb52f>] common_interrupt+0x6f/0x6f
[  181.184712]  <EOI> 
[  181.184716]  [<ffffffff8100aa45>] ? default_idle+0x25/0x280
[  181.184719]  [<ffffffff8100aa43>] ? default_idle+0x23/0x280
[  181.184722]  [<ffffffff8100b4f6>] arch_cpu_idle+0x26/0x30
[  181.184726]  [<ffffffff810afe66>] cpu_startup_entry+0x96/0x3e0
[  181.184729]  [<ffffffff810b7ad5>] ? clockevents_register_device+0xb5/0x120
[  181.184734]  [<ffffffff816e67ea>] start_secondary+0x27a/0x27c
[  181.184767] Code: 47 c1 c1 ea 03 83 c2 01 39 d0 0f 47 c2 c3 66 90 66 66 66 66 90 55 85 f6 48 89 e5 74 1b f0 80 67 18 fe 48 8b 47 40 48 85 c0 74 02 <ff> d0 5d 66 90 c3 0f 1f 80 00 00 00 00 48 8b 47 18 a8 01 b8 fb 
[  181.184770] RIP  [<ffffffff811fa97b>] bio_endio+0x1b/0x40
[  181.184772]  RSP <ffff88007d203a28>
[  181.184777] ---[ end trace 5e8fd083b8562c3c ]---
[  181.184779] Kernel panic - not syncing: Fatal exception in interrupt
[  181.185483] drm_kms_helper: panic occurred, switching back to text console


-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Alexander Gordeev]

On Wed, May 22, 2013 at 08:50:03AM +0900, Tejun Heo wrote:
> Hmmmmmm..... I'd normally apply this patch but block layer is just
> growing multi-queue support and libata is likely to be converted to mq
> in foreseeable future, so I'm a bit hesitant to make irq handling more
> sophiscated right now.  Would you be interested in looking into
> converting libata to blk mq support?  I'm pretty sure it'd yield far
> better outcome if done properly.

Hi Tejun,

As conversion of libata to blk mq has long done I tried the change
against the recent version and the results still appear worthwhile.

The numbers are taken by running 'dd if=/dev/sd{a,b} of=/dev/null'
in parallel. All time values are in us.

Before this update host lock average holdtime was 2.45 and
average waittime was 1.24. After the update average holdtime
dropped to 0.29 (about eight times) while average waittime
decreased to 0.58 (about two times).

Also, port events are handled with local interrupts enabled
and compete on individual per-port locks with average holdtime
1.25 and average waittime 1.48. So combined average holdtime
spent while holding host and port locks decreased from 2.45 to
0.29 + 1.25 = 1.54 (about 1.6 times).

The downside of this change is introduction of a kernel thread.

The upside is shorter access time to port locks and moving port
interrupts handling out of the hardware interrupt context.

Thanks!

> -- 
> tejun

-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

[PATCH v2] AHCI: Optimize interrupt processing

[Alexander Gordeev]

Split interrupt service routine into hardware context handler and
threaded context handler. That allows to protect ports with individual
locks rather than with a single host-wide lock, which results in better
parallelism.

Cc: linux-ide@vger.kernel.org
Signed-off-by: Alexander Gordeev <agordeev@redhat.com>
---
 drivers/ata/acard-ahci.c       |   3 +-
 drivers/ata/ahci.c             | 100 ++++++++++++++++++++++++++---------------
 drivers/ata/ahci.h             |   8 ++--
 drivers/ata/libahci.c          |  74 +++++++++++++++++-------------
 drivers/ata/libahci_platform.c |   3 +-
 5 files changed, 115 insertions(+), 73 deletions(-)

diff --git a/drivers/ata/acard-ahci.c b/drivers/ata/acard-ahci.c
index 25d0ac3..c962886 100644
--- a/drivers/ata/acard-ahci.c
+++ b/drivers/ata/acard-ahci.c
@@ -498,8 +498,7 @@ static int acard_ahci_init_one(struct pci_dev *pdev, const struct pci_device_id
 	acard_ahci_pci_print_info(host);
 
 	pci_set_master(pdev);
-	return ata_host_activate(host, pdev->irq, ahci_interrupt, IRQF_SHARED,
-				 &acard_ahci_sht);
+	return ahci_host_activate(host, pdev->irq, &acard_ahci_sht);
 }
 
 module_pci_driver(acard_ahci_pci_driver);
diff --git a/drivers/ata/ahci.c b/drivers/ata/ahci.c
index a29f801..52decdc 100644
--- a/drivers/ata/ahci.c
+++ b/drivers/ata/ahci.c
@@ -1211,6 +1211,9 @@ static int ahci_init_interrupts(struct pci_dev *pdev, unsigned int n_ports,
 		goto single_msi;
 	}
 
+	if (nvec > 1)
+		hpriv->flags |= AHCI_HFLAG_MULTI_MSI;
+
 	return nvec;
 
 single_msi:
@@ -1223,32 +1226,11 @@ intx:
 	return 0;
 }
 
-/**
- *	ahci_host_activate - start AHCI host, request IRQs and register it
- *	@host: target ATA host
- *	@irq: base IRQ number to request
- *	@n_msis: number of MSIs allocated for this host
- *	@irq_handler: irq_handler used when requesting IRQs
- *	@irq_flags: irq_flags used when requesting IRQs
- *
- *	Similar to ata_host_activate, but requests IRQs according to AHCI-1.1
- *	when multiple MSIs were allocated. That is one MSI per port, starting
- *	from @irq.
- *
- *	LOCKING:
- *	Inherited from calling layer (may sleep).
- *
- *	RETURNS:
- *	0 on success, -errno otherwise.
- */
-int ahci_host_activate(struct ata_host *host, int irq, unsigned int n_msis)
+static int ahci_host_activate_multi_irqs(struct ata_host *host, int irq,
+					 struct scsi_host_template *sht)
 {
 	int i, rc;
 
-	/* Sharing Last Message among several ports is not supported */
-	if (n_msis < host->n_ports)
-		return -EINVAL;
-
 	rc = ata_host_start(host);
 	if (rc)
 		return rc;
@@ -1263,8 +1245,8 @@ int ahci_host_activate(struct ata_host *host, int irq, unsigned int n_msis)
 		}
 
 		rc = devm_request_threaded_irq(host->dev, irq + i,
-					       ahci_hw_interrupt,
-					       ahci_thread_fn, IRQF_SHARED,
+					       ahci_multi_irqs_intr,
+					       ahci_port_thread_fn, IRQF_SHARED,
 					       pp->irq_desc, host->ports[i]);
 		if (rc)
 			goto out_free_irqs;
@@ -1273,7 +1255,7 @@ int ahci_host_activate(struct ata_host *host, int irq, unsigned int n_msis)
 	for (i = 0; i < host->n_ports; i++)
 		ata_port_desc(host->ports[i], "irq %d", irq + i);
 
-	rc = ata_host_register(host, &ahci_sht);
+	rc = ata_host_register(host, sht);
 	if (rc)
 		goto out_free_all_irqs;
 
@@ -1288,6 +1270,60 @@ out_free_irqs:
 	return rc;
 }
 
+static int ahci_host_activate_single_irq(struct ata_host *host, int irq,
+					 struct scsi_host_template *sht)
+{
+	int i, rc;
+
+	rc = ata_host_start(host);
+	if (rc)
+		return rc;
+
+	rc = devm_request_threaded_irq(host->dev, irq, ahci_single_irq_intr,
+				       ahci_thread_fn, IRQF_SHARED,
+				       dev_driver_string(host->dev), host);
+	if (rc)
+		return rc;
+
+	for (i = 0; i < host->n_ports; i++)
+		ata_port_desc(host->ports[i], "irq %d", irq);
+
+	rc = ata_host_register(host, sht);
+	if (rc)
+		devm_free_irq(host->dev, irq, host);
+
+	return rc;
+
+}
+
+/**
+ *	ahci_host_activate - start AHCI host, request IRQs and register it
+ *	@host: target ATA host
+ *	@irq: base IRQ number to request
+ *	@sht: scsi_host_template to use when registering the host
+ *
+ *	Similar to ata_host_activate, but requests IRQs according to AHCI-1.1
+ *	when multiple MSIs were allocated. That is one MSI per port, starting
+ *	from @irq.
+ *
+ *	LOCKING:
+ *	Inherited from calling layer (may sleep).
+ *
+ *	RETURNS:
+ *	0 on success, -errno otherwise.
+ */
+int ahci_host_activate(struct ata_host *host, int irq,
+		       struct scsi_host_template *sht)
+{
+	struct ahci_host_priv *hpriv = host->private_data;
+
+	if (hpriv->flags & AHCI_HFLAG_MULTI_MSI)
+		return ahci_host_activate_multi_irqs(host, irq, sht);
+	else
+		return ahci_host_activate_single_irq(host, irq, sht);
+}
+EXPORT_SYMBOL_GPL(ahci_host_activate);
+
 static int ahci_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 {
 	unsigned int board_id = ent->driver_data;
@@ -1296,7 +1332,7 @@ static int ahci_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 	struct device *dev = &pdev->dev;
 	struct ahci_host_priv *hpriv;
 	struct ata_host *host;
-	int n_ports, n_msis, i, rc;
+	int n_ports, i, rc;
 	int ahci_pci_bar = AHCI_PCI_BAR_STANDARD;
 
 	VPRINTK("ENTER\n");
@@ -1437,9 +1473,7 @@ static int ahci_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 	 */
 	n_ports = max(ahci_nr_ports(hpriv->cap), fls(hpriv->port_map));
 
-	n_msis = ahci_init_interrupts(pdev, n_ports, hpriv);
-	if (n_msis > 1)
-		hpriv->flags |= AHCI_HFLAG_MULTI_MSI;
+	ahci_init_interrupts(pdev, n_ports, hpriv);
 
 	host = ata_host_alloc_pinfo(&pdev->dev, ppi, n_ports);
 	if (!host)
@@ -1491,11 +1525,7 @@ static int ahci_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 
 	pci_set_master(pdev);
 
-	if (hpriv->flags & AHCI_HFLAG_MULTI_MSI)
-		return ahci_host_activate(host, pdev->irq, n_msis);
-
-	return ata_host_activate(host, pdev->irq, ahci_interrupt, IRQF_SHARED,
-				 &ahci_sht);
+	return ahci_host_activate(host, pdev->irq, &ahci_sht);
 }
 
 module_pci_driver(ahci_pci_driver);
diff --git a/drivers/ata/ahci.h b/drivers/ata/ahci.h
index 59ae0ee..c12f590 100644
--- a/drivers/ata/ahci.h
+++ b/drivers/ata/ahci.h
@@ -388,11 +388,13 @@ int ahci_port_resume(struct ata_port *ap);
 void ahci_set_em_messages(struct ahci_host_priv *hpriv,
 			  struct ata_port_info *pi);
 int ahci_reset_em(struct ata_host *host);
-irqreturn_t ahci_interrupt(int irq, void *dev_instance);
-irqreturn_t ahci_hw_interrupt(int irq, void *dev_instance);
+irqreturn_t ahci_single_irq_intr(int irq, void *dev_instance);
+irqreturn_t ahci_multi_irqs_intr(int irq, void *dev_instance);
 irqreturn_t ahci_thread_fn(int irq, void *dev_instance);
+irqreturn_t ahci_port_thread_fn(int irq, void *dev_instance);
 void ahci_print_info(struct ata_host *host, const char *scc_s);
-int ahci_host_activate(struct ata_host *host, int irq, unsigned int n_msis);
+int ahci_host_activate(struct ata_host *host, int irq,
+		       struct scsi_host_template *sht);
 void ahci_error_handler(struct ata_port *ap);
 
 static inline void __iomem *__ahci_port_base(struct ata_host *host,
diff --git a/drivers/ata/libahci.c b/drivers/ata/libahci.c
index b784e9d..169c272 100644
--- a/drivers/ata/libahci.c
+++ b/drivers/ata/libahci.c
@@ -1693,9 +1693,9 @@ static void ahci_error_intr(struct ata_port *ap, u32 irq_stat)
 		ata_port_abort(ap);
 }
 
-static void ahci_handle_port_interrupt(struct ata_port *ap,
-				       void __iomem *port_mmio, u32 status)
+static void ahci_handle_port_interrupt(struct ata_port *ap, u32 status)
 {
+	void __iomem *port_mmio = ahci_port_base(ap);
 	struct ata_eh_info *ehi = &ap->link.eh_info;
 	struct ahci_port_priv *pp = ap->private_data;
 	struct ahci_host_priv *hpriv = ap->host->private_data;
@@ -1778,22 +1778,10 @@ static void ahci_handle_port_interrupt(struct ata_port *ap,
 	}
 }
 
-static void ahci_port_intr(struct ata_port *ap)
-{
-	void __iomem *port_mmio = ahci_port_base(ap);
-	u32 status;
-
-	status = readl(port_mmio + PORT_IRQ_STAT);
-	writel(status, port_mmio + PORT_IRQ_STAT);
-
-	ahci_handle_port_interrupt(ap, port_mmio, status);
-}
-
-irqreturn_t ahci_thread_fn(int irq, void *dev_instance)
+irqreturn_t ahci_port_thread_fn(int irq, void *dev_instance)
 {
 	struct ata_port *ap = dev_instance;
 	struct ahci_port_priv *pp = ap->private_data;
-	void __iomem *port_mmio = ahci_port_base(ap);
 	unsigned long flags;
 	u32 status;
 
@@ -1804,14 +1792,43 @@ irqreturn_t ahci_thread_fn(int irq, void *dev_instance)
 	spin_unlock_irqrestore(&ap->host->lock, flags);
 
 	spin_lock_bh(ap->lock);
-	ahci_handle_port_interrupt(ap, port_mmio, status);
+	ahci_handle_port_interrupt(ap, status);
 	spin_unlock_bh(ap->lock);
 
 	return IRQ_HANDLED;
 }
+EXPORT_SYMBOL_GPL(ahci_port_thread_fn);
+
+irqreturn_t ahci_thread_fn(int irq, void *dev_instance)
+{
+	struct ata_host *host = dev_instance;
+	struct ahci_host_priv *hpriv = host->private_data;
+	u32 irq_masked = hpriv->port_map;
+	unsigned int i;
+
+	for (i = 0; i < host->n_ports; i++) {
+		struct ata_port *ap;
+
+		if (!(irq_masked & (1 << i)))
+			continue;
+
+		ap = host->ports[i];
+		if (ap) {
+			ahci_port_thread_fn(irq, ap);
+			VPRINTK("port %u\n", i);
+		} else {
+			VPRINTK("port %u (no irq)\n", i);
+			if (ata_ratelimit())
+				dev_warn(host->dev,
+					 "interrupt on disabled port %u\n", i);
+		}
+	}
+
+	return IRQ_HANDLED;
+}
 EXPORT_SYMBOL_GPL(ahci_thread_fn);
 
-static void ahci_hw_port_interrupt(struct ata_port *ap)
+static void ahci_update_intr_status(struct ata_port *ap)
 {
 	void __iomem *port_mmio = ahci_port_base(ap);
 	struct ahci_port_priv *pp = ap->private_data;
@@ -1823,7 +1840,7 @@ static void ahci_hw_port_interrupt(struct ata_port *ap)
 	pp->intr_status |= status;
 }
 
-irqreturn_t ahci_hw_interrupt(int irq, void *dev_instance)
+irqreturn_t ahci_multi_irqs_intr(int irq, void *dev_instance)
 {
 	struct ata_port *ap_this = dev_instance;
 	struct ahci_port_priv *pp = ap_this->private_data;
@@ -1859,7 +1876,7 @@ irqreturn_t ahci_hw_interrupt(int irq, void *dev_instance)
 
 		ap = host->ports[i];
 		if (ap) {
-			ahci_hw_port_interrupt(ap);
+			ahci_update_intr_status(ap);
 			VPRINTK("port %u\n", i);
 		} else {
 			VPRINTK("port %u (no irq)\n", i);
@@ -1877,9 +1894,9 @@ irqreturn_t ahci_hw_interrupt(int irq, void *dev_instance)
 
 	return IRQ_WAKE_THREAD;
 }
-EXPORT_SYMBOL_GPL(ahci_hw_interrupt);
+EXPORT_SYMBOL_GPL(ahci_multi_irqs_intr);
 
-irqreturn_t ahci_interrupt(int irq, void *dev_instance)
+irqreturn_t ahci_single_irq_intr(int irq, void *dev_instance)
 {
 	struct ata_host *host = dev_instance;
 	struct ahci_host_priv *hpriv;
@@ -1909,7 +1926,7 @@ irqreturn_t ahci_interrupt(int irq, void *dev_instance)
 
 		ap = host->ports[i];
 		if (ap) {
-			ahci_port_intr(ap);
+			ahci_update_intr_status(ap);
 			VPRINTK("port %u\n", i);
 		} else {
 			VPRINTK("port %u (no irq)\n", i);
@@ -1936,9 +1953,9 @@ irqreturn_t ahci_interrupt(int irq, void *dev_instance)
 
 	VPRINTK("EXIT\n");
 
-	return IRQ_RETVAL(handled);
+	return handled ? IRQ_WAKE_THREAD : IRQ_NONE;
 }
-EXPORT_SYMBOL_GPL(ahci_interrupt);
+EXPORT_SYMBOL_GPL(ahci_single_irq_intr);
 
 unsigned int ahci_qc_issue(struct ata_queued_cmd *qc)
 {
@@ -2349,13 +2366,8 @@ static int ahci_port_start(struct ata_port *ap)
 	 */
 	pp->intr_mask = DEF_PORT_IRQ;
 
-	/*
-	 * Switch to per-port locking in case each port has its own MSI vector.
-	 */
-	if ((hpriv->flags & AHCI_HFLAG_MULTI_MSI)) {
-		spin_lock_init(&pp->lock);
-		ap->lock = &pp->lock;
-	}
+	spin_lock_init(&pp->lock);
+	ap->lock = &pp->lock;
 
 	ap->private_data = pp;
 
diff --git a/drivers/ata/libahci_platform.c b/drivers/ata/libahci_platform.c
index 5b92c29..a085224 100644
--- a/drivers/ata/libahci_platform.c
+++ b/drivers/ata/libahci_platform.c
@@ -495,8 +495,7 @@ int ahci_platform_init_host(struct platform_device *pdev,
 	ahci_init_controller(host);
 	ahci_print_info(host, "platform");
 
-	return ata_host_activate(host, irq, ahci_interrupt, IRQF_SHARED,
-				 &ahci_platform_sht);
+	return ahci_host_activate(host, irq, &ahci_platform_sht);
 }
 EXPORT_SYMBOL_GPL(ahci_platform_init_host);
 
-- 
1.9.3

-- 
Regards,
Alexander Gordeev
agordeev@redhat.com

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Bartlomiej Zolnierkiewicz]

Hi,

On Thursday, September 11, 2014 02:42:49 PM Alexander Gordeev wrote:

> The numbers are taken by running 'dd if=/dev/sd{a,b} of=/dev/null'
> in parallel. All time values are in us.
> 
> Before this update host lock average holdtime was 2.45 and
> average waittime was 1.24. After the update average holdtime
> dropped to 0.29 (about eight times) while average waittime
> decreased to 0.58 (about two times).
> 
> Also, port events are handled with local interrupts enabled
> and compete on individual per-port locks with average holdtime
> 1.25 and average waittime 1.48. So combined average holdtime
> spent while holding host and port locks decreased from 2.45 to
> 0.29 + 1.25 = 1.54 (about 1.6 times).
> 
> The downside of this change is introduction of a kernel thread.
> 
> The upside is shorter access time to port locks and moving port
> interrupts handling out of the hardware interrupt context.

IMHO it would be great to put the above results into the patch
description (it now looks a bit skimpy).

Best regards,
--
Bartlomiej Zolnierkiewicz
Samsung R&D Institute Poland
Samsung Electronics

Re: [PATCH v2] AHCI: Optimize interrupt processing

[Tejun Heo]

On Thu, Sep 11, 2014 at 02:44:37PM +0200, Alexander Gordeev wrote:
> Split interrupt service routine into hardware context handler and
> threaded context handler. That allows to protect ports with individual
> locks rather than with a single host-wide lock, which results in better
> parallelism.

This patch is way too large.  Can you please split it up to smaller
logical changes?

Thanks.

-- 
tejun

Re: [PATCH RESEND 0/1] AHCI: Optimize interrupt processing

[Tejun Heo]

Hello,

On Thu, Sep 11, 2014 at 02:42:49PM +0200, Alexander Gordeev wrote:
> As conversion of libata to blk mq has long done I tried the change

Hmmm?  You mean scsi-mq?

> against the recent version and the results still appear worthwhile.
> 
> The numbers are taken by running 'dd if=/dev/sd{a,b} of=/dev/null'
> in parallel. All time values are in us.
> 
> Before this update host lock average holdtime was 2.45 and
> average waittime was 1.24. After the update average holdtime
> dropped to 0.29 (about eight times) while average waittime
> decreased to 0.58 (about two times).
> 
> Also, port events are handled with local interrupts enabled
> and compete on individual per-port locks with average holdtime
> 1.25 and average waittime 1.48. So combined average holdtime
> spent while holding host and port locks decreased from 2.45 to
> 0.29 + 1.25 = 1.54 (about 1.6 times).
> 
> The downside of this change is introduction of a kernel thread.

That shouldn't matter at all but can you please present the
information in a more digestable form?  e.g. CPU usage decreased from
A to B when transferring N MB/s on certain setup.

Thanks.

-- 
tejun