[PATCH net-26 0/5] cxgb4vf: minor bug fixes

[PATCH net-26 0/5] cxgb4vf: minor bug fixes

[Casey Leedom]

Several minor bug fixes in the cxgb4vf driver ...

[01/05]: Virtual Interfaces are always up ...
[02/05]: Check driver parameters in the right place ...
[03/05]: Behave properly when CONFIG_DEBUG_FS isn't defined ...
[04/05]: Quiesce Virtual Interfaces on shutdown ...
[05/05]: Use defined Mailbox Timeout

 drivers/net/cxgb4vf/cxgb4vf_main.c |  110 ++++++++++++++++++++++++++++--------
 drivers/net/cxgb4vf/t4vf_hw.c      |    2 +-
 2 files changed, 88 insertions(+), 24 deletions(-)

[PATCH net-26 1/5] cxgb4vf: Virtual Interfaces are always up ...

[Casey Leedom]

Implement new default mode of always reporting the Virtual Interface link as
being "up".  This allows different Virtual Interfaces on the same port to
continue to communicate with each other even when the physical port link is
down.  This new behavior is controlled via the module parameter
force_link_up (default 1).  The old behavior can be achieved by setting
force_link_up=0.

Signed-off-by: Casey Leedom <leedom@chelsio.com>
---
 drivers/net/cxgb4vf/cxgb4vf_main.c |   30 +++++++++++++++++++++++++++---
 1 files changed, 27 insertions(+), 3 deletions(-)

diff --git a/drivers/net/cxgb4vf/cxgb4vf_main.c b/drivers/net/cxgb4vf/cxgb4vf_main.c
index 56166ae..08c2b29 100644
--- a/drivers/net/cxgb4vf/cxgb4vf_main.c
+++ b/drivers/net/cxgb4vf/cxgb4vf_main.c
@@ -96,6 +96,18 @@ module_param(msi, int, 0644);
 MODULE_PARM_DESC(msi, "whether to use MSI-X or MSI");
 
 /*
+ * The Virtual Interfaces are connected to an internal switch on the chip
+ * which allows VIs attached to the same port to talk to each other even when
+ * the port link is down.  As a result, we generally want to always report a
+ * VI's link as being "up".
+ */
+static int force_link_up = 1;
+
+module_param(force_link_up, int, 0644);
+MODULE_PARM_DESC(force_link_up, "always report link up");
+
+
+/*
  * Fundamental constants.
  * ======================
  */
@@ -151,7 +163,8 @@ void t4vf_os_link_changed(struct adapter *adapter, int pidx, int link_ok)
 		return;
 
 	/*
-	 * Tell the OS that the link status has changed and print a short
+	 * Tell the OS that the link status has changed (if we're not
+	 * operating in the forced link "up" mode) and print a short
 	 * informative message on the console about the event.
 	 */
 	if (link_ok) {
@@ -159,7 +172,8 @@ void t4vf_os_link_changed(struct adapter *adapter, int pidx, int link_ok)
 		const char *fc;
 		const struct port_info *pi = netdev_priv(dev);
 
-		netif_carrier_on(dev);
+		if (!force_link_up)
+			netif_carrier_on(dev);
 
 		switch (pi->link_cfg.speed) {
 		case SPEED_10000:
@@ -200,7 +214,9 @@ void t4vf_os_link_changed(struct adapter *adapter, int pidx, int link_ok)
 		printk(KERN_INFO "%s: link up, %s, full-duplex, %s PAUSE\n",
 		       dev->name, s, fc);
 	} else {
-		netif_carrier_off(dev);
+		if (!force_link_up)
+			netif_carrier_off(dev);
+
 		printk(KERN_INFO "%s: link down\n", dev->name);
 	}
 }
@@ -254,6 +270,14 @@ static int link_start(struct net_device *dev)
 	 */
 	if (ret == 0)
 		ret = t4vf_enable_vi(pi->adapter, pi->viid, true, true);
+
+	/*
+	 * If we didn't experience any error and we're always reporting the
+	 * link as being "up", tell the OS that the link is up.
+	 */
+	if (ret == 0 && force_link_up)
+		netif_carrier_on(dev);
+
 	return ret;
 }
 
-- 
1.7.0.4

[PATCH net-26 2/5] cxgb4vf: Check driver parameters in the right place ...

[Casey Leedom]

Check module parameter validity in the module initialization routine instead
of the PCI Device Probe routine.

Signed-off-by: Casey Leedom <leedom@chelsio.com>
---
 drivers/net/cxgb4vf/cxgb4vf_main.c |   23 +++++++++++------------
 1 files changed, 11 insertions(+), 12 deletions(-)

diff --git a/drivers/net/cxgb4vf/cxgb4vf_main.c b/drivers/net/cxgb4vf/cxgb4vf_main.c
index 08c2b29..b12c169 100644
--- a/drivers/net/cxgb4vf/cxgb4vf_main.c
+++ b/drivers/net/cxgb4vf/cxgb4vf_main.c
@@ -2513,17 +2513,6 @@ static int __devinit cxgb4vf_pci_probe(struct pci_dev *pdev,
 	struct net_device *netdev;
 
 	/*
-	 * Vet our module parameters.
-	 */
-	if (msi != MSI_MSIX && msi != MSI_MSI) {
-		dev_err(&pdev->dev, "bad module parameter msi=%d; must be %d"
-			" (MSI-X or MSI) or %d (MSI)\n", msi, MSI_MSIX,
-			MSI_MSI);
-		err = -EINVAL;
-		goto err_out;
-	}
-
-	/*
 	 * Print our driver banner the first time we're called to initialize a
 	 * device.
 	 */
@@ -2826,7 +2815,6 @@ err_release_regions:
 err_disable_device:
 	pci_disable_device(pdev);
 
-err_out:
 	return err;
 }
 
@@ -2939,6 +2927,17 @@ static int __init cxgb4vf_module_init(void)
 {
 	int ret;
 
+	/*
+	 * Vet our module parameters.
+	 */
+	if (msi != MSI_MSIX && msi != MSI_MSI) {
+		printk(KERN_WARNING KBUILD_MODNAME
+		       ": bad module parameter msi=%d; must be %d"
+		       " (MSI-X or MSI) or %d (MSI)\n",
+		       msi, MSI_MSIX, MSI_MSI);
+		return -EINVAL;
+	}
+
 	/* Debugfs support is optional, just warn if this fails */
 	cxgb4vf_debugfs_root = debugfs_create_dir(KBUILD_MODNAME, NULL);
 	if (!cxgb4vf_debugfs_root)
-- 
1.7.0.4

[PATCH net-26 3/5] cxgb4vf: Behave properly when CONFIG_DEBUG_FS isn't defined ...

[Casey Leedom]

When CONFIG_DEBUG_FS we get "ERR_PTR()"s back from the debugfs routines
instead of NULL.  Use the right predicates to check for this.

Signed-off-by: Casey Leedom <leedom@chelsio.com>
---
 drivers/net/cxgb4vf/cxgb4vf_main.c |   16 ++++++++--------
 1 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/drivers/net/cxgb4vf/cxgb4vf_main.c b/drivers/net/cxgb4vf/cxgb4vf_main.c
index b12c169..daac6ed 100644
--- a/drivers/net/cxgb4vf/cxgb4vf_main.c
+++ b/drivers/net/cxgb4vf/cxgb4vf_main.c
@@ -2064,7 +2064,7 @@ static int __devinit setup_debugfs(struct adapter *adapter)
 {
 	int i;
 
-	BUG_ON(adapter->debugfs_root == NULL);
+	BUG_ON(IS_ERR_OR_NULL(adapter->debugfs_root));
 
 	/*
 	 * Debugfs support is best effort.
@@ -2085,7 +2085,7 @@ static int __devinit setup_debugfs(struct adapter *adapter)
  */
 static void cleanup_debugfs(struct adapter *adapter)
 {
-	BUG_ON(adapter->debugfs_root == NULL);
+	BUG_ON(IS_ERR_OR_NULL(adapter->debugfs_root));
 
 	/*
 	 * Unlike our sister routine cleanup_proc(), we don't need to remove
@@ -2724,11 +2724,11 @@ static int __devinit cxgb4vf_pci_probe(struct pci_dev *pdev,
 	/*
 	 * Set up our debugfs entries.
 	 */
-	if (cxgb4vf_debugfs_root) {
+	if (!IS_ERR_OR_NULL(cxgb4vf_debugfs_root)) {
 		adapter->debugfs_root =
 			debugfs_create_dir(pci_name(pdev),
 					   cxgb4vf_debugfs_root);
-		if (adapter->debugfs_root == NULL)
+		if (IS_ERR_OR_NULL(adapter->debugfs_root))
 			dev_warn(&pdev->dev, "could not create debugfs"
 				 " directory");
 		else
@@ -2783,7 +2783,7 @@ static int __devinit cxgb4vf_pci_probe(struct pci_dev *pdev,
 	 */
 
 err_free_debugfs:
-	if (adapter->debugfs_root) {
+	if (!IS_ERR_OR_NULL(adapter->debugfs_root)) {
 		cleanup_debugfs(adapter);
 		debugfs_remove_recursive(adapter->debugfs_root);
 	}
@@ -2852,7 +2852,7 @@ static void __devexit cxgb4vf_pci_remove(struct pci_dev *pdev)
 		/*
 		 * Tear down our debugfs entries.
 		 */
-		if (adapter->debugfs_root) {
+		if (!IS_ERR_OR_NULL(adapter->debugfs_root)) {
 			cleanup_debugfs(adapter);
 			debugfs_remove_recursive(adapter->debugfs_root);
 		}
@@ -2940,12 +2940,12 @@ static int __init cxgb4vf_module_init(void)
 
 	/* Debugfs support is optional, just warn if this fails */
 	cxgb4vf_debugfs_root = debugfs_create_dir(KBUILD_MODNAME, NULL);
-	if (!cxgb4vf_debugfs_root)
+	if (IS_ERR_OR_NULL(cxgb4vf_debugfs_root))
 		printk(KERN_WARNING KBUILD_MODNAME ": could not create"
 		       " debugfs entry, continuing\n");
 
 	ret = pci_register_driver(&cxgb4vf_driver);
-	if (ret < 0)
+	if (ret < 0 && !IS_ERR_OR_NULL(cxgb4vf_debugfs_root))
 		debugfs_remove(cxgb4vf_debugfs_root);
 	return ret;
 }
-- 
1.7.0.4

[PATCH net-26 4/5] cxgb4vf: Quiesce Virtual Interfaces on shutdown ...

[Casey Leedom]

When a Virtual Machine is rebooted, KVM currently fails to issue a Function
Level Reset against any "Attached PCI Devices" (AKA "PCI Passthrough").  In
addition to leaving the attached device in a random state in the next booted
kernel (which sort of violates the entire idea of a reboot reseting hardware
state), this leaves our peer thinking that the link is still up.  (Note that
a bug has been filed with the KVM folks, #25332, but there's been no
response on that as of yet.)  So, we add a "->shutdown()" method for the
Virtual Function PCI Device to handle administrative shutdowns like a
reboot.

Signed-off-by: Casey Leedom <leedom@chelsio.com>
---
 drivers/net/cxgb4vf/cxgb4vf_main.c |   41 ++++++++++++++++++++++++++++++++++++
 1 files changed, 41 insertions(+), 0 deletions(-)

diff --git a/drivers/net/cxgb4vf/cxgb4vf_main.c b/drivers/net/cxgb4vf/cxgb4vf_main.c
index daac6ed..f78d6e1 100644
--- a/drivers/net/cxgb4vf/cxgb4vf_main.c
+++ b/drivers/net/cxgb4vf/cxgb4vf_main.c
@@ -2886,6 +2886,46 @@ static void __devexit cxgb4vf_pci_remove(struct pci_dev *pdev)
 }
 
 /*
+ * "Shutdown" quiesce the device, stopping Ingress Packet and Interrupt
+ * delivery.
+ */
+static void __devexit cxgb4vf_pci_shutdown(struct pci_dev *pdev)
+{
+	struct adapter *adapter;
+	int pidx;
+
+	adapter = pci_get_drvdata(pdev);
+	if (!adapter)
+		return;
+
+	/*
+	 * Disable all Virtual Interfaces.  This will shut down the
+	 * delivery of all ingress packets into the chip for these
+	 * Virtual Interfaces.
+	 */
+	for_each_port(adapter, pidx) {
+		struct net_device *netdev;
+		struct port_info *pi;
+
+		if (!test_bit(pidx, &adapter->registered_device_map))
+			continue;
+
+		netdev = adapter->port[pidx];
+		if (!netdev)
+			continue;
+
+		pi = netdev_priv(netdev);
+		t4vf_enable_vi(adapter, pi->viid, false, false);
+	}
+
+	/*
+	 * Free up all Queues which will prevent further DMA and
+	 * Interrupts allowing various internal pathways to drain.
+	 */
+	t4vf_free_sge_resources(adapter);
+}
+
+/*
  * PCI Device registration data structures.
  */
 #define CH_DEVICE(devid, idx) \
@@ -2918,6 +2958,7 @@ static struct pci_driver cxgb4vf_driver = {
 	.id_table	= cxgb4vf_pci_tbl,
 	.probe		= cxgb4vf_pci_probe,
 	.remove		= __devexit_p(cxgb4vf_pci_remove),
+	.shutdown	= __devexit_p(cxgb4vf_pci_shutdown),
 };
 
 /*
-- 
1.7.0.4

[PATCH net-26 5/5] cxgb4vf: Use defined Mailbox Timeout

[Casey Leedom]

VF Driver should use mailbox command timeout specified in t4fw_interface.h
rather than hard-coded value of 500ms.

Signed-off-by: Casey Leedom <leedom@chelsio.com>
---
 drivers/net/cxgb4vf/t4vf_hw.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/net/cxgb4vf/t4vf_hw.c b/drivers/net/cxgb4vf/t4vf_hw.c
index 0f51c80..192db22 100644
--- a/drivers/net/cxgb4vf/t4vf_hw.c
+++ b/drivers/net/cxgb4vf/t4vf_hw.c
@@ -171,7 +171,7 @@ int t4vf_wr_mbox_core(struct adapter *adapter, const void *cmd, int size,
 	delay_idx = 0;
 	ms = delay[0];
 
-	for (i = 0; i < 500; i += ms) {
+	for (i = 0; i < FW_CMD_MAX_TIMEOUT; i += ms) {
 		if (sleep_ok) {
 			ms = delay[delay_idx];
 			if (delay_idx < ARRAY_SIZE(delay) - 1)
-- 
1.7.0.4

Re: [PATCH net-26 1/5] cxgb4vf: Virtual Interfaces are always up ...

[David Miller]

From: Casey Leedom <leedom@chelsio.com>
Date: Fri, 11 Feb 2011 17:00:19 -0800

> Implement new default mode of always reporting the Virtual Interface link as
> being "up".  This allows different Virtual Interfaces on the same port to
> continue to communicate with each other even when the physical port link is
> down.  This new behavior is controlled via the module parameter
> force_link_up (default 1).  The old behavior can be achieved by setting
> force_link_up=0.
> 
> Signed-off-by: Casey Leedom <leedom@chelsio.com>

No driver specific module parameters!  Add something generic and common
so other drivers can use it too.

Otherwise every user has to learn a different way to control this
attribute, depending upon the device type, which is rediculious.

How many times do we have to tell driver authors this?

Re: [PATCH net-26 4/5] cxgb4vf: Quiesce Virtual Interfaces on shutdown ...

[Anirban Chakraborty]

On Feb 11, 2011, at 5:00 PM, Casey Leedom wrote:

> When a Virtual Machine is rebooted, KVM currently fails to issue a Function
> Level Reset against any "Attached PCI Devices" (AKA "PCI Passthrough").  In
> addition to leaving the attached device in a random state in the next booted
> kernel (which sort of violates the entire idea of a reboot reseting hardware
> state), this leaves our peer thinking that the link is still up.  (Note that
> a bug has been filed with the KVM folks, #25332, but there's been no
> response on that as of yet.)  So, we add a "->shutdown()" method for the
> Virtual Function PCI Device to handle administrative shutdowns like a
> reboot.
>
> Signed-off-by: Casey Leedom <leedom@chelsio.com>
> ---
> drivers/net/cxgb4vf/cxgb4vf_main.c |   41 ++++++++++++++++++++++++++++++++++++
> 1 files changed, 41 insertions(+), 0 deletions(-)
>
> diff --git a/drivers/net/cxgb4vf/cxgb4vf_main.c b/drivers/net/cxgb4vf/cxgb4vf_main.c
> index daac6ed..f78d6e1 100644
> --- a/drivers/net/cxgb4vf/cxgb4vf_main.c
> +++ b/drivers/net/cxgb4vf/cxgb4vf_main.c
> @@ -2886,6 +2886,46 @@ static void __devexit cxgb4vf_pci_remove(struct pci_dev *pdev)
> }
>
> /*
> + * "Shutdown" quiesce the device, stopping Ingress Packet and Interrupt
> + * delivery.
> + */
> +static void __devexit cxgb4vf_pci_shutdown(struct pci_dev *pdev)
> +{
> +     struct adapter *adapter;
> +     int pidx;
> +
> +     adapter = pci_get_drvdata(pdev);
> +     if (!adapter)
> +             return;
> +
> +     /*
> +      * Disable all Virtual Interfaces.  This will shut down the
> +      * delivery of all ingress packets into the chip for these
> +      * Virtual Interfaces.
> +      */
> +     for_each_port(adapter, pidx) {
> +             struct net_device *netdev;
> +             struct port_info *pi;
> +
> +             if (!test_bit(pidx, &adapter->registered_device_map))
> +                     continue;
> +
> +             netdev = adapter->port[pidx];
> +             if (!netdev)
> +                     continue;
> +
> +             pi = netdev_priv(netdev);
> +             t4vf_enable_vi(adapter, pi->viid, false, false);
> +     }
> +
> +     /*
> +      * Free up all Queues which will prevent further DMA and
> +      * Interrupts allowing various internal pathways to drain.
> +      */
> +     t4vf_free_sge_resources(adapter);
> +}
> +
> +/*
>  * PCI Device registration data structures.
>  */
> #define CH_DEVICE(devid, idx) \
> @@ -2918,6 +2958,7 @@ static struct pci_driver cxgb4vf_driver = {
>       .id_table       = cxgb4vf_pci_tbl,
>       .probe          = cxgb4vf_pci_probe,
>       .remove         = __devexit_p(cxgb4vf_pci_remove),
> +     .shutdown       = __devexit_p(cxgb4vf_pci_shutdown),
> };
>
> /*
> --
> 1.7.0.4


You could invoke pci_reset_function as an alternative, that would make sure FLR happens.

-Anirban


This message and any attached documents contain information from QLogic Corporation or its wholly-owned subsidiaries that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.

Re: [PATCH net-26 4/5] cxgb4vf: Quiesce Virtual Interfaces on shutdown ...

[Casey Leedom]

| From: Anirban Chakraborty <anirban.chakraborty@qlogic.com>
| Date: Friday, February 11, 2011 11:17 pm
| 
| 
| You could invoke pci_reset_function as an alternative, that would make sure
| FLR happens.

  I'd love to do that but pci_reset_function() ends up taking the device 
semaphore in the underlaying routine pci_dev_reset().  I used to call 
pci_reset_function() in my probe() routine till 2.6.31 where that lock as added 
... which resulted in an instant self-deadlock ...

Casey

Re: [PATCH net-26 1/5] cxgb4vf: Virtual Interfaces are always up ...

[Casey Leedom]

| From: David Miller <davem@davemloft.net>
| Date: Friday, February 11, 2011 09:19 pm
| 
| From: Casey Leedom <leedom@chelsio.com>
| Date: Fri, 11 Feb 2011 17:00:19 -0800
| 
| > Implement new default mode of always reporting the Virtual Interface link
| > as being "up".  This allows different Virtual Interfaces on the same
| > port to continue to communicate with each other even when the physical
| > port link is down.  This new behavior is controlled via the module
| > parameter
| > force_link_up (default 1).  The old behavior can be achieved by setting
| > force_link_up=0.
| > 
| > Signed-off-by: Casey Leedom <leedom@chelsio.com>
| 
| No driver specific module parameters!  Add something generic and common
| so other drivers can use it too.
| 
| Otherwise every user has to learn a different way to control this
| attribute, depending upon the device type, which is rediculious.
| 
| How many times do we have to tell driver authors this?

  Sorry.  I wasn't aware of this rule.  My bad.  Is this writeen down somewhere 
under Documentation?  I'm not being snarky.  I really would like to know so I 
can read through the general ground rules and avoid making more mistakes in the 
future.

  As for a generic mechanism, what's the preferred way of doing this?  A new 
ethtool flag?  Sorry for being a doofus here, I'm happy to follow whatever the 
accepted standard is.  Thanks for your time and patience.

Casey

Re: [PATCH net-26 1/5] cxgb4vf: Virtual Interfaces are always up ...

[Casey Leedom]

| From: Casey Leedom <leedom@chelsio.com>
| Date: Monday, February 14, 2011 11:13 am
| 
| | No driver specific module parameters!  Add something generic and common
| | so other drivers can use it too.
| | 
| | Otherwise every user has to learn a different way to control this
| | attribute, depending upon the device type, which is rediculious.
| | 
| | How many times do we have to tell driver authors this?
| 
|   Sorry.  I wasn't aware of this rule.  My bad.  Is this writeen down
| somewhere under Documentation?  I'm not being snarky.  I really would like
| to know so I can read through the general ground rules and avoid making
| more mistakes in the future.
| 
|   As for a generic mechanism, what's the preferred way of doing this?  A
| new ethtool flag?  Sorry for being a doofus here, I'm happy to follow
| whatever the accepted standard is.  Thanks for your time and patience.

  Also, I assume then the the entire patch series is now rejected, right?  And 
that I should resubmit the patch series without the unacceptable module 
parameter, right?  I'm just trying to figure out what I need to do next.  
Thanks.

Casey

Re: [PATCH net-26 1/5] cxgb4vf: Virtual Interfaces are always up ...

[David Miller]

From: Casey Leedom <leedom@chelsio.com>
Date: Mon, 14 Feb 2011 14:31:32 -0800

> | From: Casey Leedom <leedom@chelsio.com>
> | Date: Monday, February 14, 2011 11:13 am
> | 
> | | No driver specific module parameters!  Add something generic and common
> | | so other drivers can use it too.
> | | 
> | | Otherwise every user has to learn a different way to control this
> | | attribute, depending upon the device type, which is rediculious.
> | | 
> | | How many times do we have to tell driver authors this?
> | 
> |   Sorry.  I wasn't aware of this rule.  My bad.  Is this writeen down
> | somewhere under Documentation?  I'm not being snarky.  I really would like
> | to know so I can read through the general ground rules and avoid making
> | more mistakes in the future.
> | 
> |   As for a generic mechanism, what's the preferred way of doing this?  A
> | new ethtool flag?  Sorry for being a doofus here, I'm happy to follow
> | whatever the accepted standard is.  Thanks for your time and patience.
> 
>   Also, I assume then the the entire patch series is now rejected, right?  And 
> that I should resubmit the patch series without the unacceptable module 
> parameter, right?  I'm just trying to figure out what I need to do next.  

You need to resubmit the whole series, because changing an earlier
patch causes the subsequent ones to have, at a minimum, offsets which
GIT apply will reject.