[net-next 0/8][pull request] Intel Wired LAN Driver Update

[net-next 0/8][pull request] Intel Wired LAN Driver Update

[Jeff Kirsher]

The following series contains updates to ixgbe and pci.  The pci patch
is a small update to add a flag to indicate when a device has been
assigned by KVM.  The ixgbe patches consist of the following:

  - reconfigure SR-IOV init to take advantage of the new pci flag
  - DCB cleanups/fixes
  - add WOL support for x540
  - a few cleanup/fixes

The following are changes since commit 9c9b1f24f2aa31a3cea94939edc551f68ebadc89:
  net/phy: add IC+ IP101A and support APS.
and are available in the git repository at:
  git://github.com/Jkirsher/net-next.git

Don Skidmore (1):
  ixgbe: cleanup X540 interrupt enablement

Emil Tantilov (3):
  ixgbe: avoid HW lockup when adapter is reset with Tx work pending
  ixgbe: add WOL support for X540
  ixgbe: remove global reset to the MAC

Greg Rose (2):
  pci: Add flag indicating device has been assigned by KVM
  ixgbe: Reconfigure SR-IOV Init

John Fastabend (2):
  ixgbe: DCB, do not call set_state() from IEEE mode
  ixgbe: dcb, set priority to traffic class mappings

 drivers/net/ethernet/intel/ixgbe/ixgbe.h         |    2 +
 drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c   |   15 +-
 drivers/net/ethernet/intel/ixgbe/ixgbe_82599.c   |   13 +-
 drivers/net/ethernet/intel/ixgbe/ixgbe_common.c  |  145 +++++++++-------
 drivers/net/ethernet/intel/ixgbe/ixgbe_common.h  |    2 +-
 drivers/net/ethernet/intel/ixgbe/ixgbe_dcb.c     |   36 ++++-
 drivers/net/ethernet/intel/ixgbe/ixgbe_dcb.h     |    3 +-
 drivers/net/ethernet/intel/ixgbe/ixgbe_dcb_nl.c  |   57 +++----
 drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c |   13 ++
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c    |  141 ++++-----------
 drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.c   |  207 +++++++++++++++++++++-
 drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.h   |    5 +
 drivers/net/ethernet/intel/ixgbe/ixgbe_type.h    |   10 +
 drivers/net/ethernet/intel/ixgbe/ixgbe_x540.c    |   31 +---
 include/linux/pci.h                              |    2 +
 virt/kvm/assigned-dev.c                          |    2 +
 virt/kvm/iommu.c                                 |    4 +
 17 files changed, 443 insertions(+), 245 deletions(-)

-- 
1.7.6.2

[net-next 1/8] pci: Add flag indicating device has been assigned by KVM

[Jeff Kirsher]

From: Greg Rose <gregory.v.rose@intel.com>

Device drivers that create and destroy SR-IOV virtual functions via
calls to pci_enable_sriov() and pci_disable_sriov can cause catastrophic
failures if they attempt to destroy VFs while they are assigned to
guest virtual machines.  By adding a flag for use by the KVM module
to indicate that a device is assigned a device driver can check that
flag and avoid destroying VFs while they are assigned and avoid system
failures.

Signed-off-by: Greg Rose <gregory.v.rose@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
---
 include/linux/pci.h     |    2 ++
 virt/kvm/assigned-dev.c |    2 ++
 virt/kvm/iommu.c        |    4 ++++
 3 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/include/linux/pci.h b/include/linux/pci.h
index f27893b..4f511da 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -174,6 +174,8 @@ enum pci_dev_flags {
 	PCI_DEV_FLAGS_MSI_INTX_DISABLE_BUG = (__force pci_dev_flags_t) 1,
 	/* Device configuration is irrevocably lost if disabled into D3 */
 	PCI_DEV_FLAGS_NO_D3 = (__force pci_dev_flags_t) 2,
+	/* Provide indication device is assigned by KVM */
+	PCI_DEV_FLAGS_ASSIGNED = (__force pci_dev_flags_t) 4,
 };
 
 enum pci_irq_reroute_variant {
diff --git a/virt/kvm/assigned-dev.c b/virt/kvm/assigned-dev.c
index 4e9eaeb..eaf3a50 100644
--- a/virt/kvm/assigned-dev.c
+++ b/virt/kvm/assigned-dev.c
@@ -205,6 +205,8 @@ static void kvm_free_assigned_device(struct kvm *kvm,
 	else
 		pci_restore_state(assigned_dev->dev);
 
+	assigned_dev->dev->dev_flags &= ~PCI_DEV_FLAGS_ASSIGNED;
+
 	pci_release_regions(assigned_dev->dev);
 	pci_disable_device(assigned_dev->dev);
 	pci_dev_put(assigned_dev->dev);
diff --git a/virt/kvm/iommu.c b/virt/kvm/iommu.c
index 78c80f6..967aba1 100644
--- a/virt/kvm/iommu.c
+++ b/virt/kvm/iommu.c
@@ -187,6 +187,8 @@ int kvm_assign_device(struct kvm *kvm,
 			goto out_unmap;
 	}
 
+	pdev->dev_flags |= PCI_DEV_FLAGS_ASSIGNED;
+
 	printk(KERN_DEBUG "assign device %x:%x:%x.%x\n",
 		assigned_dev->host_segnr,
 		assigned_dev->host_busnr,
@@ -215,6 +217,8 @@ int kvm_deassign_device(struct kvm *kvm,
 
 	iommu_detach_device(domain, &pdev->dev);
 
+	pdev->dev_flags &= ~PCI_DEV_FLAGS_ASSIGNED;
+
 	printk(KERN_DEBUG "deassign device %x:%x:%x.%x\n",
 		assigned_dev->host_segnr,
 		assigned_dev->host_busnr,
-- 
1.7.6.2

Re: [net-next 1/8] pci: Add flag indicating device has been assigned by KVM

[Jeff Kirsher]

On Wed, Sep 21, 2011 at 03:12, Jeff Kirsher <jeffrey.t.kirsher@intel.com> wrote:
> From: Greg Rose <gregory.v.rose@intel.com>
>
> Device drivers that create and destroy SR-IOV virtual functions via
> calls to pci_enable_sriov() and pci_disable_sriov can cause catastrophic
> failures if they attempt to destroy VFs while they are assigned to
> guest virtual machines.  By adding a flag for use by the KVM module
> to indicate that a device is assigned a device driver can check that
> flag and avoid destroying VFs while they are assigned and avoid system
> failures.
>
> Signed-off-by: Greg Rose <gregory.v.rose@intel.com>
> Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
> ---
>  include/linux/pci.h     |    2 ++
>  virt/kvm/assigned-dev.c |    2 ++
>  virt/kvm/iommu.c        |    4 ++++
>  3 files changed, 8 insertions(+), 0 deletions(-)
>
> diff --git a/include/linux/pci.h b/include/linux/pci.h
> index f27893b..4f511da 100644
> --- a/include/linux/pci.h
> +++ b/include/linux/pci.h
> @@ -174,6 +174,8 @@ enum pci_dev_flags {
>        PCI_DEV_FLAGS_MSI_INTX_DISABLE_BUG = (__force pci_dev_flags_t) 1,
>        /* Device configuration is irrevocably lost if disabled into D3 */
>        PCI_DEV_FLAGS_NO_D3 = (__force pci_dev_flags_t) 2,
> +       /* Provide indication device is assigned by KVM */
> +       PCI_DEV_FLAGS_ASSIGNED = (__force pci_dev_flags_t) 4,
>  };
>
>  enum pci_irq_reroute_variant {
> diff --git a/virt/kvm/assigned-dev.c b/virt/kvm/assigned-dev.c
> index 4e9eaeb..eaf3a50 100644
> --- a/virt/kvm/assigned-dev.c
> +++ b/virt/kvm/assigned-dev.c
> @@ -205,6 +205,8 @@ static void kvm_free_assigned_device(struct kvm *kvm,
>        else
>                pci_restore_state(assigned_dev->dev);
>
> +       assigned_dev->dev->dev_flags &= ~PCI_DEV_FLAGS_ASSIGNED;
> +
>        pci_release_regions(assigned_dev->dev);
>        pci_disable_device(assigned_dev->dev);
>        pci_dev_put(assigned_dev->dev);
> diff --git a/virt/kvm/iommu.c b/virt/kvm/iommu.c
> index 78c80f6..967aba1 100644
> --- a/virt/kvm/iommu.c
> +++ b/virt/kvm/iommu.c
> @@ -187,6 +187,8 @@ int kvm_assign_device(struct kvm *kvm,
>                        goto out_unmap;
>        }
>
> +       pdev->dev_flags |= PCI_DEV_FLAGS_ASSIGNED;
> +
>        printk(KERN_DEBUG "assign device %x:%x:%x.%x\n",
>                assigned_dev->host_segnr,
>                assigned_dev->host_busnr,
> @@ -215,6 +217,8 @@ int kvm_deassign_device(struct kvm *kvm,
>
>        iommu_detach_device(domain, &pdev->dev);
>
> +       pdev->dev_flags &= ~PCI_DEV_FLAGS_ASSIGNED;
> +
>        printk(KERN_DEBUG "deassign device %x:%x:%x.%x\n",
>                assigned_dev->host_segnr,
>                assigned_dev->host_busnr,
> --
> 1.7.6.2

Jesse/Konrad/Ian-

I sent this patch out as part of a pull request for David Miller's
net-next tree.  I know that Greg sent this originally out to the
linux-pci mailing list as a RFC. Since Greg also has a patch against
ixgbe which implemented this flag, I sent both patches for inclusion
into David Miller's net-next.

Dave is wanting to ensure that the PCI maintainers have reviewed this
and are ok with it before pulls my series of patches.

-- 
Cheers,
Jeff

Re: [net-next 1/8] pci: Add flag indicating device has been assigned by KVM

[Jesse Barnes]

[oops meant to cc everyone, yes this has my ack]

On Thu, 22 Sep 2011 21:16:18 -0700
Jeff Kirsher <jeffrey.t.kirsher@intel.com> wrote:

> On Wed, Sep 21, 2011 at 03:12, Jeff Kirsher
> <jeffrey.t.kirsher@intel.com> wrote:
> > From: Greg Rose <gregory.v.rose@intel.com>
> >
> > Device drivers that create and destroy SR-IOV virtual functions via
> > calls to pci_enable_sriov() and pci_disable_sriov can cause
> > catastrophic failures if they attempt to destroy VFs while they are
> > assigned to guest virtual machines.  By adding a flag for use by
> > the KVM module to indicate that a device is assigned a device
> > driver can check that flag and avoid destroying VFs while they are
> > assigned and avoid system failures.
> >
> > Signed-off-by: Greg Rose <gregory.v.rose@intel.com>
> > Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
> > ---
> >  include/linux/pci.h     |    2 ++
> >  virt/kvm/assigned-dev.c |    2 ++
> >  virt/kvm/iommu.c        |    4 ++++
> >  3 files changed, 8 insertions(+), 0 deletions(-)
> >
> > diff --git a/include/linux/pci.h b/include/linux/pci.h
> > index f27893b..4f511da 100644
> > --- a/include/linux/pci.h
> > +++ b/include/linux/pci.h
> > @@ -174,6 +174,8 @@ enum pci_dev_flags {
> >        PCI_DEV_FLAGS_MSI_INTX_DISABLE_BUG = (__force
> > pci_dev_flags_t) 1, /* Device configuration is irrevocably lost if
> > disabled into D3 */ PCI_DEV_FLAGS_NO_D3 = (__force pci_dev_flags_t)
> > 2,
> > +       /* Provide indication device is assigned by KVM */
> > +       PCI_DEV_FLAGS_ASSIGNED = (__force pci_dev_flags_t) 4,
> >  };
> >
> >  enum pci_irq_reroute_variant {
> > diff --git a/virt/kvm/assigned-dev.c b/virt/kvm/assigned-dev.c
> > index 4e9eaeb..eaf3a50 100644
> > --- a/virt/kvm/assigned-dev.c
> > +++ b/virt/kvm/assigned-dev.c
> > @@ -205,6 +205,8 @@ static void kvm_free_assigned_device(struct kvm
> > *kvm, else
> >                pci_restore_state(assigned_dev->dev);
> >
> > +       assigned_dev->dev->dev_flags &= ~PCI_DEV_FLAGS_ASSIGNED;
> > +
> >        pci_release_regions(assigned_dev->dev);
> >        pci_disable_device(assigned_dev->dev);
> >        pci_dev_put(assigned_dev->dev);
> > diff --git a/virt/kvm/iommu.c b/virt/kvm/iommu.c
> > index 78c80f6..967aba1 100644
> > --- a/virt/kvm/iommu.c
> > +++ b/virt/kvm/iommu.c
> > @@ -187,6 +187,8 @@ int kvm_assign_device(struct kvm *kvm,
> >                        goto out_unmap;
> >        }
> >
> > +       pdev->dev_flags |= PCI_DEV_FLAGS_ASSIGNED;
> > +
> >        printk(KERN_DEBUG "assign device %x:%x:%x.%x\n",
> >                assigned_dev->host_segnr,
> >                assigned_dev->host_busnr,
> > @@ -215,6 +217,8 @@ int kvm_deassign_device(struct kvm *kvm,
> >
> >        iommu_detach_device(domain, &pdev->dev);
> >
> > +       pdev->dev_flags &= ~PCI_DEV_FLAGS_ASSIGNED;
> > +
> >        printk(KERN_DEBUG "deassign device %x:%x:%x.%x\n",
> >                assigned_dev->host_segnr,
> >                assigned_dev->host_busnr,
> > --
> > 1.7.6.2
> 
> Jesse/Konrad/Ian-
> 
> I sent this patch out as part of a pull request for David Miller's
> net-next tree.  I know that Greg sent this originally out to the
> linux-pci mailing list as a RFC. Since Greg also has a patch against
> ixgbe which implemented this flag, I sent both patches for inclusion
> into David Miller's net-next.
> 
> Dave is wanting to ensure that the PCI maintainers have reviewed this
> and are ok with it before pulls my series of patches.
> 

Re: [net-next 1/8] pci: Add flag indicating device has been assigned by KVM

[Jeff Kirsher]

[-- Attachment #1: Type: text/plain, Size: 3859 bytes --]

On Thu, 2011-09-22 at 22:19 -0700, Jesse Barnes wrote:
> On Thu, 22 Sep 2011 21:16:18 -0700
> Jeff Kirsher <jeffrey.t.kirsher@intel.com> wrote:
> 
> > On Wed, Sep 21, 2011 at 03:12, Jeff Kirsher
> > <jeffrey.t.kirsher@intel.com> wrote:
> > > From: Greg Rose <gregory.v.rose@intel.com>
> > >
> > > Device drivers that create and destroy SR-IOV virtual functions via
> > > calls to pci_enable_sriov() and pci_disable_sriov can cause
> > > catastrophic failures if they attempt to destroy VFs while they are
> > > assigned to guest virtual machines.  By adding a flag for use by
> > > the KVM module to indicate that a device is assigned a device
> > > driver can check that flag and avoid destroying VFs while they are
> > > assigned and avoid system failures.
> > >
> > > Signed-off-by: Greg Rose <gregory.v.rose@intel.com>
> > > Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
> > > ---
> > >  include/linux/pci.h     |    2 ++
> > >  virt/kvm/assigned-dev.c |    2 ++
> > >  virt/kvm/iommu.c        |    4 ++++
> > >  3 files changed, 8 insertions(+), 0 deletions(-)
> > >
> > > diff --git a/include/linux/pci.h b/include/linux/pci.h
> > > index f27893b..4f511da 100644
> > > --- a/include/linux/pci.h
> > > +++ b/include/linux/pci.h
> > > @@ -174,6 +174,8 @@ enum pci_dev_flags {
> > >        PCI_DEV_FLAGS_MSI_INTX_DISABLE_BUG = (__force
> > > pci_dev_flags_t) 1, /* Device configuration is irrevocably lost if
> > > disabled into D3 */ PCI_DEV_FLAGS_NO_D3 = (__force pci_dev_flags_t)
> > > 2,
> > > +       /* Provide indication device is assigned by KVM */
> > > +       PCI_DEV_FLAGS_ASSIGNED = (__force pci_dev_flags_t) 4,
> > >  };
> > >
> > >  enum pci_irq_reroute_variant {
> > > diff --git a/virt/kvm/assigned-dev.c b/virt/kvm/assigned-dev.c
> > > index 4e9eaeb..eaf3a50 100644
> > > --- a/virt/kvm/assigned-dev.c
> > > +++ b/virt/kvm/assigned-dev.c
> > > @@ -205,6 +205,8 @@ static void kvm_free_assigned_device(struct kvm
> > > *kvm, else
> > >                pci_restore_state(assigned_dev->dev);
> > >
> > > +       assigned_dev->dev->dev_flags &= ~PCI_DEV_FLAGS_ASSIGNED;
> > > +
> > >        pci_release_regions(assigned_dev->dev);
> > >        pci_disable_device(assigned_dev->dev);
> > >        pci_dev_put(assigned_dev->dev);
> > > diff --git a/virt/kvm/iommu.c b/virt/kvm/iommu.c
> > > index 78c80f6..967aba1 100644
> > > --- a/virt/kvm/iommu.c
> > > +++ b/virt/kvm/iommu.c
> > > @@ -187,6 +187,8 @@ int kvm_assign_device(struct kvm *kvm,
> > >                        goto out_unmap;
> > >        }
> > >
> > > +       pdev->dev_flags |= PCI_DEV_FLAGS_ASSIGNED;
> > > +
> > >        printk(KERN_DEBUG "assign device %x:%x:%x.%x\n",
> > >                assigned_dev->host_segnr,
> > >                assigned_dev->host_busnr,
> > > @@ -215,6 +217,8 @@ int kvm_deassign_device(struct kvm *kvm,
> > >
> > >        iommu_detach_device(domain, &pdev->dev);
> > >
> > > +       pdev->dev_flags &= ~PCI_DEV_FLAGS_ASSIGNED;
> > > +
> > >        printk(KERN_DEBUG "deassign device %x:%x:%x.%x\n",
> > >                assigned_dev->host_segnr,
> > >                assigned_dev->host_busnr,
> > > --
> > > 1.7.6.2
> > 
> > Jesse/Konrad/Ian-
> > 
> > I sent this patch out as part of a pull request for David Miller's
> > net-next tree.  I know that Greg sent this originally out to the
> > linux-pci mailing list as a RFC. Since Greg also has a patch against
> > ixgbe which implemented this flag, I sent both patches for inclusion
> > into David Miller's net-next.
> > 
> > Dave is wanting to ensure that the PCI maintainers have reviewed this
> > and are ok with it before pulls my series of patches.
> 
> Yeah I think I looked at this one in the past, my ack must have been
> dropped.  You can add it if you want.
> 
> Jesse

Thanks Jesse!

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [net-next 1/8] pci: Add flag indicating device has been assigned by KVM

[Ian Campbell]

On Thu, 2011-09-22 at 21:16 -0700, Jeff Kirsher wrote:
> 
> 
> Jesse/Konrad/Ian-
> 
> I sent this patch out as part of a pull request for David Miller's
> net-next tree.  I know that Greg sent this originally out to the
> linux-pci mailing list as a RFC. Since Greg also has a patch against
> ixgbe which implemented this flag, I sent both patches for inclusion
> into David Miller's net-next.
> 
> Dave is wanting to ensure that the PCI maintainers have reviewed this
> and are ok with it before pulls my series of patches.

I'm not a PCI maintainer by any stretch of the imagination but FWIW this
change is fine by me.

My original reason for commenting on this patch was just to wonder
whether this would also be useful for Xen and I think the answer is we
should patch xen-pciback to use this new flag but I've not had time to
look into that.

I suppose by that measure the comment could be less KVM specific:
> +       /* Provide indication device is assigned by KVM */
> +       PCI_DEV_FLAGS_ASSIGNED = (__force pci_dev_flags_t) 4,

but that's not exactly a big deal.

I suppose really the flag indicates "VF in use" rather than necessarily
"assigned"? Would it be just as bad to have a VF driver in the host
active when the PF was unloaded?

Ian.

-- 
Ian Campbell
Current Noise: Mudhoney - Mudride (Live In Berlin)

Well, I think Perl should run faster than C.  :-)
		-- Larry Wall in <199801200306.TAA11638@wall.org>

RE: [net-next 1/8] pci: Add flag indicating device has been assigned by KVM

[Rose, Gregory V]

> -----Original Message-----
> From: Ian Campbell [mailto:ijc@hellion.org.uk]
> Sent: Friday, September 23, 2011 12:28 AM
> To: Kirsher, Jeffrey T
> Cc: davem@davemloft.net; konrad.wilk@oracle.com; Jesse Barnes; Rose,
> Gregory V; netdev@vger.kernel.org; gospo@redhat.com; linux-
> pci@vger.kernel.org
> Subject: Re: [net-next 1/8] pci: Add flag indicating device has been
> assigned by KVM
> 
> On Thu, 2011-09-22 at 21:16 -0700, Jeff Kirsher wrote:
> >
> >
> > Jesse/Konrad/Ian-
> >
> > I sent this patch out as part of a pull request for David Miller's
> > net-next tree.  I know that Greg sent this originally out to the
> > linux-pci mailing list as a RFC. Since Greg also has a patch against
> > ixgbe which implemented this flag, I sent both patches for inclusion
> > into David Miller's net-next.
> >
> > Dave is wanting to ensure that the PCI maintainers have reviewed this
> > and are ok with it before pulls my series of patches.
> 
> I'm not a PCI maintainer by any stretch of the imagination but FWIW this
> change is fine by me.
> 
> My original reason for commenting on this patch was just to wonder
> whether this would also be useful for Xen and I think the answer is we
> should patch xen-pciback to use this new flag but I've not had time to
> look into that.
> 
> I suppose by that measure the comment could be less KVM specific:
> > +       /* Provide indication device is assigned by KVM */
> > +       PCI_DEV_FLAGS_ASSIGNED = (__force pci_dev_flags_t) 4,

We can resubmit with a more generic comment, maybe this:

/* Provide indication device is assigned by a Virtual Machine Manager */

> 
> but that's not exactly a big deal.
> 
> I suppose really the flag indicates "VF in use" rather than necessarily
> "assigned"? Would it be just as bad to have a VF driver in the host
> active when the PF was unloaded?

There is no issue with unloading an active VF in the host because the hot remove event is seen by the host and accesses to the device stop.  However, when you hot remove the device the host and it is assigned to a VM the VM is unaware of the event and has no way of propagating the hot remove event.  So it really is a VF assigned to VM flag.  There's no need to mark VFs as in use in the host.

- Greg

RE: [net-next 1/8] pci: Add flag indicating device has been assigned by KVM

[Ian Campbell]

On Fri, 2011-09-23 at 07:41 -0700, Rose, Gregory V wrote:
> > -----Original Message-----
> > From: Ian Campbell [mailto:ijc@hellion.org.uk]
> > Sent: Friday, September 23, 2011 12:28 AM
> > To: Kirsher, Jeffrey T
> > Cc: davem@davemloft.net; konrad.wilk@oracle.com; Jesse Barnes; Rose,
> > Gregory V; netdev@vger.kernel.org; gospo@redhat.com; linux-
> > pci@vger.kernel.org
> > Subject: Re: [net-next 1/8] pci: Add flag indicating device has been
> > assigned by KVM
> > 
> > On Thu, 2011-09-22 at 21:16 -0700, Jeff Kirsher wrote:
> > >
> > >
> > > Jesse/Konrad/Ian-
> > >
> > > I sent this patch out as part of a pull request for David Miller's
> > > net-next tree.  I know that Greg sent this originally out to the
> > > linux-pci mailing list as a RFC. Since Greg also has a patch against
> > > ixgbe which implemented this flag, I sent both patches for inclusion
> > > into David Miller's net-next.
> > >
> > > Dave is wanting to ensure that the PCI maintainers have reviewed this
> > > and are ok with it before pulls my series of patches.
> > 
> > I'm not a PCI maintainer by any stretch of the imagination but FWIW this
> > change is fine by me.
> > 
> > My original reason for commenting on this patch was just to wonder
> > whether this would also be useful for Xen and I think the answer is we
> > should patch xen-pciback to use this new flag but I've not had time to
> > look into that.
> > 
> > I suppose by that measure the comment could be less KVM specific:
> > > +       /* Provide indication device is assigned by KVM */
> > > +       PCI_DEV_FLAGS_ASSIGNED = (__force pci_dev_flags_t) 4,
> 
> We can resubmit with a more generic comment, maybe this:
> 
> /* Provide indication device is assigned by a Virtual Machine Manager */

Sounds good to me.
> 
> > 
> > but that's not exactly a big deal.
> > 
> > I suppose really the flag indicates "VF in use" rather than necessarily
> > "assigned"? Would it be just as bad to have a VF driver in the host
> > active when the PF was unloaded?
> 
> There is no issue with unloading an active VF in the host because the
> hot remove event is seen by the host and accesses to the device stop.
> However, when you hot remove the device the host and it is assigned to
> a VM the VM is unaware of the event and has no way of propagating the
> hot remove event.  So it really is a VF assigned to VM flag.  There's
> no need to mark VFs as in use in the host.

Ah, ok. Thanks for explaining.

Ian.

-- 
Ian Campbell
Current Noise: High On Fire - Blessed Black Wings

Famous, adj.:
	Conspicuously miserable.
		-- Ambrose Bierce, "The Devil's Dictionary"

RE: [net-next 1/8] pci: Add flag indicating device has been assigned by KVM

[Rose, Gregory V]

> -----Original Message-----
> From: Ian Campbell [mailto:ijc@hellion.org.uk]
> Sent: Friday, September 23, 2011 8:04 AM
> To: Rose, Gregory V
> Cc: Kirsher, Jeffrey T; davem@davemloft.net; konrad.wilk@oracle.com; Jesse
> Barnes; netdev@vger.kernel.org; gospo@redhat.com; linux-
> pci@vger.kernel.org
> Subject: RE: [net-next 1/8] pci: Add flag indicating device has been
> assigned by KVM
> 
> On Fri, 2011-09-23 at 07:41 -0700, Rose, Gregory V wrote:
> > > -----Original Message-----
> > > From: Ian Campbell [mailto:ijc@hellion.org.uk]
> > > Sent: Friday, September 23, 2011 12:28 AM
> > > To: Kirsher, Jeffrey T
> > > Cc: davem@davemloft.net; konrad.wilk@oracle.com; Jesse Barnes; Rose,
> > > Gregory V; netdev@vger.kernel.org; gospo@redhat.com; linux-
> > > pci@vger.kernel.org
> > > Subject: Re: [net-next 1/8] pci: Add flag indicating device has been
> > > assigned by KVM
> > >
> > > I suppose by that measure the comment could be less KVM specific:
> > > > +       /* Provide indication device is assigned by KVM */
> > > > +       PCI_DEV_FLAGS_ASSIGNED = (__force pci_dev_flags_t) 4,
> >
> > We can resubmit with a more generic comment, maybe this:
> >
> > /* Provide indication device is assigned by a Virtual Machine Manager */
> 
> Sounds good to me.

Dave, Jeff,

Should I resubmit the patch or would it be more convenient to post a follow on patch that fixes up the comment?  Either way is fine by me.

- Greg

RE: [net-next 1/8] pci: Add flag indicating device has been assigned by KVM

[Jeff Kirsher]

[-- Attachment #1: Type: text/plain, Size: 1680 bytes --]

On Fri, 2011-09-23 at 08:11 -0700, Rose, Gregory V wrote:
> > -----Original Message-----
> > From: Ian Campbell [mailto:ijc@hellion.org.uk]
> > Sent: Friday, September 23, 2011 8:04 AM
> > To: Rose, Gregory V
> > Cc: Kirsher, Jeffrey T; davem@davemloft.net; konrad.wilk@oracle.com; Jesse
> > Barnes; netdev@vger.kernel.org; gospo@redhat.com; linux-
> > pci@vger.kernel.org
> > Subject: RE: [net-next 1/8] pci: Add flag indicating device has been
> > assigned by KVM
> > 
> > On Fri, 2011-09-23 at 07:41 -0700, Rose, Gregory V wrote:
> > > > -----Original Message-----
> > > > From: Ian Campbell [mailto:ijc@hellion.org.uk]
> > > > Sent: Friday, September 23, 2011 12:28 AM
> > > > To: Kirsher, Jeffrey T
> > > > Cc: davem@davemloft.net; konrad.wilk@oracle.com; Jesse Barnes; Rose,
> > > > Gregory V; netdev@vger.kernel.org; gospo@redhat.com; linux-
> > > > pci@vger.kernel.org
> > > > Subject: Re: [net-next 1/8] pci: Add flag indicating device has been
> > > > assigned by KVM
> > > >
> > > > I suppose by that measure the comment could be less KVM specific:
> > > > > +       /* Provide indication device is assigned by KVM */
> > > > > +       PCI_DEV_FLAGS_ASSIGNED = (__force pci_dev_flags_t) 4,
> > >
> > > We can resubmit with a more generic comment, maybe this:
> > >
> > > /* Provide indication device is assigned by a Virtual Machine Manager */
> > 
> > Sounds good to me.
> 
> Dave, Jeff,
> 
> Should I resubmit the patch or would it be more convenient to post a follow on patch that fixes up the comment?  Either way is fine by me.
> 
> - Greg
> 

let's fix up the patch in my tree and add Jesse Barnes's ACK at the same
time.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [net-next 1/8] pci: Add flag indicating device has been assigned by KVM

[Konrad Rzeszutek Wilk]

On Fri, Sep 23, 2011 at 08:45:09AM -0700, Jeff Kirsher wrote:
> On Fri, 2011-09-23 at 08:11 -0700, Rose, Gregory V wrote:
> > > -----Original Message-----
> > > From: Ian Campbell [mailto:ijc@hellion.org.uk]
> > > Sent: Friday, September 23, 2011 8:04 AM
> > > To: Rose, Gregory V
> > > Cc: Kirsher, Jeffrey T; davem@davemloft.net; konrad.wilk@oracle.com; Jesse
> > > Barnes; netdev@vger.kernel.org; gospo@redhat.com; linux-
> > > pci@vger.kernel.org
> > > Subject: RE: [net-next 1/8] pci: Add flag indicating device has been
> > > assigned by KVM
> > > 
> > > On Fri, 2011-09-23 at 07:41 -0700, Rose, Gregory V wrote:
> > > > > -----Original Message-----
> > > > > From: Ian Campbell [mailto:ijc@hellion.org.uk]
> > > > > Sent: Friday, September 23, 2011 12:28 AM
> > > > > To: Kirsher, Jeffrey T
> > > > > Cc: davem@davemloft.net; konrad.wilk@oracle.com; Jesse Barnes; Rose,
> > > > > Gregory V; netdev@vger.kernel.org; gospo@redhat.com; linux-
> > > > > pci@vger.kernel.org
> > > > > Subject: Re: [net-next 1/8] pci: Add flag indicating device has been
> > > > > assigned by KVM
> > > > >
> > > > > I suppose by that measure the comment could be less KVM specific:
> > > > > > +       /* Provide indication device is assigned by KVM */
> > > > > > +       PCI_DEV_FLAGS_ASSIGNED = (__force pci_dev_flags_t) 4,
> > > >
> > > > We can resubmit with a more generic comment, maybe this:
> > > >
> > > > /* Provide indication device is assigned by a Virtual Machine Manager */
> > > 
> > > Sounds good to me.
> > 
> > Dave, Jeff,
> > 
> > Should I resubmit the patch or would it be more convenient to post a follow on patch that fixes up the comment?  Either way is fine by me.
> > 
> > - Greg
> > 
> 
> let's fix up the patch in my tree and add Jesse Barnes's ACK at the same
> time.

Were are the patches now? I am kind of curious to see the changes to the xen pciback.

RE: [net-next 1/8] pci: Add flag indicating device has been assigned by KVM

[Rose, Gregory V]

> -----Original Message-----
> From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
> Sent: Friday, September 23, 2011 9:43 AM
> To: Kirsher, Jeffrey T
> Cc: Rose, Gregory V; Ian Campbell; davem@davemloft.net; Jesse Barnes;
> netdev@vger.kernel.org; gospo@redhat.com; linux-pci@vger.kernel.org
> Subject: Re: [net-next 1/8] pci: Add flag indicating device has been
> assigned by KVM
> 
> On Fri, Sep 23, 2011 at 08:45:09AM -0700, Jeff Kirsher wrote:
> > On Fri, 2011-09-23 at 08:11 -0700, Rose, Gregory V wrote:
> > > > -----Original Message-----
> > > > From: Ian Campbell [mailto:ijc@hellion.org.uk]
> > > > Sent: Friday, September 23, 2011 8:04 AM
> > > > To: Rose, Gregory V
> > > > Cc: Kirsher, Jeffrey T; davem@davemloft.net; konrad.wilk@oracle.com;
> Jesse
> > > > Barnes; netdev@vger.kernel.org; gospo@redhat.com; linux-
> > > > pci@vger.kernel.org
> > > > Subject: RE: [net-next 1/8] pci: Add flag indicating device has been
> > > > assigned by KVM
> > > >
> > > > On Fri, 2011-09-23 at 07:41 -0700, Rose, Gregory V wrote:
> > > > > > -----Original Message-----
> > > > > > From: Ian Campbell [mailto:ijc@hellion.org.uk]
> > > > > > Sent: Friday, September 23, 2011 12:28 AM
> > > > > > To: Kirsher, Jeffrey T
> > > > > > Cc: davem@davemloft.net; konrad.wilk@oracle.com; Jesse Barnes;
> Rose,
> > > > > > Gregory V; netdev@vger.kernel.org; gospo@redhat.com; linux-
> > > > > > pci@vger.kernel.org
> > > > > > Subject: Re: [net-next 1/8] pci: Add flag indicating device has
> been
> > > > > > assigned by KVM
> > > > > >
> > > > > > I suppose by that measure the comment could be less KVM
> specific:
> > > > > > > +       /* Provide indication device is assigned by KVM */
> > > > > > > +       PCI_DEV_FLAGS_ASSIGNED = (__force pci_dev_flags_t) 4,
> > > > >
> > > > > We can resubmit with a more generic comment, maybe this:
> > > > >
> > > > > /* Provide indication device is assigned by a Virtual Machine
> Manager */
> > > >
> > > > Sounds good to me.
> > >
> > > Dave, Jeff,
> > >
> > > Should I resubmit the patch or would it be more convenient to post a
> follow on patch that fixes up the comment?  Either way is fine by me.
> > >
> > > - Greg
> > >
> >
> > let's fix up the patch in my tree and add Jesse Barnes's ACK at the same
> > time.
> 
> Were are the patches now? I am kind of curious to see the changes to the
> xen pciback.
> 
> 

Version 2 of this patch with the modified comment was just posted to netdev.  Ian Campbell had mentioned that the Xen pciback driver should be modified to use this flag but I don't believe any patches for it have been created yet.

- Greg
 

Re: [net-next 1/8] pci: Add flag indicating device has been assigned by KVM

[Konrad Rzeszutek Wilk]

On Fri, Sep 23, 2011 at 10:04:59AM -0700, Rose, Gregory V wrote:
> > -----Original Message-----
> > From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
> > Sent: Friday, September 23, 2011 9:43 AM
> > To: Kirsher, Jeffrey T
> > Cc: Rose, Gregory V; Ian Campbell; davem@davemloft.net; Jesse Barnes;
> > netdev@vger.kernel.org; gospo@redhat.com; linux-pci@vger.kernel.org
> > Subject: Re: [net-next 1/8] pci: Add flag indicating device has been
> > assigned by KVM
> > 
> > On Fri, Sep 23, 2011 at 08:45:09AM -0700, Jeff Kirsher wrote:
> > > On Fri, 2011-09-23 at 08:11 -0700, Rose, Gregory V wrote:
> > > > > -----Original Message-----
> > > > > From: Ian Campbell [mailto:ijc@hellion.org.uk]
> > > > > Sent: Friday, September 23, 2011 8:04 AM
> > > > > To: Rose, Gregory V
> > > > > Cc: Kirsher, Jeffrey T; davem@davemloft.net; konrad.wilk@oracle.com;
> > Jesse
> > > > > Barnes; netdev@vger.kernel.org; gospo@redhat.com; linux-
> > > > > pci@vger.kernel.org
> > > > > Subject: RE: [net-next 1/8] pci: Add flag indicating device has been
> > > > > assigned by KVM
> > > > >
> > > > > On Fri, 2011-09-23 at 07:41 -0700, Rose, Gregory V wrote:
> > > > > > > -----Original Message-----
> > > > > > > From: Ian Campbell [mailto:ijc@hellion.org.uk]
> > > > > > > Sent: Friday, September 23, 2011 12:28 AM
> > > > > > > To: Kirsher, Jeffrey T
> > > > > > > Cc: davem@davemloft.net; konrad.wilk@oracle.com; Jesse Barnes;
> > Rose,
> > > > > > > Gregory V; netdev@vger.kernel.org; gospo@redhat.com; linux-
> > > > > > > pci@vger.kernel.org
> > > > > > > Subject: Re: [net-next 1/8] pci: Add flag indicating device has
> > been
> > > > > > > assigned by KVM
> > > > > > >
> > > > > > > I suppose by that measure the comment could be less KVM
> > specific:
> > > > > > > > +       /* Provide indication device is assigned by KVM */
> > > > > > > > +       PCI_DEV_FLAGS_ASSIGNED = (__force pci_dev_flags_t) 4,
> > > > > >
> > > > > > We can resubmit with a more generic comment, maybe this:
> > > > > >
> > > > > > /* Provide indication device is assigned by a Virtual Machine
> > Manager */
> > > > >
> > > > > Sounds good to me.
> > > >
> > > > Dave, Jeff,
> > > >
> > > > Should I resubmit the patch or would it be more convenient to post a
> > follow on patch that fixes up the comment?  Either way is fine by me.
> > > >
> > > > - Greg
> > > >
> > >
> > > let's fix up the patch in my tree and add Jesse Barnes's ACK at the same
> > > time.
> > 
> > Were are the patches now? I am kind of curious to see the changes to the
> > xen pciback.
> > 
> > 
> 
> Version 2 of this patch with the modified comment was just posted to netdev.  Ian Campbell had mentioned that the Xen pciback driver should be modified to use this flag but I don't believe any patches for it have been created yet.

http://marc.info/?i=20110729165446.GA6731@dumpdata.com

?

RE: [net-next 1/8] pci: Add flag indicating device has been assigned by KVM

[Rose, Gregory V]

> -----Original Message-----
> From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
> Sent: Friday, September 23, 2011 11:00 AM
> To: Rose, Gregory V
> Cc: Kirsher, Jeffrey T; Ian Campbell; davem@davemloft.net; Jesse Barnes;
> netdev@vger.kernel.org; gospo@redhat.com; linux-pci@vger.kernel.org
> Subject: Re: [net-next 1/8] pci: Add flag indicating device has been
> assigned by KVM
> 
> On Fri, Sep 23, 2011 at 10:04:59AM -0700, Rose, Gregory V wrote:
> > > -----Original Message-----
> > > From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
> > > Sent: Friday, September 23, 2011 9:43 AM
> > > To: Kirsher, Jeffrey T
> > > Cc: Rose, Gregory V; Ian Campbell; davem@davemloft.net; Jesse Barnes;
> > > netdev@vger.kernel.org; gospo@redhat.com; linux-pci@vger.kernel.org
> > > Subject: Re: [net-next 1/8] pci: Add flag indicating device has been
> > > assigned by KVM
> > >
> > > On Fri, Sep 23, 2011 at 08:45:09AM -0700, Jeff Kirsher wrote:
> > > > On Fri, 2011-09-23 at 08:11 -0700, Rose, Gregory V wrote:
> > > > > > -----Original Message-----
> > > > > > From: Ian Campbell [mailto:ijc@hellion.org.uk]
> > > > > > Sent: Friday, September 23, 2011 8:04 AM
> > > > > > To: Rose, Gregory V
> > > > > > Cc: Kirsher, Jeffrey T; davem@davemloft.net;
> konrad.wilk@oracle.com;
> > > Jesse
> > > > > > Barnes; netdev@vger.kernel.org; gospo@redhat.com; linux-
> > > > > > pci@vger.kernel.org
> > > > > > Subject: RE: [net-next 1/8] pci: Add flag indicating device has
> been
> > > > > > assigned by KVM
> > > > > >
> > > > > > On Fri, 2011-09-23 at 07:41 -0700, Rose, Gregory V wrote:
> > > > > > > > -----Original Message-----
> > > > > > > > From: Ian Campbell [mailto:ijc@hellion.org.uk]
> > > > > > > > Sent: Friday, September 23, 2011 12:28 AM
> > > > > > > > To: Kirsher, Jeffrey T
> > > > > > > > Cc: davem@davemloft.net; konrad.wilk@oracle.com; Jesse
> Barnes;
> > > Rose,
> > > > > > > > Gregory V; netdev@vger.kernel.org; gospo@redhat.com; linux-
> > > > > > > > pci@vger.kernel.org
> > > > > > > > Subject: Re: [net-next 1/8] pci: Add flag indicating device
> has
> > > been
> > > > > > > > assigned by KVM
> > > > > > > >
> > > > > > > > I suppose by that measure the comment could be less KVM
> > > specific:
> > > > > > > > > +       /* Provide indication device is assigned by KVM */
> > > > > > > > > +       PCI_DEV_FLAGS_ASSIGNED = (__force pci_dev_flags_t)
> 4,
> > > > > > >
> > > > > > > We can resubmit with a more generic comment, maybe this:
> > > > > > >
> > > > > > > /* Provide indication device is assigned by a Virtual Machine
> > > Manager */
> > > > > >
> > > > > > Sounds good to me.
> > > > >
> > > > > Dave, Jeff,
> > > > >
> > > > > Should I resubmit the patch or would it be more convenient to post
> a
> > > follow on patch that fixes up the comment?  Either way is fine by me.
> > > > >
> > > > > - Greg
> > > > >
> > > >
> > > > let's fix up the patch in my tree and add Jesse Barnes's ACK at the
> same
> > > > time.
> > >
> > > Were are the patches now? I am kind of curious to see the changes to
> the
> > > xen pciback.
> > >
> > >
> >
> > Version 2 of this patch with the modified comment was just posted to
> netdev.  Ian Campbell had mentioned that the Xen pciback driver should be
> modified to use this flag but I don't believe any patches for it have been
> created yet.
> 
> http://marc.info/?i=20110729165446.GA6731@dumpdata.com
> 

I'm not sure who generated the patches to xen-pciback.  It wasn't me and I couldn't tell from the links you provided who did generate that patch.  I don't see it now though.

- Greg

Re: [net-next 1/8] pci: Add flag indicating device has been assigned by KVM

[Konrad Rzeszutek Wilk]

> > > Version 2 of this patch with the modified comment was just posted to
> > netdev.  Ian Campbell had mentioned that the Xen pciback driver should be
> > modified to use this flag but I don't believe any patches for it have been
> > created yet.
> > 
> > http://marc.info/?i=20110729165446.GA6731@dumpdata.com
> > 
> 
> I'm not sure who generated the patches to xen-pciback.  It wasn't me and I couldn't tell from the links you provided who did generate that patch.  I don't see it now though.

I provided it.

And I asked for a git tree so I could double check that it actually
worked as I don't have the full set of your patches.

Can you provide me a link to your public git tree or just repost it once more?
> 
> - Greg

RE: [net-next 1/8] pci: Add flag indicating device has been assigned by KVM

[Rose, Gregory V]

> -----Original Message-----
> From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
> Sent: Friday, September 23, 2011 12:06 PM
> To: Rose, Gregory V
> Cc: Kirsher, Jeffrey T; Ian Campbell; davem@davemloft.net; Jesse Barnes;
> netdev@vger.kernel.org; gospo@redhat.com; linux-pci@vger.kernel.org
> Subject: Re: [net-next 1/8] pci: Add flag indicating device has been
> assigned by KVM
> 
> > > > Version 2 of this patch with the modified comment was just posted to
> > > netdev.  Ian Campbell had mentioned that the Xen pciback driver should
> be
> > > modified to use this flag but I don't believe any patches for it have
> been
> > > created yet.
> > >
> > > http://marc.info/?i=20110729165446.GA6731@dumpdata.com
> > >
> >
> > I'm not sure who generated the patches to xen-pciback.  It wasn't me and
> I couldn't tell from the links you provided who did generate that patch.
> I don't see it now though.
> 
> I provided it.
> 
> And I asked for a git tree so I could double check that it actually
> worked as I don't have the full set of your patches.
> 
> Can you provide me a link to your public git tree or just repost it once
> more?

Ah, I get it now.

Jeff Kirsher is the maintainer of our git trees.  Hopefully he'll come around and provide that info for you soon.

- Greg

Re: [net-next 1/8] pci: Add flag indicating device has been assigned by KVM

[Jeff Kirsher]

[-- Attachment #1: Type: text/plain, Size: 1085 bytes --]

On Fri, 2011-09-23 at 12:05 -0700, Konrad Rzeszutek Wilk wrote:
> > > > Version 2 of this patch with the modified comment was just posted to
> > > netdev.  Ian Campbell had mentioned that the Xen pciback driver should be
> > > modified to use this flag but I don't believe any patches for it have been
> > > created yet.
> > > 
> > > http://marc.info/?i=20110729165446.GA6731@dumpdata.com
> > > 
> > 
> > I'm not sure who generated the patches to xen-pciback.  It wasn't me and I couldn't tell from the links you provided who did generate that patch.  I don't see it now though.
> 
> I provided it.
> 
> And I asked for a git tree so I could double check that it actually
> worked as I don't have the full set of your patches.
> 
> Can you provide me a link to your public git tree or just repost it once more?
> > 
> > - Greg

You can pull my tree or David Miller's net-next tree.  Both have this
patch now.  Here are the URL's

David Miller's net-next tree:
git://github.com/davem330/net-next.git

My net-next tree:
git://github.com/Jkirsher/net-next.git

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [net-next 1/8] pci: Add flag indicating device has been assigned by KVM [and Xen]

[Konrad Rzeszutek Wilk]

On Fri, Sep 23, 2011 at 09:41:39PM -0700, Jeff Kirsher wrote:
> On Fri, 2011-09-23 at 12:05 -0700, Konrad Rzeszutek Wilk wrote:
> > > > > Version 2 of this patch with the modified comment was just posted to
> > > > netdev.  Ian Campbell had mentioned that the Xen pciback driver should be
> > > > modified to use this flag but I don't believe any patches for it have been
> > > > created yet.
> > > > 
> > > > http://marc.info/?i=20110729165446.GA6731@dumpdata.com
..
Any plans to support bnx4 and igb with this feature? Any ETA?

Dave,

Please also add this patch to your net-next queue.

>From 53cada5f5f614a856f6ea5a09dc565d6a455d7c2 Mon Sep 17 00:00:00 2001
From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Date: Mon, 26 Sep 2011 12:22:01 -0400
Subject: [PATCH] xen/pciback: Add flag indicating device has been assigned by Xen

Device drivers that create and destroy SR-IOV virtual functions via
calls to pci_enable_sriov() and pci_disable_sriov can cause catastrophic
failures if they attempt to destroy VFs while they are assigned to
guest virtual machines.  By adding a flag for use by the Xen PCI back
to indicate that a device is assigned a device driver can check that
flag and avoid destroying VFs while they are assigned and avoid system
failures.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/xen-pciback/xenbus.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/drivers/xen/xen-pciback/xenbus.c b/drivers/xen/xen-pciback/xenbus.c
index 978d2c6..18db31f 100644
--- a/drivers/xen/xen-pciback/xenbus.c
+++ b/drivers/xen/xen-pciback/xenbus.c
@@ -249,6 +249,7 @@ static int xen_pcibk_export_device(struct xen_pcibk_device *pdev,
 		goto out;
 
 	dev_dbg(&dev->dev, "registering for %d\n", pdev->xdev->otherend_id);
+	dev->dev_flags |= PCI_DEV_FLAGS_ASSIGNED;
 	if (xen_register_device_domain_owner(dev,
 					     pdev->xdev->otherend_id) != 0) {
 		dev_err(&dev->dev, "device has been assigned to another " \
@@ -288,6 +289,7 @@ static int xen_pcibk_remove_device(struct xen_pcibk_device *pdev,
 	}
 
 	dev_dbg(&dev->dev, "unregistering for %d\n", pdev->xdev->otherend_id);
+	dev->dev_flags &= ~PCI_DEV_FLAGS_ASSIGNED;
 	xen_unregister_device_domain_owner(dev);
 
 	xen_pcibk_release_pci_dev(pdev, dev);
-- 
1.7.4.1

RE: [net-next 1/8] pci: Add flag indicating device has been assigned by KVM [and Xen]

[Rose, Gregory V]

> -----Original Message-----
> From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
> Sent: Monday, September 26, 2011 9:26 AM
> To: Kirsher, Jeffrey T; davem@davemloft.net
> Cc: Rose, Gregory V; Ian Campbell; davem@davemloft.net; Jesse Barnes;
> netdev@vger.kernel.org; gospo@redhat.com; linux-pci@vger.kernel.org
> Subject: Re: [net-next 1/8] pci: Add flag indicating device has been
> assigned by KVM [and Xen]
> 
> On Fri, Sep 23, 2011 at 09:41:39PM -0700, Jeff Kirsher wrote:
> > On Fri, 2011-09-23 at 12:05 -0700, Konrad Rzeszutek Wilk wrote:
> > > > > > Version 2 of this patch with the modified comment was just
> posted to
> > > > > netdev.  Ian Campbell had mentioned that the Xen pciback driver
> should be
> > > > > modified to use this flag but I don't believe any patches for it
> have been
> > > > > created yet.
> > > > >
> > > > > http://marc.info/?i=20110729165446.GA6731@dumpdata.com
> ..
> Any plans to support bnx4 and igb with this feature?

I'm updating the igb driver to support this feature right now.  I would hesitate to get into other drivers and muck with them because the changes to support this feature are not trivial and also involve policy decisions that I'd prefer to leave up to the folks who maintain those drivers.  Plus I have no other vendor hardware available to test the changes.

> Any ETA?

I should have the igb driver work done in the next week or so.

- Greg

Re: [net-next 1/8] pci: Add flag indicating device has been assigned by KVM [and Xen]

[Konrad Rzeszutek Wilk]

> > Any plans to support bnx4 and igb with this feature?
> 
> I'm updating the igb driver to support this feature right now.  I would hesitate to get into other drivers and muck with them because the changes to support this feature are not trivial and also involve policy decisions that I'd prefer to leave up to the folks who maintain those drivers.  Plus I have no other vendor hardware available to test the changes.
> 

Sounds fair. Did you CC them?

> > Any ETA?
> 
> I should have the igb driver work done in the next week or so.

Ok, please CC me on the code when you post it.
> 
> - Greg

RE: [net-next 1/8] pci: Add flag indicating device has been assigned by KVM [and Xen]

[Rose, Gregory V]

> -----Original Message-----
> From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
> Sent: Monday, September 26, 2011 9:43 AM
> To: Rose, Gregory V
> Cc: Kirsher, Jeffrey T; davem@davemloft.net; Ian Campbell; Jesse Barnes;
> netdev@vger.kernel.org; gospo@redhat.com; linux-pci@vger.kernel.org
> Subject: Re: [net-next 1/8] pci: Add flag indicating device has been
> assigned by KVM [and Xen]
> 
> > > Any plans to support bnx4 and igb with this feature?
> >
> > I'm updating the igb driver to support this feature right now.  I would
> hesitate to get into other drivers and muck with them because the changes
> to support this feature are not trivial and also involve policy decisions
> that I'd prefer to leave up to the folks who maintain those drivers.  Plus
> I have no other vendor hardware available to test the changes.
> >
> 
> Sounds fair. Did you CC them?

I'll grep through the drivers and find who calls pci_disable_sriov() and send the respective driver maintainers a notification letting them know of the patches.

> 
> > > Any ETA?
> >
> > I should have the igb driver work done in the next week or so.
> 
> Ok, please CC me on the code when you post it.

Will do.

- Greg

[net-next 2/8] ixgbe: Reconfigure SR-IOV Init

[Jeff Kirsher]

From: Greg Rose <gregory.v.rose@intel.com>

Use the PCI device flag indicating if a VF is assigned to a guest VM
to guard against destroying VFs upon driver removal.  Implement
additional feature to detect if VFs already exist when the driver
is loaded and if so configure them and set the driver state to
SR-IOV enabled.

Signed-off-by: Greg Rose <gregory.v.rose@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
---
 drivers/net/ethernet/intel/ixgbe/ixgbe.h       |    1 +
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c  |  105 +-----------
 drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.c |  207 +++++++++++++++++++++++-
 drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.h |    5 +
 drivers/net/ethernet/intel/ixgbe/ixgbe_type.h  |    4 +
 5 files changed, 225 insertions(+), 97 deletions(-)

diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe.h b/drivers/net/ethernet/intel/ixgbe/ixgbe.h
index 2c9fdf8..b43b2cd 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe.h
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe.h
@@ -128,6 +128,7 @@ struct vf_data_storage {
 	u16 pf_vlan; /* When set, guest VLAN config not allowed. */
 	u16 pf_qos;
 	u16 tx_rate;
+	struct pci_dev *vfdev;
 };
 
 struct vf_macvlans {
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
index 49e82de..8b86c41 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
@@ -134,42 +134,6 @@ MODULE_VERSION(DRV_VERSION);
 
 #define DEFAULT_DEBUG_LEVEL_SHIFT 3
 
-static inline void ixgbe_disable_sriov(struct ixgbe_adapter *adapter)
-{
-	struct ixgbe_hw *hw = &adapter->hw;
-	u32 gcr;
-	u32 gpie;
-	u32 vmdctl;
-
-#ifdef CONFIG_PCI_IOV
-	/* disable iov and allow time for transactions to clear */
-	pci_disable_sriov(adapter->pdev);
-#endif
-
-	/* turn off device IOV mode */
-	gcr = IXGBE_READ_REG(hw, IXGBE_GCR_EXT);
-	gcr &= ~(IXGBE_GCR_EXT_SRIOV);
-	IXGBE_WRITE_REG(hw, IXGBE_GCR_EXT, gcr);
-	gpie = IXGBE_READ_REG(hw, IXGBE_GPIE);
-	gpie &= ~IXGBE_GPIE_VTMODE_MASK;
-	IXGBE_WRITE_REG(hw, IXGBE_GPIE, gpie);
-
-	/* set default pool back to 0 */
-	vmdctl = IXGBE_READ_REG(hw, IXGBE_VT_CTL);
-	vmdctl &= ~IXGBE_VT_CTL_POOL_MASK;
-	IXGBE_WRITE_REG(hw, IXGBE_VT_CTL, vmdctl);
-	IXGBE_WRITE_FLUSH(hw);
-
-	/* take a breather then clean up driver data */
-	msleep(100);
-
-	kfree(adapter->vfinfo);
-	adapter->vfinfo = NULL;
-
-	adapter->num_vfs = 0;
-	adapter->flags &= ~IXGBE_FLAG_SRIOV_ENABLED;
-}
-
 static void ixgbe_service_event_schedule(struct ixgbe_adapter *adapter)
 {
 	if (!test_bit(__IXGBE_DOWN, &adapter->state) &&
@@ -7064,11 +7028,8 @@ static void __devinit ixgbe_probe_vf(struct ixgbe_adapter *adapter,
 {
 #ifdef CONFIG_PCI_IOV
 	struct ixgbe_hw *hw = &adapter->hw;
-	int err;
-	int num_vf_macvlans, i;
-	struct vf_macvlans *mv_list;
 
-	if (hw->mac.type == ixgbe_mac_82598EB || !max_vfs)
+	if (hw->mac.type == ixgbe_mac_82598EB)
 		return;
 
 	/* The 82599 supports up to 64 VFs per physical function
@@ -7077,60 +7038,7 @@ static void __devinit ixgbe_probe_vf(struct ixgbe_adapter *adapter,
 	 * physical function
 	 */
 	adapter->num_vfs = (max_vfs > 63) ? 63 : max_vfs;
-	adapter->flags |= IXGBE_FLAG_SRIOV_ENABLED;
-	err = pci_enable_sriov(adapter->pdev, adapter->num_vfs);
-	if (err) {
-		e_err(probe, "Failed to enable PCI sriov: %d\n", err);
-		goto err_novfs;
-	}
-
-	num_vf_macvlans = hw->mac.num_rar_entries -
-		(IXGBE_MAX_PF_MACVLANS + 1 + adapter->num_vfs);
-
-	adapter->mv_list = mv_list = kcalloc(num_vf_macvlans,
-					     sizeof(struct vf_macvlans),
-					     GFP_KERNEL);
-	if (mv_list) {
-		/* Initialize list of VF macvlans */
-		INIT_LIST_HEAD(&adapter->vf_mvs.l);
-		for (i = 0; i < num_vf_macvlans; i++) {
-			mv_list->vf = -1;
-			mv_list->free = true;
-			mv_list->rar_entry = hw->mac.num_rar_entries -
-				(i + adapter->num_vfs + 1);
-			list_add(&mv_list->l, &adapter->vf_mvs.l);
-			mv_list++;
-		}
-	}
-
-	/* If call to enable VFs succeeded then allocate memory
-	 * for per VF control structures.
-	 */
-	adapter->vfinfo =
-		kcalloc(adapter->num_vfs,
-			sizeof(struct vf_data_storage), GFP_KERNEL);
-	if (adapter->vfinfo) {
-		/* Now that we're sure SR-IOV is enabled
-		 * and memory allocated set up the mailbox parameters
-		 */
-		ixgbe_init_mbx_params_pf(hw);
-		memcpy(&hw->mbx.ops, ii->mbx_ops,
-		       sizeof(hw->mbx.ops));
-
-		/* Disable RSC when in SR-IOV mode */
-		adapter->flags2 &= ~(IXGBE_FLAG2_RSC_CAPABLE |
-				     IXGBE_FLAG2_RSC_ENABLED);
-		return;
-	}
-
-	/* Oh oh */
-	e_err(probe, "Unable to allocate memory for VF Data Storage - "
-	      "SRIOV disabled\n");
-	pci_disable_sriov(adapter->pdev);
-
-err_novfs:
-	adapter->flags &= ~IXGBE_FLAG_SRIOV_ENABLED;
-	adapter->num_vfs = 0;
+	ixgbe_enable_sriov(adapter, ii);
 #endif /* CONFIG_PCI_IOV */
 }
 
@@ -7580,8 +7488,13 @@ static void __devexit ixgbe_remove(struct pci_dev *pdev)
 	if (netdev->reg_state == NETREG_REGISTERED)
 		unregister_netdev(netdev);
 
-	if (adapter->flags & IXGBE_FLAG_SRIOV_ENABLED)
-		ixgbe_disable_sriov(adapter);
+	if (adapter->flags & IXGBE_FLAG_SRIOV_ENABLED) {
+		if (!(ixgbe_check_vf_assignment(adapter)))
+			ixgbe_disable_sriov(adapter);
+		else
+			e_dev_warn("Unloading driver while VFs are assigned "
+				   "- VFs will not be deallocated\n");
+	}
 
 	ixgbe_clear_interrupt_scheme(adapter);
 
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.c
index d99d01e..468ddd0 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.c
@@ -40,9 +40,174 @@
 #endif
 
 #include "ixgbe.h"
-
+#include "ixgbe_type.h"
 #include "ixgbe_sriov.h"
 
+#ifdef CONFIG_PCI_IOV
+static int ixgbe_find_enabled_vfs(struct ixgbe_adapter *adapter)
+{
+	struct pci_dev *pdev = adapter->pdev;
+	struct pci_dev *pvfdev;
+	u16 vf_devfn = 0;
+	int device_id;
+	int vfs_found = 0;
+
+	switch (adapter->hw.mac.type) {
+	case ixgbe_mac_82599EB:
+		device_id = IXGBE_DEV_ID_82599_VF;
+		break;
+	case ixgbe_mac_X540:
+		device_id = IXGBE_DEV_ID_X540_VF;
+		break;
+	default:
+		device_id = 0;
+		break;
+	}
+
+	vf_devfn = pdev->devfn + 0x80;
+	pvfdev = pci_get_device(IXGBE_INTEL_VENDOR_ID, device_id, NULL);
+	while (pvfdev) {
+		if (pvfdev->devfn == vf_devfn)
+			vfs_found++;
+		vf_devfn += 2;
+		pvfdev = pci_get_device(IXGBE_INTEL_VENDOR_ID,
+					device_id, pvfdev);
+	}
+
+	return vfs_found;
+}
+
+void ixgbe_enable_sriov(struct ixgbe_adapter *adapter,
+			 const struct ixgbe_info *ii)
+{
+	struct ixgbe_hw *hw = &adapter->hw;
+	int err = 0;
+	int num_vf_macvlans, i;
+	struct vf_macvlans *mv_list;
+	int pre_existing_vfs = 0;
+
+	pre_existing_vfs = ixgbe_find_enabled_vfs(adapter);
+	if (!pre_existing_vfs && !adapter->num_vfs)
+		return;
+
+	/* If there are pre-existing VFs then we have to force
+	 * use of that many because they were not deleted the last
+	 * time someone removed the PF driver.  That would have
+	 * been because they were allocated to guest VMs and can't
+	 * be removed.  Go ahead and just re-enable the old amount.
+	 * If the user wants to change the number of VFs they can
+	 * use ethtool while making sure no VFs are allocated to
+	 * guest VMs... i.e. the right way.
+	 */
+	if (pre_existing_vfs) {
+		adapter->num_vfs = pre_existing_vfs;
+		dev_warn(&adapter->pdev->dev, "Virtual Functions already "
+			 "enabled for this device - Please reload all "
+			 "VF drivers to avoid spoofed packet errors\n");
+	} else {
+		err = pci_enable_sriov(adapter->pdev, adapter->num_vfs);
+	}
+	if (err) {
+		e_err(probe, "Failed to enable PCI sriov: %d\n", err);
+		goto err_novfs;
+	}
+	adapter->flags |= IXGBE_FLAG_SRIOV_ENABLED;
+
+	e_info(probe, "SR-IOV enabled with %d VFs\n", adapter->num_vfs);
+
+	num_vf_macvlans = hw->mac.num_rar_entries -
+	(IXGBE_MAX_PF_MACVLANS + 1 + adapter->num_vfs);
+
+	adapter->mv_list = mv_list = kcalloc(num_vf_macvlans,
+					     sizeof(struct vf_macvlans),
+					     GFP_KERNEL);
+	if (mv_list) {
+		/* Initialize list of VF macvlans */
+		INIT_LIST_HEAD(&adapter->vf_mvs.l);
+		for (i = 0; i < num_vf_macvlans; i++) {
+			mv_list->vf = -1;
+			mv_list->free = true;
+			mv_list->rar_entry = hw->mac.num_rar_entries -
+				(i + adapter->num_vfs + 1);
+			list_add(&mv_list->l, &adapter->vf_mvs.l);
+			mv_list++;
+		}
+	}
+
+	/* If call to enable VFs succeeded then allocate memory
+	 * for per VF control structures.
+	 */
+	adapter->vfinfo =
+		kcalloc(adapter->num_vfs,
+			sizeof(struct vf_data_storage), GFP_KERNEL);
+	if (adapter->vfinfo) {
+		/* Now that we're sure SR-IOV is enabled
+		 * and memory allocated set up the mailbox parameters
+		 */
+		ixgbe_init_mbx_params_pf(hw);
+		memcpy(&hw->mbx.ops, ii->mbx_ops,
+		       sizeof(hw->mbx.ops));
+
+		/* Disable RSC when in SR-IOV mode */
+		adapter->flags2 &= ~(IXGBE_FLAG2_RSC_CAPABLE |
+				     IXGBE_FLAG2_RSC_ENABLED);
+		return;
+	}
+
+	/* Oh oh */
+	e_err(probe, "Unable to allocate memory for VF Data Storage - "
+	      "SRIOV disabled\n");
+	pci_disable_sriov(adapter->pdev);
+
+err_novfs:
+	adapter->flags &= ~IXGBE_FLAG_SRIOV_ENABLED;
+	adapter->num_vfs = 0;
+}
+#endif /* #ifdef CONFIG_PCI_IOV */
+
+void ixgbe_disable_sriov(struct ixgbe_adapter *adapter)
+{
+	struct ixgbe_hw *hw = &adapter->hw;
+	u32 gcr;
+	u32 gpie;
+	u32 vmdctl;
+	int i;
+
+#ifdef CONFIG_PCI_IOV
+	/* disable iov and allow time for transactions to clear */
+	pci_disable_sriov(adapter->pdev);
+#endif
+
+	/* turn off device IOV mode */
+	gcr = IXGBE_READ_REG(hw, IXGBE_GCR_EXT);
+	gcr &= ~(IXGBE_GCR_EXT_SRIOV);
+	IXGBE_WRITE_REG(hw, IXGBE_GCR_EXT, gcr);
+	gpie = IXGBE_READ_REG(hw, IXGBE_GPIE);
+	gpie &= ~IXGBE_GPIE_VTMODE_MASK;
+	IXGBE_WRITE_REG(hw, IXGBE_GPIE, gpie);
+
+	/* set default pool back to 0 */
+	vmdctl = IXGBE_READ_REG(hw, IXGBE_VT_CTL);
+	vmdctl &= ~IXGBE_VT_CTL_POOL_MASK;
+	IXGBE_WRITE_REG(hw, IXGBE_VT_CTL, vmdctl);
+	IXGBE_WRITE_FLUSH(hw);
+
+	/* take a breather then clean up driver data */
+	msleep(100);
+
+	/* Release reference to VF devices */
+	for (i = 0; i < adapter->num_vfs; i++) {
+		if (adapter->vfinfo[i].vfdev)
+			pci_dev_put(adapter->vfinfo[i].vfdev);
+	}
+	kfree(adapter->vfinfo);
+	kfree(adapter->mv_list);
+	adapter->vfinfo = NULL;
+
+	adapter->num_vfs = 0;
+	adapter->flags &= ~IXGBE_FLAG_SRIOV_ENABLED;
+}
+
 static int ixgbe_set_vf_multicasts(struct ixgbe_adapter *adapter,
 				   int entries, u16 *hash_list, u32 vf)
 {
@@ -273,11 +438,26 @@ static int ixgbe_set_vf_macvlan(struct ixgbe_adapter *adapter,
 	return 0;
 }
 
+int ixgbe_check_vf_assignment(struct ixgbe_adapter *adapter)
+{
+	int i;
+	for (i = 0; i < adapter->num_vfs; i++) {
+		if (adapter->vfinfo[i].vfdev->dev_flags &
+				PCI_DEV_FLAGS_ASSIGNED)
+			return true;
+	}
+	return false;
+}
+
 int ixgbe_vf_configuration(struct pci_dev *pdev, unsigned int event_mask)
 {
 	unsigned char vf_mac_addr[6];
 	struct ixgbe_adapter *adapter = pci_get_drvdata(pdev);
 	unsigned int vfn = (event_mask & 0x3f);
+	struct pci_dev *pvfdev;
+	unsigned int device_id;
+	u16 thisvf_devfn = (pdev->devfn + 0x80 + (vfn << 1)) |
+				(pdev->devfn & 1);
 
 	bool enable = ((event_mask & 0x10000000U) != 0);
 
@@ -290,6 +470,31 @@ int ixgbe_vf_configuration(struct pci_dev *pdev, unsigned int event_mask)
 		 * for it later.
 		 */
 		memcpy(adapter->vfinfo[vfn].vf_mac_addresses, vf_mac_addr, 6);
+
+		switch (adapter->hw.mac.type) {
+		case ixgbe_mac_82599EB:
+			device_id = IXGBE_DEV_ID_82599_VF;
+			break;
+		case ixgbe_mac_X540:
+			device_id = IXGBE_DEV_ID_X540_VF;
+			break;
+		default:
+			device_id = 0;
+			break;
+		}
+
+		pvfdev = pci_get_device(IXGBE_INTEL_VENDOR_ID, device_id, NULL);
+		while (pvfdev) {
+			if (pvfdev->devfn == thisvf_devfn)
+				break;
+			pvfdev = pci_get_device(IXGBE_INTEL_VENDOR_ID,
+						device_id, pvfdev);
+		}
+		if (pvfdev)
+			adapter->vfinfo[vfn].vfdev = pvfdev;
+		else
+			e_err(drv, "Couldn't find pci dev ptr for VF %4.4x\n",
+			      thisvf_devfn);
 	}
 
 	return 0;
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.h b/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.h
index 3417556..2781847 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.h
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.h
@@ -41,6 +41,11 @@ int ixgbe_ndo_set_vf_bw(struct net_device *netdev, int vf, int tx_rate);
 int ixgbe_ndo_get_vf_config(struct net_device *netdev,
 			    int vf, struct ifla_vf_info *ivi);
 void ixgbe_check_vf_rate_limit(struct ixgbe_adapter *adapter);
+void ixgbe_disable_sriov(struct ixgbe_adapter *adapter);
+void ixgbe_enable_sriov(struct ixgbe_adapter *adapter,
+			const struct ixgbe_info *ii);
+int ixgbe_check_vf_assignment(struct ixgbe_adapter *adapter);
+
 
 #endif /* _IXGBE_SRIOV_H_ */
 
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_type.h b/drivers/net/ethernet/intel/ixgbe/ixgbe_type.h
index a9f8839..56a7adf 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_type.h
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_type.h
@@ -65,6 +65,10 @@
 #define IXGBE_DEV_ID_82599_LS            0x154F
 #define IXGBE_DEV_ID_X540T               0x1528
 
+/* VF Device IDs */
+#define IXGBE_DEV_ID_82599_VF           0x10ED
+#define IXGBE_DEV_ID_X540_VF            0x1515
+
 /* General Registers */
 #define IXGBE_CTRL      0x00000
 #define IXGBE_STATUS    0x00008
-- 
1.7.6.2

[net-next 3/8] ixgbe: DCB, do not call set_state() from IEEE mode

[Jeff Kirsher]

From: John Fastabend <john.r.fastabend@intel.com>

The DCB CEE command set_state() will complete successfully
but is misleading because it enables IEEE mode. After
this patch the command is failed.

And IEEE PFC/ETS is managed from ieee paths now instead
of using CEE primitives.

Signed-off-by: John Fastabend <john.r.fastabend@intel.com>
Tested-by: Ross Brattain <ross.b.brattain@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
---
 drivers/net/ethernet/intel/ixgbe/ixgbe_dcb.c    |   36 +++++++++++++++-
 drivers/net/ethernet/intel/ixgbe/ixgbe_dcb.h    |    3 +-
 drivers/net/ethernet/intel/ixgbe/ixgbe_dcb_nl.c |   54 ++++++++--------------
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c   |   18 +++++---
 4 files changed, 69 insertions(+), 42 deletions(-)

diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_dcb.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_dcb.c
index 9d88c31..83bf7cc 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_dcb.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_dcb.c
@@ -40,7 +40,8 @@
  * hardware. The IEEE 802.1Qaz specification do not use bandwidth
  * groups so this is much simplified from the CEE case.
  */
-s32 ixgbe_ieee_credits(__u8 *bw, __u16 *refill, __u16 *max, int max_frame)
+static s32 ixgbe_ieee_credits(__u8 *bw, __u16 *refill,
+			      __u16 *max, int max_frame)
 {
 	int min_percent = 100;
 	int min_credit, multiplier;
@@ -291,6 +292,39 @@ s32 ixgbe_dcb_hw_pfc_config(struct ixgbe_hw *hw, u8 pfc_en)
 	return ret;
 }
 
+s32 ixgbe_dcb_hw_ets(struct ixgbe_hw *hw, struct ieee_ets *ets, int max_frame)
+{
+	__u16 refill[IEEE_8021QAZ_MAX_TCS], max[IEEE_8021QAZ_MAX_TCS];
+	__u8 prio_type[IEEE_8021QAZ_MAX_TCS];
+	int i;
+
+	/* naively give each TC a bwg to map onto CEE hardware */
+	__u8 bwg_id[IEEE_8021QAZ_MAX_TCS] = {0, 1, 2, 3, 4, 5, 6, 7};
+
+	/* Map TSA onto CEE prio type */
+	for (i = 0; i < IEEE_8021QAZ_MAX_TCS; i++) {
+		switch (ets->tc_tsa[i]) {
+		case IEEE_8021QAZ_TSA_STRICT:
+			prio_type[i] = 2;
+			break;
+		case IEEE_8021QAZ_TSA_ETS:
+			prio_type[i] = 0;
+			break;
+		default:
+			/* Hardware only supports priority strict or
+			 * ETS transmission selection algorithms if
+			 * we receive some other value from dcbnl
+			 * throw an error
+			 */
+			return -EINVAL;
+		}
+	}
+
+	ixgbe_ieee_credits(ets->tc_tx_bw, refill, max, max_frame);
+	return ixgbe_dcb_hw_ets_config(hw, refill, max,
+				       bwg_id, prio_type, ets->prio_tc);
+}
+
 s32 ixgbe_dcb_hw_ets_config(struct ixgbe_hw *hw,
 			    u16 *refill, u16 *max, u8 *bwg_id,
 			    u8 *prio_type, u8 *prio_tc)
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_dcb.h b/drivers/net/ethernet/intel/ixgbe/ixgbe_dcb.h
index e85826a..0a68aa7 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_dcb.h
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_dcb.h
@@ -29,6 +29,7 @@
 #ifndef _DCB_CONFIG_H_
 #define _DCB_CONFIG_H_
 
+#include <linux/dcbnl.h>
 #include "ixgbe_type.h"
 
 /* DCB data structures */
@@ -147,11 +148,11 @@ void ixgbe_dcb_unpack_bwgid(struct ixgbe_dcb_config *, int, u8 *);
 void ixgbe_dcb_unpack_prio(struct ixgbe_dcb_config *, int, u8 *);
 
 /* DCB credits calculation */
-s32 ixgbe_ieee_credits(__u8 *bw, __u16 *refill, __u16 *max, int max_frame);
 s32 ixgbe_dcb_calculate_tc_credits(struct ixgbe_hw *,
 				   struct ixgbe_dcb_config *, int, u8);
 
 /* DCB hw initialization */
+s32 ixgbe_dcb_hw_ets(struct ixgbe_hw *hw, struct ieee_ets *ets, int max);
 s32 ixgbe_dcb_hw_ets_config(struct ixgbe_hw *hw, u16 *refill, u16 *max,
 			    u8 *bwg_id, u8 *prio_type, u8 *tc_prio);
 s32 ixgbe_dcb_hw_pfc_config(struct ixgbe_hw *hw, u8 pfc_en);
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_dcb_nl.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_dcb_nl.c
index 0422e35..22caad7 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_dcb_nl.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_dcb_nl.c
@@ -114,6 +114,10 @@ static u8 ixgbe_dcbnl_set_state(struct net_device *netdev, u8 state)
 	u8 err = 0;
 	struct ixgbe_adapter *adapter = netdev_priv(netdev);
 
+	/* Fail command if not in CEE mode */
+	if (!(adapter->dcbx_cap & DCB_CAP_DCBX_VER_CEE))
+		return 1;
+
 	/* verify there is something to do, if not then exit */
 	if (!!state != !(adapter->flags & IXGBE_FLAG_DCB_ENABLED))
 		return err;
@@ -301,6 +305,10 @@ static u8 ixgbe_dcbnl_set_all(struct net_device *netdev)
 	u8 up = dcb_getapp(netdev, &app);
 #endif
 
+	/* Fail command if not in CEE mode */
+	if (!(adapter->dcbx_cap & DCB_CAP_DCBX_VER_CEE))
+		return 1;
+
 	ret = ixgbe_copy_dcb_cfg(&adapter->temp_dcb_cfg, &adapter->dcb_cfg,
 				 MAX_TRAFFIC_CLASS);
 	if (ret)
@@ -537,13 +545,9 @@ static int ixgbe_dcbnl_ieee_setets(struct net_device *dev,
 				   struct ieee_ets *ets)
 {
 	struct ixgbe_adapter *adapter = netdev_priv(dev);
-	__u16 refill[IEEE_8021QAZ_MAX_TCS], max[IEEE_8021QAZ_MAX_TCS];
-	__u8 prio_type[IEEE_8021QAZ_MAX_TCS];
 	int max_frame = dev->mtu + ETH_HLEN + ETH_FCS_LEN;
-	int i, err;
-	__u64 *p = (__u64 *) ets->prio_tc;
-	/* naively give each TC a bwg to map onto CEE hardware */
-	__u8 bwg_id[IEEE_8021QAZ_MAX_TCS] = {0, 1, 2, 3, 4, 5, 6, 7};
+	int i;
+	__u8 max_tc = 0;
 
 	if (!(adapter->dcbx_cap & DCB_CAP_DCBX_VER_IEEE))
 		return -EINVAL;
@@ -557,34 +561,18 @@ static int ixgbe_dcbnl_ieee_setets(struct net_device *dev,
 
 	memcpy(adapter->ixgbe_ieee_ets, ets, sizeof(*adapter->ixgbe_ieee_ets));
 
-	/* Map TSA onto CEE prio type */
 	for (i = 0; i < IEEE_8021QAZ_MAX_TCS; i++) {
-		switch (ets->tc_tsa[i]) {
-		case IEEE_8021QAZ_TSA_STRICT:
-			prio_type[i] = 2;
-			break;
-		case IEEE_8021QAZ_TSA_ETS:
-			prio_type[i] = 0;
-			break;
-		default:
-			/* Hardware only supports priority strict or
-			 * ETS transmission selection algorithms if
-			 * we receive some other value from dcbnl
-			 * throw an error
-			 */
-			return -EINVAL;
-		}
+		if (ets->prio_tc[i] > max_tc)
+			max_tc = ets->prio_tc[i];
 	}
 
-	if (*p)
-		ixgbe_dcbnl_set_state(dev, 1);
-	else
-		ixgbe_dcbnl_set_state(dev, 0);
+	if (max_tc)
+		max_tc++;
 
-	ixgbe_ieee_credits(ets->tc_tx_bw, refill, max, max_frame);
-	err = ixgbe_dcb_hw_ets_config(&adapter->hw, refill, max,
-				      bwg_id, prio_type, ets->prio_tc);
-	return err;
+	if (max_tc != netdev_get_num_tc(dev))
+		ixgbe_setup_tc(dev, max_tc);
+
+	return ixgbe_dcb_hw_ets(&adapter->hw, ets, max_frame);
 }
 
 static int ixgbe_dcbnl_ieee_getpfc(struct net_device *dev,
@@ -615,7 +603,6 @@ static int ixgbe_dcbnl_ieee_setpfc(struct net_device *dev,
 				   struct ieee_pfc *pfc)
 {
 	struct ixgbe_adapter *adapter = netdev_priv(dev);
-	int err;
 
 	if (!(adapter->dcbx_cap & DCB_CAP_DCBX_VER_IEEE))
 		return -EINVAL;
@@ -628,8 +615,7 @@ static int ixgbe_dcbnl_ieee_setpfc(struct net_device *dev,
 	}
 
 	memcpy(adapter->ixgbe_ieee_pfc, pfc, sizeof(*adapter->ixgbe_ieee_pfc));
-	err = ixgbe_dcb_hw_pfc_config(&adapter->hw, pfc->pfc_en);
-	return err;
+	return ixgbe_dcb_hw_pfc_config(&adapter->hw, pfc->pfc_en);
 }
 
 #ifdef IXGBE_FCOE
@@ -740,7 +726,7 @@ static u8 ixgbe_dcbnl_setdcbx(struct net_device *dev, u8 mode)
 		 */
 		ixgbe_dcbnl_ieee_setets(dev, &ets);
 		ixgbe_dcbnl_ieee_setpfc(dev, &pfc);
-		ixgbe_dcbnl_set_state(dev, 0);
+		ixgbe_setup_tc(dev, 0);
 	}
 
 	return 0;
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
index 8b86c41..4cbc3f9 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
@@ -3319,12 +3319,18 @@ static void ixgbe_configure_dcb(struct ixgbe_adapter *adapter)
 	} else {
 		struct net_device *dev = adapter->netdev;
 
-		if (adapter->ixgbe_ieee_ets)
-			dev->dcbnl_ops->ieee_setets(dev,
-						    adapter->ixgbe_ieee_ets);
-		if (adapter->ixgbe_ieee_pfc)
-			dev->dcbnl_ops->ieee_setpfc(dev,
-						    adapter->ixgbe_ieee_pfc);
+		if (adapter->ixgbe_ieee_ets) {
+			struct ieee_ets *ets = adapter->ixgbe_ieee_ets;
+			int max_frame = dev->mtu + ETH_HLEN + ETH_FCS_LEN;
+
+			ixgbe_dcb_hw_ets(&adapter->hw, ets, max_frame);
+		}
+
+		if (adapter->ixgbe_ieee_pfc) {
+			struct ieee_pfc *pfc = adapter->ixgbe_ieee_pfc;
+
+			ixgbe_dcb_hw_pfc_config(&adapter->hw, pfc->pfc_en);
+		}
 	}
 
 	/* Enable RSS Hash per TC */
-- 
1.7.6.2

[net-next 4/8] ixgbe: cleanup X540 interrupt enablement

[Jeff Kirsher]

From: Don Skidmore <donald.c.skidmore@intel.com>

We don't need SFP+ plugable support for X540 hardware (copper only) so
don't enable the SFP+ interrupts.

Signed-off-by: Don Skidmore <donald.c.skidmore@intel.com>
Tested-by: Phil Schmitt <phillip.j.schmitt@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
---
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
index 4cbc3f9..fae2f44 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
@@ -1863,10 +1863,10 @@ static inline void ixgbe_irq_enable(struct ixgbe_adapter *adapter, bool queues,
 		mask |= IXGBE_EIMS_GPI_SDP1;
 	switch (adapter->hw.mac.type) {
 	case ixgbe_mac_82599EB:
-	case ixgbe_mac_X540:
-		mask |= IXGBE_EIMS_ECC;
 		mask |= IXGBE_EIMS_GPI_SDP1;
 		mask |= IXGBE_EIMS_GPI_SDP2;
+	case ixgbe_mac_X540:
+		mask |= IXGBE_EIMS_ECC;
 		mask |= IXGBE_EIMS_MAILBOX;
 		break;
 	default:
-- 
1.7.6.2

[net-next 5/8] ixgbe: dcb, set priority to traffic class mappings

[Jeff Kirsher]

From: John Fastabend <john.r.fastabend@intel.com>

This patch adds support for configuring the priority to
traffic class mapping.

Signed-off-by: John Fastabend <john.r.fastabend@intel.com>
Tested-by: Ross Brattain <ross.b.brattain@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
---
 drivers/net/ethernet/intel/ixgbe/ixgbe_dcb_nl.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_dcb_nl.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_dcb_nl.c
index 22caad7..1d38955 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_dcb_nl.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_dcb_nl.c
@@ -572,6 +572,9 @@ static int ixgbe_dcbnl_ieee_setets(struct net_device *dev,
 	if (max_tc != netdev_get_num_tc(dev))
 		ixgbe_setup_tc(dev, max_tc);
 
+	for (i = 0; i < IEEE_8021QAZ_MAX_TCS; i++)
+		netdev_set_prio_tc_map(dev, i, ets->prio_tc[i]);
+
 	return ixgbe_dcb_hw_ets(&adapter->hw, ets, max_frame);
 }
 
-- 
1.7.6.2

[net-next 6/8] ixgbe: avoid HW lockup when adapter is reset with Tx work pending

[Jeff Kirsher]

From: Emil Tantilov <emil.s.tantilov@intel.com>

This change is meant to avoid a hardware lockup when Tx work is still
pending and we request a reset.

Signed-off-by: Emil Tantilov <emil.s.tantilov@intel.com>
Tested-by: Phil Schmitt <phillip.j.schmitt@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
---
 drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c  |   15 +--
 drivers/net/ethernet/intel/ixgbe/ixgbe_82599.c  |   13 +-
 drivers/net/ethernet/intel/ixgbe/ixgbe_common.c |  145 +++++++++++++----------
 drivers/net/ethernet/intel/ixgbe/ixgbe_common.h |    2 +-
 drivers/net/ethernet/intel/ixgbe/ixgbe_type.h   |    2 +
 drivers/net/ethernet/intel/ixgbe/ixgbe_x540.c   |   12 +-
 6 files changed, 104 insertions(+), 85 deletions(-)

diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
index 22504f2..b816a62 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
@@ -759,7 +759,9 @@ static s32 ixgbe_reset_hw_82598(struct ixgbe_hw *hw)
 	u8  analog_val;
 
 	/* Call adapter stop to disable tx/rx and clear interrupts */
-	hw->mac.ops.stop_adapter(hw);
+	status = hw->mac.ops.stop_adapter(hw);
+	if (status != 0)
+		goto reset_hw_out;
 
 	/*
 	 * Power up the Atlas Tx lanes if they are currently powered down.
@@ -802,19 +804,12 @@ static s32 ixgbe_reset_hw_82598(struct ixgbe_hw *hw)
 		phy_status = hw->phy.ops.init(hw);
 		if (phy_status == IXGBE_ERR_SFP_NOT_SUPPORTED)
 			goto reset_hw_out;
-		else if (phy_status == IXGBE_ERR_SFP_NOT_PRESENT)
-			goto no_phy_reset;
+		if (phy_status == IXGBE_ERR_SFP_NOT_PRESENT)
+			goto mac_reset_top;
 
 		hw->phy.ops.reset(hw);
 	}
 
-no_phy_reset:
-	/*
-	 * Prevent the PCI-E bus from from hanging by disabling PCI-E master
-	 * access and verify no pending requests before reset
-	 */
-	ixgbe_disable_pcie_master(hw);
-
 mac_reset_top:
 	/*
 	 * Issue global reset to the MAC.  This needs to be a SW reset.
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_82599.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_82599.c
index a5ff435..5abd520 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_82599.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_82599.c
@@ -910,7 +910,12 @@ static s32 ixgbe_reset_hw_82599(struct ixgbe_hw *hw)
 	bool link_up = false;
 
 	/* Call adapter stop to disable tx/rx and clear interrupts */
-	hw->mac.ops.stop_adapter(hw);
+	status = hw->mac.ops.stop_adapter(hw);
+	if (status != 0)
+		goto reset_hw_out;
+
+	/* flush pending Tx transactions */
+	ixgbe_clear_tx_pending(hw);
 
 	/* PHY ops must be identified and initialized prior to reset */
 
@@ -933,12 +938,6 @@ static s32 ixgbe_reset_hw_82599(struct ixgbe_hw *hw)
 	if (hw->phy.reset_disable == false && hw->phy.ops.reset != NULL)
 		hw->phy.ops.reset(hw);
 
-	/*
-	 * Prevent the PCI-E bus from from hanging by disabling PCI-E master
-	 * access and verify no pending requests before reset
-	 */
-	ixgbe_disable_pcie_master(hw);
-
 mac_reset_top:
 	/*
 	 * Issue global reset to the MAC. Needs to be SW reset if link is up.
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_common.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_common.c
index 91986af..84ed9ef 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_common.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_common.c
@@ -61,6 +61,7 @@ static s32 ixgbe_write_eeprom_buffer_bit_bang(struct ixgbe_hw *hw, u16 offset,
 					     u16 words, u16 *data);
 static s32 ixgbe_detect_eeprom_page_size_generic(struct ixgbe_hw *hw,
 						 u16 offset);
+static s32 ixgbe_disable_pcie_master(struct ixgbe_hw *hw);
 
 /**
  *  ixgbe_start_hw_generic - Prepare hardware for Tx/Rx
@@ -496,7 +497,6 @@ void ixgbe_set_lan_id_multi_port_pcie(struct ixgbe_hw *hw)
  **/
 s32 ixgbe_stop_adapter_generic(struct ixgbe_hw *hw)
 {
-	u32 number_of_queues;
 	u32 reg_val;
 	u16 i;
 
@@ -507,35 +507,35 @@ s32 ixgbe_stop_adapter_generic(struct ixgbe_hw *hw)
 	hw->adapter_stopped = true;
 
 	/* Disable the receive unit */
-	reg_val = IXGBE_READ_REG(hw, IXGBE_RXCTRL);
-	reg_val &= ~(IXGBE_RXCTRL_RXEN);
-	IXGBE_WRITE_REG(hw, IXGBE_RXCTRL, reg_val);
-	IXGBE_WRITE_FLUSH(hw);
-	usleep_range(2000, 4000);
+	IXGBE_WRITE_REG(hw, IXGBE_RXCTRL, 0);
 
-	/* Clear interrupt mask to stop from interrupts being generated */
+	/* Clear interrupt mask to stop interrupts from being generated */
 	IXGBE_WRITE_REG(hw, IXGBE_EIMC, IXGBE_IRQ_CLEAR_MASK);
 
-	/* Clear any pending interrupts */
+	/* Clear any pending interrupts, flush previous writes */
 	IXGBE_READ_REG(hw, IXGBE_EICR);
 
 	/* Disable the transmit unit.  Each queue must be disabled. */
-	number_of_queues = hw->mac.max_tx_queues;
-	for (i = 0; i < number_of_queues; i++) {
-		reg_val = IXGBE_READ_REG(hw, IXGBE_TXDCTL(i));
-		if (reg_val & IXGBE_TXDCTL_ENABLE) {
-			reg_val &= ~IXGBE_TXDCTL_ENABLE;
-			IXGBE_WRITE_REG(hw, IXGBE_TXDCTL(i), reg_val);
-		}
+	for (i = 0; i < hw->mac.max_tx_queues; i++)
+		IXGBE_WRITE_REG(hw, IXGBE_TXDCTL(i), IXGBE_TXDCTL_SWFLSH);
+
+	/* Disable the receive unit by stopping each queue */
+	for (i = 0; i < hw->mac.max_rx_queues; i++) {
+		reg_val = IXGBE_READ_REG(hw, IXGBE_RXDCTL(i));
+		reg_val &= ~IXGBE_RXDCTL_ENABLE;
+		reg_val |= IXGBE_RXDCTL_SWFLSH;
+		IXGBE_WRITE_REG(hw, IXGBE_RXDCTL(i), reg_val);
 	}
 
+	/* flush all queues disables */
+	IXGBE_WRITE_FLUSH(hw);
+	usleep_range(1000, 2000);
+
 	/*
 	 * Prevent the PCI-E bus from from hanging by disabling PCI-E master
 	 * access and verify no pending requests
 	 */
-	ixgbe_disable_pcie_master(hw);
-
-	return 0;
+	return ixgbe_disable_pcie_master(hw);
 }
 
 /**
@@ -2458,75 +2458,57 @@ out:
  *  bit hasn't caused the master requests to be disabled, else 0
  *  is returned signifying master requests disabled.
  **/
-s32 ixgbe_disable_pcie_master(struct ixgbe_hw *hw)
+static s32 ixgbe_disable_pcie_master(struct ixgbe_hw *hw)
 {
 	struct ixgbe_adapter *adapter = hw->back;
-	u32 i;
-	u32 reg_val;
-	u32 number_of_queues;
 	s32 status = 0;
-	u16 dev_status = 0;
+	u32 i;
+	u16 value;
+
+	/* Always set this bit to ensure any future transactions are blocked */
+	IXGBE_WRITE_REG(hw, IXGBE_CTRL, IXGBE_CTRL_GIO_DIS);
 
-	/* Just jump out if bus mastering is already disabled */
+	/* Exit if master requests are blocked */
 	if (!(IXGBE_READ_REG(hw, IXGBE_STATUS) & IXGBE_STATUS_GIO))
 		goto out;
 
-	/* Disable the receive unit by stopping each queue */
-	number_of_queues = hw->mac.max_rx_queues;
-	for (i = 0; i < number_of_queues; i++) {
-		reg_val = IXGBE_READ_REG(hw, IXGBE_RXDCTL(i));
-		if (reg_val & IXGBE_RXDCTL_ENABLE) {
-			reg_val &= ~IXGBE_RXDCTL_ENABLE;
-			IXGBE_WRITE_REG(hw, IXGBE_RXDCTL(i), reg_val);
-		}
-	}
-
-	reg_val = IXGBE_READ_REG(hw, IXGBE_CTRL);
-	reg_val |= IXGBE_CTRL_GIO_DIS;
-	IXGBE_WRITE_REG(hw, IXGBE_CTRL, reg_val);
-
+	/* Poll for master request bit to clear */
 	for (i = 0; i < IXGBE_PCI_MASTER_DISABLE_TIMEOUT; i++) {
-		if (!(IXGBE_READ_REG(hw, IXGBE_STATUS) & IXGBE_STATUS_GIO))
-			goto check_device_status;
 		udelay(100);
+		if (!(IXGBE_READ_REG(hw, IXGBE_STATUS) & IXGBE_STATUS_GIO))
+			goto out;
 	}
 
+	/*
+	 * Two consecutive resets are required via CTRL.RST per datasheet
+	 * 5.2.5.3.2 Master Disable.  We set a flag to inform the reset routine
+	 * of this need.  The first reset prevents new master requests from
+	 * being issued by our device.  We then must wait 1usec or more for any
+	 * remaining completions from the PCIe bus to trickle in, and then reset
+	 * again to clear out any effects they may have had on our device.
+	 */
 	hw_dbg(hw, "GIO Master Disable bit didn't clear - requesting resets\n");
-	status = IXGBE_ERR_MASTER_REQUESTS_PENDING;
+	hw->mac.flags |= IXGBE_FLAGS_DOUBLE_RESET_REQUIRED;
 
 	/*
 	 * Before proceeding, make sure that the PCIe block does not have
 	 * transactions pending.
 	 */
-check_device_status:
 	for (i = 0; i < IXGBE_PCI_MASTER_DISABLE_TIMEOUT; i++) {
-		pci_read_config_word(adapter->pdev, IXGBE_PCI_DEVICE_STATUS,
-							 &dev_status);
-		if (!(dev_status & IXGBE_PCI_DEVICE_STATUS_TRANSACTION_PENDING))
-			break;
 		udelay(100);
+		pci_read_config_word(adapter->pdev, IXGBE_PCI_DEVICE_STATUS,
+							 &value);
+		if (!(value & IXGBE_PCI_DEVICE_STATUS_TRANSACTION_PENDING))
+			goto out;
 	}
 
-	if (i == IXGBE_PCI_MASTER_DISABLE_TIMEOUT)
-		hw_dbg(hw, "PCIe transaction pending bit also did not clear.\n");
-	else
-		goto out;
-
-	/*
-	 * Two consecutive resets are required via CTRL.RST per datasheet
-	 * 5.2.5.3.2 Master Disable.  We set a flag to inform the reset routine
-	 * of this need.  The first reset prevents new master requests from
-	 * being issued by our device.  We then must wait 1usec for any
-	 * remaining completions from the PCIe bus to trickle in, and then reset
-	 * again to clear out any effects they may have had on our device.
-	 */
-	 hw->mac.flags |= IXGBE_FLAGS_DOUBLE_RESET_REQUIRED;
+	hw_dbg(hw, "PCIe transaction pending bit also did not clear.\n");
+	status = IXGBE_ERR_MASTER_REQUESTS_PENDING;
 
 out:
 	return status;
 }
 
-
 /**
  *  ixgbe_acquire_swfw_sync - Acquire SWFW semaphore
  *  @hw: pointer to hardware structure
@@ -3509,3 +3491,44 @@ s32 ixgbe_set_fw_drv_ver_generic(struct ixgbe_hw *hw, u8 maj, u8 min,
 out:
 	return ret_val;
 }
+
+/**
+ * ixgbe_clear_tx_pending - Clear pending TX work from the PCIe fifo
+ * @hw: pointer to the hardware structure
+ *
+ * The 82599 and x540 MACs can experience issues if TX work is still pending
+ * when a reset occurs.  This function prevents this by flushing the PCIe
+ * buffers on the system.
+ **/
+void ixgbe_clear_tx_pending(struct ixgbe_hw *hw)
+{
+	u32 gcr_ext, hlreg0;
+
+	/*
+	 * If double reset is not requested then all transactions should
+	 * already be clear and as such there is no work to do
+	 */
+	if (!(hw->mac.flags & IXGBE_FLAGS_DOUBLE_RESET_REQUIRED))
+		return;
+
+	/*
+	 * Set loopback enable to prevent any transmits from being sent
+	 * should the link come up.  This assumes that the RXCTRL.RXEN bit
+	 * has already been cleared.
+	 */
+	hlreg0 = IXGBE_READ_REG(hw, IXGBE_HLREG0);
+	IXGBE_WRITE_REG(hw, IXGBE_HLREG0, hlreg0 | IXGBE_HLREG0_LPBK);
+
+	/* initiate cleaning flow for buffers in the PCIe transaction layer */
+	gcr_ext = IXGBE_READ_REG(hw, IXGBE_GCR_EXT);
+	IXGBE_WRITE_REG(hw, IXGBE_GCR_EXT,
+			gcr_ext | IXGBE_GCR_EXT_BUFFERS_CLEAR);
+
+	/* Flush all writes and allow 20usec for all transactions to clear */
+	IXGBE_WRITE_FLUSH(hw);
+	udelay(20);
+
+	/* restore previous register values */
+	IXGBE_WRITE_REG(hw, IXGBE_GCR_EXT, gcr_ext);
+	IXGBE_WRITE_REG(hw, IXGBE_HLREG0, hlreg0);
+}
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_common.h b/drivers/net/ethernet/intel/ixgbe/ixgbe_common.h
index f24fd64..863f9c1 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_common.h
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_common.h
@@ -81,7 +81,6 @@ s32 ixgbe_fc_autoneg(struct ixgbe_hw *hw);
 s32 ixgbe_validate_mac_addr(u8 *mac_addr);
 s32 ixgbe_acquire_swfw_sync(struct ixgbe_hw *hw, u16 mask);
 void ixgbe_release_swfw_sync(struct ixgbe_hw *hw, u16 mask);
-s32 ixgbe_disable_pcie_master(struct ixgbe_hw *hw);
 s32 ixgbe_get_san_mac_addr_generic(struct ixgbe_hw *hw, u8 *san_mac_addr);
 s32 ixgbe_set_vmdq_generic(struct ixgbe_hw *hw, u32 rar, u32 vmdq);
 s32 ixgbe_clear_vmdq_generic(struct ixgbe_hw *hw, u32 rar, u32 vmdq);
@@ -101,6 +100,7 @@ void ixgbe_set_vlan_anti_spoofing(struct ixgbe_hw *hw, bool enable, int vf);
 s32 ixgbe_get_device_caps_generic(struct ixgbe_hw *hw, u16 *device_caps);
 s32 ixgbe_set_fw_drv_ver_generic(struct ixgbe_hw *hw, u8 maj, u8 min,
 				 u8 build, u8 ver);
+void ixgbe_clear_tx_pending(struct ixgbe_hw *hw);
 
 void ixgbe_set_rxpba_generic(struct ixgbe_hw *hw, int num_pb,
 			     u32 headroom, int strategy);
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_type.h b/drivers/net/ethernet/intel/ixgbe/ixgbe_type.h
index 56a7adf..b119cd6 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_type.h
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_type.h
@@ -770,6 +770,7 @@
 #define IXGBE_GCR_CAP_VER2              0x00040000
 
 #define IXGBE_GCR_EXT_MSIX_EN           0x80000000
+#define IXGBE_GCR_EXT_BUFFERS_CLEAR     0x40000000
 #define IXGBE_GCR_EXT_VT_MODE_16        0x00000001
 #define IXGBE_GCR_EXT_VT_MODE_32        0x00000002
 #define IXGBE_GCR_EXT_VT_MODE_64        0x00000003
@@ -1822,6 +1823,7 @@ enum {
 #define IXGBE_RXCTRL_RXEN       0x00000001  /* Enable Receiver */
 #define IXGBE_RXCTRL_DMBYPS     0x00000002  /* Descriptor Monitor Bypass */
 #define IXGBE_RXDCTL_ENABLE     0x02000000  /* Enable specific Rx Queue */
+#define IXGBE_RXDCTL_SWFLSH     0x04000000  /* Rx Desc. write-back flushing */
 #define IXGBE_RXDCTL_RLPMLMASK  0x00003FFF  /* Only supported on the X540 */
 #define IXGBE_RXDCTL_RLPML_EN   0x00008000
 #define IXGBE_RXDCTL_VME        0x40000000  /* VLAN mode enable */
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_x540.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_x540.c
index bbfe8c4..84bb51d 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_x540.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_x540.c
@@ -99,13 +99,12 @@ static s32 ixgbe_reset_hw_X540(struct ixgbe_hw *hw)
 	bool link_up = false;
 
 	/* Call adapter stop to disable tx/rx and clear interrupts */
-	hw->mac.ops.stop_adapter(hw);
+	status = hw->mac.ops.stop_adapter(hw);
+	if (status != 0)
+		goto reset_hw_out;
 
-	/*
-	 * Prevent the PCI-E bus from from hanging by disabling PCI-E master
-	 * access and verify no pending requests before reset
-	 */
-	ixgbe_disable_pcie_master(hw);
+	/* flush pending Tx transactions */
+	ixgbe_clear_tx_pending(hw);
 
 mac_reset_top:
 	/*
@@ -180,6 +179,7 @@ mac_reset_top:
 	hw->mac.ops.get_wwn_prefix(hw, &hw->mac.wwnn_prefix,
 	                           &hw->mac.wwpn_prefix);
 
+reset_hw_out:
 	return status;
 }
 
-- 
1.7.6.2

[net-next 7/8] ixgbe: add WOL support for X540

[Jeff Kirsher]

From: Emil Tantilov <emil.s.tantilov@intel.com>

Add support for WOL as determined by the EEPROM.

Signed-off-by: Emil Tantilov <emil.s.tantilov@intel.com>
Tested-by: Phil Schmitt <phillip.j.schmitt@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
---
 drivers/net/ethernet/intel/ixgbe/ixgbe.h         |    1 +
 drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c |   13 +++++++++++++
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c    |   14 ++++++++++++--
 drivers/net/ethernet/intel/ixgbe/ixgbe_type.h    |    4 ++++
 4 files changed, 30 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe.h b/drivers/net/ethernet/intel/ixgbe/ixgbe.h
index b43b2cd..1f4a4ca 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe.h
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe.h
@@ -491,6 +491,7 @@ struct ixgbe_adapter {
 	u64 rsc_total_flush;
 	u32 wol;
 	u16 eeprom_version;
+	u16 eeprom_cap;
 
 	int node;
 	u32 led_reg;
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
index 63cd2a1..debcf5f 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
@@ -1888,6 +1888,7 @@ static int ixgbe_wol_exclusion(struct ixgbe_adapter *adapter,
 {
 	struct ixgbe_hw *hw = &adapter->hw;
 	int retval = 1;
+	u16 wol_cap = adapter->eeprom_cap & IXGBE_DEVICE_CAPS_WOL_MASK;
 
 	/* WOL not supported except for the following */
 	switch(hw->device_id) {
@@ -1911,6 +1912,18 @@ static int ixgbe_wol_exclusion(struct ixgbe_adapter *adapter,
 	case IXGBE_DEV_ID_82599_KX4:
 		retval = 0;
 		break;
+	case IXGBE_DEV_ID_X540T:
+		/* check eeprom to see if enabled wol */
+		if ((wol_cap == IXGBE_DEVICE_CAPS_WOL_PORT0_1) ||
+		    ((wol_cap == IXGBE_DEVICE_CAPS_WOL_PORT0) &&
+		     (hw->bus.func == 0))) {
+			retval = 0;
+			break;
+		}
+
+		/* All others not supported */
+		wol->supported = 0;
+		break;
 	default:
 		wol->supported = 0;
 	}
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
index fae2f44..5f50f1b 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
@@ -7074,6 +7074,7 @@ static int __devinit ixgbe_probe(struct pci_dev *pdev,
 	u16 device_caps;
 #endif
 	u32 eec;
+	u16 wol_cap;
 
 	/* Catch broken hardware that put the wrong VF device ID in
 	 * the PCIe SR-IOV capability.
@@ -7338,6 +7339,8 @@ static int __devinit ixgbe_probe(struct pci_dev *pdev,
 		netdev->features &= ~NETIF_F_RXHASH;
 	}
 
+	/* WOL not supported for all but the following */
+	adapter->wol = 0;
 	switch (pdev->device) {
 	case IXGBE_DEV_ID_82599_SFP:
 		/* Only this subdevice supports WOL */
@@ -7352,8 +7355,15 @@ static int __devinit ixgbe_probe(struct pci_dev *pdev,
 	case IXGBE_DEV_ID_82599_KX4:
 		adapter->wol = IXGBE_WUFC_MAG;
 		break;
-	default:
-		adapter->wol = 0;
+	case IXGBE_DEV_ID_X540T:
+		/* Check eeprom to see if it is enabled */
+		hw->eeprom.ops.read(hw, 0x2c, &adapter->eeprom_cap);
+		wol_cap = adapter->eeprom_cap & IXGBE_DEVICE_CAPS_WOL_MASK;
+
+		if ((wol_cap == IXGBE_DEVICE_CAPS_WOL_PORT0_1) ||
+		    ((wol_cap == IXGBE_DEVICE_CAPS_WOL_PORT0) &&
+		     (hw->bus.func == 0)))
+			adapter->wol = IXGBE_WUFC_MAG;
 		break;
 	}
 	device_set_wakeup_enable(&adapter->pdev->dev, adapter->wol);
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_type.h b/drivers/net/ethernet/intel/ixgbe/ixgbe_type.h
index b119cd6..9a03341 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_type.h
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_type.h
@@ -1754,6 +1754,10 @@ enum {
 #define IXGBE_ALT_SAN_MAC_ADDR_CAPS_SANMAC  0x0 /* Alt. SAN MAC exists */
 #define IXGBE_ALT_SAN_MAC_ADDR_CAPS_ALTWWN  0x1 /* Alt. WWN base exists */
 
+#define IXGBE_DEVICE_CAPS_WOL_PORT0_1  0x4 /* WoL supported on ports 0 & 1 */
+#define IXGBE_DEVICE_CAPS_WOL_PORT0    0x8 /* WoL supported on port 0 */
+#define IXGBE_DEVICE_CAPS_WOL_MASK     0xC /* Mask for WoL capabilities */
+
 /* PCI Bus Info */
 #define IXGBE_PCI_DEVICE_STATUS   0xAA
 #define IXGBE_PCI_DEVICE_STATUS_TRANSACTION_PENDING   0x0020
-- 
1.7.6.2

[net-next 8/8] ixgbe: remove global reset to the MAC

[Jeff Kirsher]

From: Emil Tantilov <emil.s.tantilov@intel.com>

Reloading FW during resets can cause issues. Remove the full reset
as it is not needed.

Signed-off-by: Emil Tantilov <emil.s.tantilov@intel.com>
Tested-by: Phil Schmitt <phillip.j.schmitt@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
---
 drivers/net/ethernet/intel/ixgbe/ixgbe_x540.c |   19 ++-----------------
 1 files changed, 2 insertions(+), 17 deletions(-)

diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_x540.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_x540.c
index 84bb51d..96e0b20 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_x540.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_x540.c
@@ -93,10 +93,8 @@ static s32 ixgbe_setup_mac_link_X540(struct ixgbe_hw *hw,
  **/
 static s32 ixgbe_reset_hw_X540(struct ixgbe_hw *hw)
 {
-	ixgbe_link_speed link_speed;
 	s32 status;
 	u32 ctrl, i;
-	bool link_up = false;
 
 	/* Call adapter stop to disable tx/rx and clear interrupts */
 	status = hw->mac.ops.stop_adapter(hw);
@@ -107,19 +105,7 @@ static s32 ixgbe_reset_hw_X540(struct ixgbe_hw *hw)
 	ixgbe_clear_tx_pending(hw);
 
 mac_reset_top:
-	/*
-	 * Issue global reset to the MAC.  Needs to be SW reset if link is up.
-	 * If link reset is used when link is up, it might reset the PHY when
-	 * mng is using it.  If link is down or the flag to force full link
-	 * reset is set, then perform link reset.
-	 */
-	ctrl = IXGBE_CTRL_LNK_RST;
-	if (!hw->force_full_reset) {
-		hw->mac.ops.check_link(hw, &link_speed, &link_up, false);
-		if (link_up)
-			ctrl = IXGBE_CTRL_RST;
-	}
-
+	ctrl = IXGBE_CTRL_RST;
 	ctrl |= IXGBE_READ_REG(hw, IXGBE_CTRL);
 	IXGBE_WRITE_REG(hw, IXGBE_CTRL, ctrl);
 	IXGBE_WRITE_FLUSH(hw);
@@ -136,8 +122,7 @@ mac_reset_top:
 		status = IXGBE_ERR_RESET_FAILED;
 		hw_dbg(hw, "Reset polling failed to complete.\n");
 	}
-
-	msleep(50);
+	msleep(100);
 
 	/*
 	 * Double resets are required for recovery from certain error
-- 
1.7.6.2

Re: [net-next 0/8][pull request] Intel Wired LAN Driver Update

[David Miller]

From: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Date: Wed, 21 Sep 2011 03:12:46 -0700

>   - reconfigure SR-IOV init to take advantage of the new pci flag

Has this gotten any review from the PCI maintainers?

RE: [net-next 0/8][pull request] Intel Wired LAN Driver Update

[Rose, Gregory V]

> -----Original Message-----
> From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org]
> On Behalf Of David Miller
> Sent: Wednesday, September 21, 2011 12:14 PM
> To: Kirsher, Jeffrey T
> Cc: netdev@vger.kernel.org; gospo@redhat.com
> Subject: Re: [net-next 0/8][pull request] Intel Wired LAN Driver Update
> 
> From: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
> Date: Wed, 21 Sep 2011 03:12:46 -0700
> 
> >   - reconfigure SR-IOV init to take advantage of the new pci flag
> 
> Has this gotten any review from the PCI maintainers?

Yes, when it was posted as an RFC back in July.  It was deemed acceptable at the time.

See inline mail below.

We probably should have copied Ian Campbell on Konrad Wilk.  I have added them to the CC line in this response.

- Greg

From:	netdev-owner@vger.kernel.org on behalf of Rose, Gregory V [gregory.v.rose@intel.com]
Sent:	Friday, July 29, 2011 9:55 AM
To:	Jesse Barnes; Ian Campbell
Cc:	Konrad Rzeszutek Wilk; netdev@vger.kernel.org; davem@davemloft.net; bhutchings@solarflare.com; Kirsher, Jeffrey T; linux-pci@vger.kernel.org
Subject:	RE: [RFC net-next PATCH 1/4] pci: Add flag indicating device has been assigned by KVM

> -----Original Message-----
> From: Jesse Barnes [mailto:jbarnes@virtuousgeek.org]
> Sent: Friday, July 29, 2011 9:52 AM
> To: Ian Campbell
> Cc: Rose, Gregory V; Konrad Rzeszutek Wilk; netdev@vger.kernel.org;
> davem@davemloft.net; bhutchings@solarflare.com; Kirsher, Jeffrey T; linux-
> pci@vger.kernel.org
> Subject: Re: [RFC net-next PATCH 1/4] pci: Add flag indicating device has
> been assigned by KVM
> 
> On Thu, 28 Jul 2011 16:11:17 +0100
> Ian Campbell <ijc@hellion.org.uk> wrote:
> 
> > On Wed, 2011-07-27 at 15:17 -0700, Greg Rose wrote:
> > > Device drivers that create and destroy SR-IOV virtual functions via
> > > calls to pci_enable_sriov() and pci_disable_sriov can cause
> catastrophic
> > > failures if they attempt to destroy VFs while they are assigned to
> > > guest virtual machines.  By adding a flag for use by the KVM module
> > > to indicate that a device is assigned a device driver can check that
> > > flag and avoid destroying VFs while they are assigned and avoid system
> > > failures.
> > >
> > > Signed-off-by: Greg Rose <gregory.v.rose@intel.com>
> > > ---
> > >
> > >  include/linux/pci.h     |    2 ++
> >
> > I added Jesse and linux-pci to CC.
> >
> > >  virt/kvm/assigned-dev.c |    2 ++
> > >  virt/kvm/iommu.c        |    4 ++++
> > >  3 files changed, 8 insertions(+), 0 deletions(-)
> >
> > I suppose this would also be useful in Xen's pciback or any other system
> > which does passthrough? (Konrad CC'd for pciback)
> >
> > Is there some common lower layer we could hook this in to? (does
> > iommu_attach/detach_device make sense?) Or shall we just add the flag
> > manipulations to pciback as well?
> >
> > Ian.
> >
> > >
> > > diff --git a/include/linux/pci.h b/include/linux/pci.h
> > > index 2d29218..a297ca2 100644
> > > --- a/include/linux/pci.h
> > > +++ b/include/linux/pci.h
> > > @@ -174,6 +174,8 @@ enum pci_dev_flags {
> > >  	PCI_DEV_FLAGS_MSI_INTX_DISABLE_BUG = (__force pci_dev_flags_t) 1,
> > >  	/* Device configuration is irrevocably lost if disabled into D3 */
> > >  	PCI_DEV_FLAGS_NO_D3 = (__force pci_dev_flags_t) 2,
> > > +	/* Provide indication device is assigned by KVM */
> > > +	PCI_DEV_FLAGS_ASSIGNED = (__force pci_dev_flags_t) 4,
> > >  };
> 
> Looks fine, but I'd make the comment less redundant with the code, e.g.
> "set when the device is assigned to a guest instance" or somesuch.

Sure, sounds good to me.

Rev 2 of the RFC patches will be out in a couple of weeks, I'm away next week.

Thanks,

- Greg

> 
> --
> Jesse Barnes, Intel Open Source Technology Center
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html