* [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init
@ 2012-01-27 19:10 Vaidyanathan Srinivasan
2012-01-27 21:05 ` Yinghai Lu
2012-01-30 3:18 ` Ram Pai
0 siblings, 2 replies; 9+ messages in thread
From: Vaidyanathan Srinivasan @ 2012-01-27 19:10 UTC (permalink / raw)
To: Ram Pai, Jesse Barnes, Yinghai Lu; +Cc: linux-pci, linux-kernel
Hi Ram and Jesse,
I found a trivial issue with page size alignment check on IBM POWER
box with 64k base page size. In sriov_init(), changing the check from
PAGE_SIZE (arch and config dependent) to HW_PAGE_SIZE (always 4k) was
required to use one of the sriov adapter as PF since the
resource_size() comes up as 0x8000 and PAGE_SIZE would be 0x10000 for
pseries boxes.
I think resource_size() could be less than SystemPageSize, but I would
like your comments/ack/nack on any consequences of checking for only
4k alignment here in a system with larger base page size.
Thanks,
Vaidy
---
pci: check for 4k resource_size alignment in sriov_init
pci sriov_init should check for 4k page size alignment of resource_size
even if base page size is larger -- like 64k in powerpc.
Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
index 0321fa3..5816fa0 100644
--- a/drivers/pci/iov.c
+++ b/drivers/pci/iov.c
@@ -474,7 +474,7 @@ found:
pos + PCI_SRIOV_BAR + i * 4);
if (!res->flags)
continue;
- if (resource_size(res) & (PAGE_SIZE - 1)) {
+ if (resource_size(res) & (HW_PAGE_SIZE - 1)) {
rc = -EIO;
goto failed;
}
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init
2012-01-27 19:10 [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init Vaidyanathan Srinivasan
@ 2012-01-27 21:05 ` Yinghai Lu
2012-01-29 13:11 ` Vaidyanathan Srinivasan
2012-01-30 3:18 ` Ram Pai
1 sibling, 1 reply; 9+ messages in thread
From: Yinghai Lu @ 2012-01-27 21:05 UTC (permalink / raw)
To: svaidy; +Cc: Ram Pai, Jesse Barnes, linux-pci, linux-kernel
On Fri, Jan 27, 2012 at 11:10 AM, Vaidyanathan Srinivasan
<svaidy@linux.vnet.ibm.com> wrote:
> Hi Ram and Jesse,
>
> I found a trivial issue with page size alignment check on IBM POWER
> box with 64k base page size. In sriov_init(), changing the check from
> PAGE_SIZE (arch and config dependent) to HW_PAGE_SIZE (always 4k) was
> required to use one of the sriov adapter as PF since the
> resource_size() comes up as 0x8000 and PAGE_SIZE would be 0x10000 for
> pseries boxes.
>
> I think resource_size() could be less than SystemPageSize, but I would
> like your comments/ack/nack on any consequences of checking for only
> 4k alignment here in a system with larger base page size.
>
> Thanks,
> Vaidy
>
> ---
>
> pci: check for 4k resource_size alignment in sriov_init
>
> pci sriov_init should check for 4k page size alignment of resource_size
> even if base page size is larger -- like 64k in powerpc.
>
> Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
>
> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> index 0321fa3..5816fa0 100644
> --- a/drivers/pci/iov.c
> +++ b/drivers/pci/iov.c
> @@ -474,7 +474,7 @@ found:
> pos + PCI_SRIOV_BAR + i * 4);
> if (!res->flags)
> continue;
> - if (resource_size(res) & (PAGE_SIZE - 1)) {
> + if (resource_size(res) & (HW_PAGE_SIZE - 1)) {
> rc = -EIO;
> goto failed;
> }
>
but HW_PAGE_SIZE is only defined for powerpc.
also there is PAGE_SHIFT around in that function.
maybe you can just define another MARCO according to IOV spec?
Thanks
Yinghai
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init
2012-01-27 21:05 ` Yinghai Lu
@ 2012-01-29 13:11 ` Vaidyanathan Srinivasan
0 siblings, 0 replies; 9+ messages in thread
From: Vaidyanathan Srinivasan @ 2012-01-29 13:11 UTC (permalink / raw)
To: Yinghai Lu; +Cc: Ram Pai, Jesse Barnes, linux-pci, linux-kernel
* Yinghai Lu <yinghai@kernel.org> [2012-01-27 13:05:41]:
> On Fri, Jan 27, 2012 at 11:10 AM, Vaidyanathan Srinivasan
> <svaidy@linux.vnet.ibm.com> wrote:
> > Hi Ram and Jesse,
> >
> > I found a trivial issue with page size alignment check on IBM POWER
> > box with 64k base page size. In sriov_init(), changing the check from
> > PAGE_SIZE (arch and config dependent) to HW_PAGE_SIZE (always 4k) was
> > required to use one of the sriov adapter as PF since the
> > resource_size() comes up as 0x8000 and PAGE_SIZE would be 0x10000 for
> > pseries boxes.
> >
> > I think resource_size() could be less than SystemPageSize, but I would
> > like your comments/ack/nack on any consequences of checking for only
> > 4k alignment here in a system with larger base page size.
> >
> > Thanks,
> > Vaidy
> >
> > ---
> >
> > pci: check for 4k resource_size alignment in sriov_init
> >
> > pci sriov_init should check for 4k page size alignment of resource_size
> > even if base page size is larger -- like 64k in powerpc.
> >
> > Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
> >
> > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> > index 0321fa3..5816fa0 100644
> > --- a/drivers/pci/iov.c
> > +++ b/drivers/pci/iov.c
> > @@ -474,7 +474,7 @@ found:
> > pos + PCI_SRIOV_BAR + i * 4);
> > if (!res->flags)
> > continue;
> > - if (resource_size(res) & (PAGE_SIZE - 1)) {
> > + if (resource_size(res) & (HW_PAGE_SIZE - 1)) {
> > rc = -EIO;
> > goto failed;
> > }
> >
>
> but HW_PAGE_SIZE is only defined for powerpc.
My bad, I picked the #define used in other powerpc code.
> also there is PAGE_SHIFT around in that function.
This gets defined correctly if CONFIG_PPC_64K_PAGES=y. But the actual
problem is the need for a generic 4K #define for x86 and other archs.
> maybe you can just define another MARCO according to IOV spec?
This is a good idea. I could not find an IOV specific requirement.
The resource size has to be a multiple of 4K so that the overall
resource-size() * PCI_SRIOV_TOTAL_VF will be a multiple of 4K or more.
Let me share a patch that has a simple #define in drivers/pci/pci.h,
which is needed to get SRIOV cards started on powerpc box.
--Vaidy
---
pci: check for 4k resource_size alignment in sriov_init
pci sriov_init should check for 4k page size alignment of resource_size
even if base page size is larger like 64k in powerpc.
Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
index 0321fa3..10adede 100644
--- a/drivers/pci/iov.c
+++ b/drivers/pci/iov.c
@@ -474,7 +474,7 @@ found:
pos + PCI_SRIOV_BAR + i * 4);
if (!res->flags)
continue;
- if (resource_size(res) & (PAGE_SIZE - 1)) {
+ if (resource_size(res) & (HW_PAGE_SIZE_4K - 1)) {
rc = -EIO;
goto failed;
}
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 1009a5e..68703ab 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -6,6 +6,9 @@
#define PCI_CFG_SPACE_SIZE 256
#define PCI_CFG_SPACE_EXP_SIZE 4096
+/* Constants used in the PCI core code */
+#define HW_PAGE_SIZE_4K 0x1000
+
/* Functions internal to the PCI core code */
extern int pci_uevent(struct device *dev, struct kobj_uevent_env *env);
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init
2012-01-27 19:10 [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init Vaidyanathan Srinivasan
2012-01-27 21:05 ` Yinghai Lu
@ 2012-01-30 3:18 ` Ram Pai
2012-01-31 17:44 ` Vaidyanathan Srinivasan
1 sibling, 1 reply; 9+ messages in thread
From: Ram Pai @ 2012-01-30 3:18 UTC (permalink / raw)
To: Vaidyanathan Srinivasan
Cc: Ram Pai, Jesse Barnes, Yinghai Lu, linux-pci, linux-kernel
On Sat, Jan 28, 2012 at 12:40:32AM +0530, Vaidyanathan Srinivasan wrote:
> Hi Ram and Jesse,
>
> I found a trivial issue with page size alignment check on IBM POWER
> box with 64k base page size. In sriov_init(), changing the check from
> PAGE_SIZE (arch and config dependent) to HW_PAGE_SIZE (always 4k) was
> required to use one of the sriov adapter as PF since the
> resource_size() comes up as 0x8000 and PAGE_SIZE would be 0x10000 for
> pseries boxes.
>
> I think resource_size() could be less than SystemPageSize, but I would
> like your comments/ack/nack on any consequences of checking for only
> 4k alignment here in a system with larger base page size.
As per the SRIOV specs, the resource has to be System page size aligned.
PFs are required to support 4-KB, 8-KB, 64-KB, 256-KB, 1-MB, and 4-MB
page sizes. In your case if your adapter's PF is not supporting 64K page size
then I think it is not conforming to the PCI SRIOV spec.
RP
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init
2012-01-30 3:18 ` Ram Pai
@ 2012-01-31 17:44 ` Vaidyanathan Srinivasan
2012-02-01 6:21 ` Ram Pai
0 siblings, 1 reply; 9+ messages in thread
From: Vaidyanathan Srinivasan @ 2012-01-31 17:44 UTC (permalink / raw)
To: Ram Pai; +Cc: Jesse Barnes, Yinghai Lu, linux-pci, linux-kernel
* Ram Pai <linuxram@us.ibm.com> [2012-01-30 11:18:45]:
> On Sat, Jan 28, 2012 at 12:40:32AM +0530, Vaidyanathan Srinivasan wrote:
> > Hi Ram and Jesse,
> >
> > I found a trivial issue with page size alignment check on IBM POWER
> > box with 64k base page size. In sriov_init(), changing the check from
> > PAGE_SIZE (arch and config dependent) to HW_PAGE_SIZE (always 4k) was
> > required to use one of the sriov adapter as PF since the
> > resource_size() comes up as 0x8000 and PAGE_SIZE would be 0x10000 for
> > pseries boxes.
> >
> > I think resource_size() could be less than SystemPageSize, but I would
> > like your comments/ack/nack on any consequences of checking for only
> > 4k alignment here in a system with larger base page size.
>
> As per the SRIOV specs, the resource has to be System page size aligned.
>
> PFs are required to support 4-KB, 8-KB, 64-KB, 256-KB, 1-MB, and 4-MB
> page sizes. In your case if your adapter's PF is not supporting 64K page size
> then I think it is not conforming to the PCI SRIOV spec.
Hi Ram,
Thanks for the pointer. I did some more experiments and found that
the card does support 64k page size, but the PCI_SRIOV_SYS_PGSIZE was
set to default 4k when we do the query and check resource_size().
You were correct, the resource_size() has to come up with 64k on 64k
PAGE_SIZE system. We should not change that check. I was able to
get a working solution by setting PCI_SRIOV_SYS_PGSIZE to 64k before
we do the query.
This was the case in the original code before you moved these to
sriov_enable(). If it is ok to leave the SYS_PGSIZE setting in
sriov_init(), then I have the following fix that works for me.
Please review and let me know your comments.
Thanks,
Vaidy
---
pci: set pci sriov page size before reading sriov bar
For an SRIOV device, PCI_SRIOV_SYS_PGSIZE should be set before
the PCI_SRIOV_BAR is queried. The sys pagesize defaults to 4k,
so this change is required on powerpc box with 64k base page size.
This is a regression caused due to moving SRIOV init to sriov_enable().
| commit afd24ece5c76af87f6fc477f2747b83a764f161c
| Author: Ram Pai <linuxram@us.ibm.com>
| PCI: delay configuration of SRIOV capability
| The SRIOV capability, namely page size and total_vfs of a device are
| configured during enumeration phase of the device. This can potentially
| interfere with the PCI operations of the platform, if the IOV capability
| of the device is not enabled.
Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
index 0321fa3..0dab5ec 100644
--- a/drivers/pci/iov.c
+++ b/drivers/pci/iov.c
@@ -347,8 +347,6 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
return rc;
}
- pci_write_config_dword(dev, iov->pos + PCI_SRIOV_SYS_PGSIZE, iov->pgsz);
-
iov->ctrl |= PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE;
pci_cfg_access_lock(dev);
pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl);
@@ -466,6 +464,7 @@ found:
return -EIO;
pgsz &= ~(pgsz - 1);
+ pci_write_config_dword(dev, pos + PCI_SRIOV_SYS_PGSIZE, pgsz);
nres = 0;
for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init
2012-01-31 17:44 ` Vaidyanathan Srinivasan
@ 2012-02-01 6:21 ` Ram Pai
2012-02-01 13:02 ` Vaidyanathan Srinivasan
0 siblings, 1 reply; 9+ messages in thread
From: Ram Pai @ 2012-02-01 6:21 UTC (permalink / raw)
To: Vaidyanathan Srinivasan
Cc: Ram Pai, Jesse Barnes, Yinghai Lu, linux-pci, linux-kernel
On Tue, Jan 31, 2012 at 11:14:02PM +0530, Vaidyanathan Srinivasan wrote:
> * Ram Pai <linuxram@us.ibm.com> [2012-01-30 11:18:45]:
>
> > On Sat, Jan 28, 2012 at 12:40:32AM +0530, Vaidyanathan Srinivasan wrote:
> > > Hi Ram and Jesse,
> > >
> > > I found a trivial issue with page size alignment check on IBM POWER
> > > box with 64k base page size. In sriov_init(), changing the check from
> > > PAGE_SIZE (arch and config dependent) to HW_PAGE_SIZE (always 4k) was
> > > required to use one of the sriov adapter as PF since the
> > > resource_size() comes up as 0x8000 and PAGE_SIZE would be 0x10000 for
> > > pseries boxes.
> > >
> > > I think resource_size() could be less than SystemPageSize, but I would
> > > like your comments/ack/nack on any consequences of checking for only
> > > 4k alignment here in a system with larger base page size.
> >
> > As per the SRIOV specs, the resource has to be System page size aligned.
> >
> > PFs are required to support 4-KB, 8-KB, 64-KB, 256-KB, 1-MB, and 4-MB
> > page sizes. In your case if your adapter's PF is not supporting 64K page size
> > then I think it is not conforming to the PCI SRIOV spec.
>
> Hi Ram,
>
> Thanks for the pointer. I did some more experiments and found that
> the card does support 64k page size, but the PCI_SRIOV_SYS_PGSIZE was
> set to default 4k when we do the query and check resource_size().
>
> You were correct, the resource_size() has to come up with 64k on 64k
> PAGE_SIZE system. We should not change that check. I was able to
> get a working solution by setting PCI_SRIOV_SYS_PGSIZE to 64k before
> we do the query.
>
> This was the case in the original code before you moved these to
> sriov_enable(). If it is ok to leave the SYS_PGSIZE setting in
> sriov_init(), then I have the following fix that works for me.
>
> Please review and let me know your comments.
>
> Thanks,
> Vaidy
> ---
>
> pci: set pci sriov page size before reading sriov bar
>
> For an SRIOV device, PCI_SRIOV_SYS_PGSIZE should be set before
> the PCI_SRIOV_BAR is queried. The sys pagesize defaults to 4k,
> so this change is required on powerpc box with 64k base page size.
>
> This is a regression caused due to moving SRIOV init to sriov_enable().
>
> | commit afd24ece5c76af87f6fc477f2747b83a764f161c
> | Author: Ram Pai <linuxram@us.ibm.com>
>
> | PCI: delay configuration of SRIOV capability
> | The SRIOV capability, namely page size and total_vfs of a device are
> | configured during enumeration phase of the device. This can potentially
> | interfere with the PCI operations of the platform, if the IOV capability
> | of the device is not enabled.
>
> Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
>
> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> index 0321fa3..0dab5ec 100644
> --- a/drivers/pci/iov.c
> +++ b/drivers/pci/iov.c
> @@ -347,8 +347,6 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
> return rc;
> }
>
> - pci_write_config_dword(dev, iov->pos + PCI_SRIOV_SYS_PGSIZE, iov->pgsz);
> -
> iov->ctrl |= PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE;
> pci_cfg_access_lock(dev);
> pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl);
> @@ -466,6 +464,7 @@ found:
> return -EIO;
>
> pgsz &= ~(pgsz - 1);
> + pci_write_config_dword(dev, pos + PCI_SRIOV_SYS_PGSIZE, pgsz);
>
> nres = 0;
> for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
ACK. I think it is better to revert afd24ece5c76af87f6fc477f2747b83a764f161c.
RP
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init
2012-02-01 6:21 ` Ram Pai
@ 2012-02-01 13:02 ` Vaidyanathan Srinivasan
2012-02-10 19:54 ` Jesse Barnes
0 siblings, 1 reply; 9+ messages in thread
From: Vaidyanathan Srinivasan @ 2012-02-01 13:02 UTC (permalink / raw)
To: Ram Pai; +Cc: Jesse Barnes, Yinghai Lu, linux-pci, linux-kernel
* Ram Pai <linuxram@us.ibm.com> [2012-02-01 14:21:45]:
> On Tue, Jan 31, 2012 at 11:14:02PM +0530, Vaidyanathan Srinivasan wrote:
> > * Ram Pai <linuxram@us.ibm.com> [2012-01-30 11:18:45]:
> >
> > > On Sat, Jan 28, 2012 at 12:40:32AM +0530, Vaidyanathan Srinivasan wrote:
> > > > Hi Ram and Jesse,
> > > >
> > > > I found a trivial issue with page size alignment check on IBM POWER
> > > > box with 64k base page size. In sriov_init(), changing the check from
> > > > PAGE_SIZE (arch and config dependent) to HW_PAGE_SIZE (always 4k) was
> > > > required to use one of the sriov adapter as PF since the
> > > > resource_size() comes up as 0x8000 and PAGE_SIZE would be 0x10000 for
> > > > pseries boxes.
> > > >
> > > > I think resource_size() could be less than SystemPageSize, but I would
> > > > like your comments/ack/nack on any consequences of checking for only
> > > > 4k alignment here in a system with larger base page size.
> > >
> > > As per the SRIOV specs, the resource has to be System page size aligned.
> > >
> > > PFs are required to support 4-KB, 8-KB, 64-KB, 256-KB, 1-MB, and 4-MB
> > > page sizes. In your case if your adapter's PF is not supporting 64K page size
> > > then I think it is not conforming to the PCI SRIOV spec.
> >
> > Hi Ram,
> >
> > Thanks for the pointer. I did some more experiments and found that
> > the card does support 64k page size, but the PCI_SRIOV_SYS_PGSIZE was
> > set to default 4k when we do the query and check resource_size().
> >
> > You were correct, the resource_size() has to come up with 64k on 64k
> > PAGE_SIZE system. We should not change that check. I was able to
> > get a working solution by setting PCI_SRIOV_SYS_PGSIZE to 64k before
> > we do the query.
> >
> > This was the case in the original code before you moved these to
> > sriov_enable(). If it is ok to leave the SYS_PGSIZE setting in
> > sriov_init(), then I have the following fix that works for me.
> >
> > Please review and let me know your comments.
> >
> > Thanks,
> > Vaidy
> > ---
> >
> > pci: set pci sriov page size before reading sriov bar
> >
> > For an SRIOV device, PCI_SRIOV_SYS_PGSIZE should be set before
> > the PCI_SRIOV_BAR is queried. The sys pagesize defaults to 4k,
> > so this change is required on powerpc box with 64k base page size.
> >
> > This is a regression caused due to moving SRIOV init to sriov_enable().
> >
> > | commit afd24ece5c76af87f6fc477f2747b83a764f161c
> > | Author: Ram Pai <linuxram@us.ibm.com>
> >
> > | PCI: delay configuration of SRIOV capability
> > | The SRIOV capability, namely page size and total_vfs of a device are
> > | configured during enumeration phase of the device. This can potentially
> > | interfere with the PCI operations of the platform, if the IOV capability
> > | of the device is not enabled.
> >
> > Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
> >
> > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> > index 0321fa3..0dab5ec 100644
> > --- a/drivers/pci/iov.c
> > +++ b/drivers/pci/iov.c
> > @@ -347,8 +347,6 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
> > return rc;
> > }
> >
> > - pci_write_config_dword(dev, iov->pos + PCI_SRIOV_SYS_PGSIZE, iov->pgsz);
> > -
> > iov->ctrl |= PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE;
> > pci_cfg_access_lock(dev);
> > pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl);
> > @@ -466,6 +464,7 @@ found:
> > return -EIO;
> >
> > pgsz &= ~(pgsz - 1);
> > + pci_write_config_dword(dev, pos + PCI_SRIOV_SYS_PGSIZE, pgsz);
> >
> > nres = 0;
> > for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
>
>
> ACK. I think it is better to revert afd24ece5c76af87f6fc477f2747b83a764f161c.
Hi Ram,
Thanks for the ack. But afd24ece5c76af87f6fc477f2747b83a764f161c has
one more change of moving
pci_write_config_word(dev, pos + PCI_SRIOV_NUM_VF, total) to sriov_enable().
This change is required so that we set the PCI_SRIOV_NUM_VF only
during sriov_enable.
So we should not revert the entire commit, we can just add this change.
--Vaidy
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init
2012-02-01 13:02 ` Vaidyanathan Srinivasan
@ 2012-02-10 19:54 ` Jesse Barnes
2012-02-13 3:08 ` Ram Pai
0 siblings, 1 reply; 9+ messages in thread
From: Jesse Barnes @ 2012-02-10 19:54 UTC (permalink / raw)
To: svaidy; +Cc: Ram Pai, Yinghai Lu, linux-pci, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 4719 bytes --]
On Wed, 1 Feb 2012 18:32:06 +0530
Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com> wrote:
> * Ram Pai <linuxram@us.ibm.com> [2012-02-01 14:21:45]:
>
> > On Tue, Jan 31, 2012 at 11:14:02PM +0530, Vaidyanathan Srinivasan wrote:
> > > * Ram Pai <linuxram@us.ibm.com> [2012-01-30 11:18:45]:
> > >
> > > > On Sat, Jan 28, 2012 at 12:40:32AM +0530, Vaidyanathan Srinivasan wrote:
> > > > > Hi Ram and Jesse,
> > > > >
> > > > > I found a trivial issue with page size alignment check on IBM POWER
> > > > > box with 64k base page size. In sriov_init(), changing the check from
> > > > > PAGE_SIZE (arch and config dependent) to HW_PAGE_SIZE (always 4k) was
> > > > > required to use one of the sriov adapter as PF since the
> > > > > resource_size() comes up as 0x8000 and PAGE_SIZE would be 0x10000 for
> > > > > pseries boxes.
> > > > >
> > > > > I think resource_size() could be less than SystemPageSize, but I would
> > > > > like your comments/ack/nack on any consequences of checking for only
> > > > > 4k alignment here in a system with larger base page size.
> > > >
> > > > As per the SRIOV specs, the resource has to be System page size aligned.
> > > >
> > > > PFs are required to support 4-KB, 8-KB, 64-KB, 256-KB, 1-MB, and 4-MB
> > > > page sizes. In your case if your adapter's PF is not supporting 64K page size
> > > > then I think it is not conforming to the PCI SRIOV spec.
> > >
> > > Hi Ram,
> > >
> > > Thanks for the pointer. I did some more experiments and found that
> > > the card does support 64k page size, but the PCI_SRIOV_SYS_PGSIZE was
> > > set to default 4k when we do the query and check resource_size().
> > >
> > > You were correct, the resource_size() has to come up with 64k on 64k
> > > PAGE_SIZE system. We should not change that check. I was able to
> > > get a working solution by setting PCI_SRIOV_SYS_PGSIZE to 64k before
> > > we do the query.
> > >
> > > This was the case in the original code before you moved these to
> > > sriov_enable(). If it is ok to leave the SYS_PGSIZE setting in
> > > sriov_init(), then I have the following fix that works for me.
> > >
> > > Please review and let me know your comments.
> > >
> > > Thanks,
> > > Vaidy
> > > ---
> > >
> > > pci: set pci sriov page size before reading sriov bar
> > >
> > > For an SRIOV device, PCI_SRIOV_SYS_PGSIZE should be set before
> > > the PCI_SRIOV_BAR is queried. The sys pagesize defaults to 4k,
> > > so this change is required on powerpc box with 64k base page size.
> > >
> > > This is a regression caused due to moving SRIOV init to sriov_enable().
> > >
> > > | commit afd24ece5c76af87f6fc477f2747b83a764f161c
> > > | Author: Ram Pai <linuxram@us.ibm.com>
> > >
> > > | PCI: delay configuration of SRIOV capability
> > > | The SRIOV capability, namely page size and total_vfs of a device are
> > > | configured during enumeration phase of the device. This can potentially
> > > | interfere with the PCI operations of the platform, if the IOV capability
> > > | of the device is not enabled.
> > >
> > > Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
> > >
> > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> > > index 0321fa3..0dab5ec 100644
> > > --- a/drivers/pci/iov.c
> > > +++ b/drivers/pci/iov.c
> > > @@ -347,8 +347,6 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
> > > return rc;
> > > }
> > >
> > > - pci_write_config_dword(dev, iov->pos + PCI_SRIOV_SYS_PGSIZE, iov->pgsz);
> > > -
> > > iov->ctrl |= PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE;
> > > pci_cfg_access_lock(dev);
> > > pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl);
> > > @@ -466,6 +464,7 @@ found:
> > > return -EIO;
> > >
> > > pgsz &= ~(pgsz - 1);
> > > + pci_write_config_dword(dev, pos + PCI_SRIOV_SYS_PGSIZE, pgsz);
> > >
> > > nres = 0;
> > > for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
> >
> >
> > ACK. I think it is better to revert afd24ece5c76af87f6fc477f2747b83a764f161c.
>
> Hi Ram,
>
> Thanks for the ack. But afd24ece5c76af87f6fc477f2747b83a764f161c has
> one more change of moving
> pci_write_config_word(dev, pos + PCI_SRIOV_NUM_VF, total) to sriov_enable().
>
> This change is required so that we set the PCI_SRIOV_NUM_VF only
> during sriov_enable.
>
> So we should not revert the entire commit, we can just add this change.
So which is it Ram, the ack or the revert? :)
Having the right page size early seems like the right solution...
--
Jesse Barnes, Intel Open Source Technology Center
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init
2012-02-10 19:54 ` Jesse Barnes
@ 2012-02-13 3:08 ` Ram Pai
0 siblings, 0 replies; 9+ messages in thread
From: Ram Pai @ 2012-02-13 3:08 UTC (permalink / raw)
To: Jesse Barnes; +Cc: svaidy, Ram Pai, Yinghai Lu, linux-pci, linux-kernel
On Fri, Feb 10, 2012 at 11:54:52AM -0800, Jesse Barnes wrote:
> On Wed, 1 Feb 2012 18:32:06 +0530
> Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com> wrote:
>
> > * Ram Pai <linuxram@us.ibm.com> [2012-02-01 14:21:45]:
> >
> > > On Tue, Jan 31, 2012 at 11:14:02PM +0530, Vaidyanathan Srinivasan wrote:
> > > > * Ram Pai <linuxram@us.ibm.com> [2012-01-30 11:18:45]:
> > > >
> > > > > On Sat, Jan 28, 2012 at 12:40:32AM +0530, Vaidyanathan Srinivasan wrote:
> > > > > > Hi Ram and Jesse,
> > > > > >
> > > > > > I found a trivial issue with page size alignment check on IBM POWER
> > > > > > box with 64k base page size. In sriov_init(), changing the check from
> > > > > > PAGE_SIZE (arch and config dependent) to HW_PAGE_SIZE (always 4k) was
> > > > > > required to use one of the sriov adapter as PF since the
> > > > > > resource_size() comes up as 0x8000 and PAGE_SIZE would be 0x10000 for
> > > > > > pseries boxes.
> > > > > >
> > > > > > I think resource_size() could be less than SystemPageSize, but I would
> > > > > > like your comments/ack/nack on any consequences of checking for only
> > > > > > 4k alignment here in a system with larger base page size.
> > > > >
> > > > > As per the SRIOV specs, the resource has to be System page size aligned.
> > > > >
> > > > > PFs are required to support 4-KB, 8-KB, 64-KB, 256-KB, 1-MB, and 4-MB
> > > > > page sizes. In your case if your adapter's PF is not supporting 64K page size
> > > > > then I think it is not conforming to the PCI SRIOV spec.
> > > >
> > > > Hi Ram,
> > > >
> > > > Thanks for the pointer. I did some more experiments and found that
> > > > the card does support 64k page size, but the PCI_SRIOV_SYS_PGSIZE was
> > > > set to default 4k when we do the query and check resource_size().
> > > >
> > > > You were correct, the resource_size() has to come up with 64k on 64k
> > > > PAGE_SIZE system. We should not change that check. I was able to
> > > > get a working solution by setting PCI_SRIOV_SYS_PGSIZE to 64k before
> > > > we do the query.
> > > >
> > > > This was the case in the original code before you moved these to
> > > > sriov_enable(). If it is ok to leave the SYS_PGSIZE setting in
> > > > sriov_init(), then I have the following fix that works for me.
> > > >
> > > > Please review and let me know your comments.
> > > >
> > > > Thanks,
> > > > Vaidy
> > > > ---
> > > >
> > > > pci: set pci sriov page size before reading sriov bar
> > > >
> > > > For an SRIOV device, PCI_SRIOV_SYS_PGSIZE should be set before
> > > > the PCI_SRIOV_BAR is queried. The sys pagesize defaults to 4k,
> > > > so this change is required on powerpc box with 64k base page size.
> > > >
> > > > This is a regression caused due to moving SRIOV init to sriov_enable().
> > > >
> > > > | commit afd24ece5c76af87f6fc477f2747b83a764f161c
> > > > | Author: Ram Pai <linuxram@us.ibm.com>
> > > >
> > > > | PCI: delay configuration of SRIOV capability
> > > > | The SRIOV capability, namely page size and total_vfs of a device are
> > > > | configured during enumeration phase of the device. This can potentially
> > > > | interfere with the PCI operations of the platform, if the IOV capability
> > > > | of the device is not enabled.
> > > >
> > > > Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
> > > >
> > > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> > > > index 0321fa3..0dab5ec 100644
> > > > --- a/drivers/pci/iov.c
> > > > +++ b/drivers/pci/iov.c
> > > > @@ -347,8 +347,6 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
> > > > return rc;
> > > > }
> > > >
> > > > - pci_write_config_dword(dev, iov->pos + PCI_SRIOV_SYS_PGSIZE, iov->pgsz);
> > > > -
> > > > iov->ctrl |= PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE;
> > > > pci_cfg_access_lock(dev);
> > > > pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl);
> > > > @@ -466,6 +464,7 @@ found:
> > > > return -EIO;
> > > >
> > > > pgsz &= ~(pgsz - 1);
> > > > + pci_write_config_dword(dev, pos + PCI_SRIOV_SYS_PGSIZE, pgsz);
> > > >
> > > > nres = 0;
> > > > for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
> > >
> > >
> > > ACK. I think it is better to revert afd24ece5c76af87f6fc477f2747b83a764f161c.
> >
> > Hi Ram,
> >
> > Thanks for the ack. But afd24ece5c76af87f6fc477f2747b83a764f161c has
> > one more change of moving
> > pci_write_config_word(dev, pos + PCI_SRIOV_NUM_VF, total) to sriov_enable().
> >
> > This change is required so that we set the PCI_SRIOV_NUM_VF only
> > during sriov_enable.
> >
> > So we should not revert the entire commit, we can just add this change.
>
> So which is it Ram, the ack or the revert? :)
Jesse,
As Vaidy mentioned, revert is not the right solution. So dont revert.
But apply Vaidy's patch.
>
> Having the right page size early seems like the right solution...
Yes.
RP
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2012-02-13 3:08 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-01-27 19:10 [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init Vaidyanathan Srinivasan
2012-01-27 21:05 ` Yinghai Lu
2012-01-29 13:11 ` Vaidyanathan Srinivasan
2012-01-30 3:18 ` Ram Pai
2012-01-31 17:44 ` Vaidyanathan Srinivasan
2012-02-01 6:21 ` Ram Pai
2012-02-01 13:02 ` Vaidyanathan Srinivasan
2012-02-10 19:54 ` Jesse Barnes
2012-02-13 3:08 ` Ram Pai
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).