Fixes for 64bit window handling on AMD CPUs

Fixes for 64bit window handling on AMD CPUs

[Christian König]

Hi Bjorn,

please include the following three patches in your next fix pull request for -rc2.

They just restrict the fixup of the 64bit window on AMD CPU to avoid problems with Xen and multi socket systems.

If you prefer a pull request just say so and I will setup a branch on a public server.

Thanks,
Christian.

[PATCH 1/3] x86/PCI: fix infinity loop in search for 64bit BAR placement

[Christian König]

Break the loop if we can't find some address space for a 64bit BAR.

Signed-off-by: Christian König <christian.koenig@amd.com>
---
 arch/x86/pci/fixup.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/arch/x86/pci/fixup.c b/arch/x86/pci/fixup.c
index e59378bf37d9..e857b3ac5755 100644
--- a/arch/x86/pci/fixup.c
+++ b/arch/x86/pci/fixup.c
@@ -695,8 +695,13 @@ static void pci_amd_enable_64bit_bar(struct pci_dev *dev)
 	res->end = 0xfd00000000ull - 1;
 
 	/* Just grab the free area behind system memory for this */
-	while ((conflict = request_resource_conflict(&iomem_resource, res)))
+	while ((conflict = request_resource_conflict(&iomem_resource, res))) {
+		if (conflict->end >= res->end) {
+			kfree(res);
+			return;
+		}
 		res->start = conflict->end + 1;
+	}
 
 	dev_info(&dev->dev, "adding root bus resource %pR\n", res);
 
-- 
2.11.0

[PATCH 2/3] x86/PCI: only enable a 64bit BAR on single socket AMD Family 15h systems

[Christian König]

When we have a multi socket system each CPU core needs the same setup. Since
this is tricky to do in the fixup code disable enabling a 64bit BAR on multi
socket systems for now.

Signed-off-by: Christian König <christian.koenig@amd.com>
---
 arch/x86/pci/fixup.c | 20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)

diff --git a/arch/x86/pci/fixup.c b/arch/x86/pci/fixup.c
index e857b3ac5755..c817ab85dc82 100644
--- a/arch/x86/pci/fixup.c
+++ b/arch/x86/pci/fixup.c
@@ -664,6 +664,16 @@ static void pci_amd_enable_64bit_bar(struct pci_dev *dev)
 	unsigned i;
 	u32 base, limit, high;
 	struct resource *res, *conflict;
+	struct pci_dev *other;
+
+	/* Check that we are the only device of that type */
+	other = pci_get_device(dev->vendor, dev->device, NULL);
+	if (other != dev ||
+	    (other = pci_get_device(dev->vendor, dev->device, other))) {
+		/* This is a multi socket system, don't touch it for now */
+		pci_dev_put(other);
+		return;
+	}
 
 	for (i = 0; i < 8; i++) {
 		pci_read_config_dword(dev, AMD_141b_MMIO_BASE(i), &base);
@@ -718,10 +728,10 @@ static void pci_amd_enable_64bit_bar(struct pci_dev *dev)
 
 	pci_bus_add_resource(dev->bus, res, 0);
 }
-DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x1401, pci_amd_enable_64bit_bar);
-DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x141b, pci_amd_enable_64bit_bar);
-DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x1571, pci_amd_enable_64bit_bar);
-DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x15b1, pci_amd_enable_64bit_bar);
-DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x1601, pci_amd_enable_64bit_bar);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x1401, pci_amd_enable_64bit_bar);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x141b, pci_amd_enable_64bit_bar);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x1571, pci_amd_enable_64bit_bar);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x15b1, pci_amd_enable_64bit_bar);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x1601, pci_amd_enable_64bit_bar);
 
 #endif
-- 
2.11.0

[PATCH 3/3] x86/PCI: limit the size of the 64bit BAR to 256GB

[Christian König]

This avoids problems with Xen which hides some memory resources from the
OS and potentially also allows memory hotplug while this fixup is
enabled.

Signed-off-by: Christian König <christian.koenig@amd.com>
---
 arch/x86/pci/fixup.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/pci/fixup.c b/arch/x86/pci/fixup.c
index c817ab85dc82..149adbc7f2a3 100644
--- a/arch/x86/pci/fixup.c
+++ b/arch/x86/pci/fixup.c
@@ -701,7 +701,7 @@ static void pci_amd_enable_64bit_bar(struct pci_dev *dev)
 	res->name = "PCI Bus 0000:00";
 	res->flags = IORESOURCE_PREFETCH | IORESOURCE_MEM |
 		IORESOURCE_MEM_64 | IORESOURCE_WINDOW;
-	res->start = 0x100000000ull;
+	res->start = 0xbd00000000ull;
 	res->end = 0xfd00000000ull - 1;
 
 	/* Just grab the free area behind system memory for this */
-- 
2.11.0

Re: [PATCH 3/3] x86/PCI: limit the size of the 64bit BAR to 256GB

[Bjorn Helgaas]

On Wed, Nov 29, 2017 at 03:12:29PM +0100, Christian König wrote:
> This avoids problems with Xen which hides some memory resources from the
> OS and potentially also allows memory hotplug while this fixup is
> enabled.

The patch itself is OK, but the changelog doesn't say enough about
what the problem is.  I have no clue about what the Xen issue is or
why limiting the BAR to 256GB avoids the problem or what this has to
do with memory hotplug.

For example, we should be able to tell why 256GB is the right number.
Maybe there's something specific in Xen you can reference?  Maybe an
example of what goes wrong with some details?

> Signed-off-by: Christian König <christian.koenig@amd.com>
> ---
>  arch/x86/pci/fixup.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/arch/x86/pci/fixup.c b/arch/x86/pci/fixup.c
> index c817ab85dc82..149adbc7f2a3 100644
> --- a/arch/x86/pci/fixup.c
> +++ b/arch/x86/pci/fixup.c
> @@ -701,7 +701,7 @@ static void pci_amd_enable_64bit_bar(struct pci_dev *dev)
>  	res->name = "PCI Bus 0000:00";
>  	res->flags = IORESOURCE_PREFETCH | IORESOURCE_MEM |
>  		IORESOURCE_MEM_64 | IORESOURCE_WINDOW;
> -	res->start = 0x100000000ull;
> +	res->start = 0xbd00000000ull;
>  	res->end = 0xfd00000000ull - 1;
>  
>  	/* Just grab the free area behind system memory for this */
> -- 
> 2.11.0
> 

Re: Fixes for 64bit window handling on AMD CPUs

[Bjorn Helgaas]

On Wed, Nov 29, 2017 at 03:12:26PM +0100, Christian König wrote:
> Hi Bjorn,
> 
> please include the following three patches in your next fix pull request for -rc2.
> 
> They just restrict the fixup of the 64bit window on AMD CPU to avoid problems with Xen and multi socket systems.

Applied the first two to for-linus for v4.15, thanks!

I'll apply the third as soon as we get the changelog clarified.

Re: [PATCH 3/3] x86/PCI: limit the size of the 64bit BAR to 256GB

[Bjorn Helgaas]

On Wed, Dec 06, 2017 at 01:51:18PM -0600, Bjorn Helgaas wrote:
> On Wed, Nov 29, 2017 at 03:12:29PM +0100, Christian König wrote:
> > This avoids problems with Xen which hides some memory resources from the
> > OS and potentially also allows memory hotplug while this fixup is
> > enabled.
> 
> The patch itself is OK, but the changelog doesn't say enough about
> what the problem is.  I have no clue about what the Xen issue is or
> why limiting the BAR to 256GB avoids the problem or what this has to
> do with memory hotplug.
> 
> For example, we should be able to tell why 256GB is the right number.
> Maybe there's something specific in Xen you can reference?  Maybe an
> example of what goes wrong with some details?

Ping?  Is this change required to fix issues people are seeing?  If
so, we either need to rework the changelog and get it merged, or
revert the quirk as a whole.

I tentatively applied the first two patches to for-linus, but I
haven't asked Linus to pull them because I assumed we really needed
all three.

Bjorn

> > Signed-off-by: Christian König <christian.koenig@amd.com>
> > ---
> >  arch/x86/pci/fixup.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/arch/x86/pci/fixup.c b/arch/x86/pci/fixup.c
> > index c817ab85dc82..149adbc7f2a3 100644
> > --- a/arch/x86/pci/fixup.c
> > +++ b/arch/x86/pci/fixup.c
> > @@ -701,7 +701,7 @@ static void pci_amd_enable_64bit_bar(struct pci_dev *dev)
> >  	res->name = "PCI Bus 0000:00";
> >  	res->flags = IORESOURCE_PREFETCH | IORESOURCE_MEM |
> >  		IORESOURCE_MEM_64 | IORESOURCE_WINDOW;
> > -	res->start = 0x100000000ull;
> > +	res->start = 0xbd00000000ull;
> >  	res->end = 0xfd00000000ull - 1;
> >  
> >  	/* Just grab the free area behind system memory for this */
> > -- 
> > 2.11.0
> > 

Re: [PATCH 3/3] x86/PCI: limit the size of the 64bit BAR to 256GB

[Boris Ostrovsky]

On 12/08/2017 12:56 PM, Bjorn Helgaas wrote:
> On Wed, Dec 06, 2017 at 01:51:18PM -0600, Bjorn Helgaas wrote:
>> On Wed, Nov 29, 2017 at 03:12:29PM +0100, Christian König wrote:
>>> This avoids problems with Xen which hides some memory resources from the
>>> OS and potentially also allows memory hotplug while this fixup is
>>> enabled.
>> The patch itself is OK, but the changelog doesn't say enough about
>> what the problem is.  I have no clue about what the Xen issue is or
>> why limiting the BAR to 256GB avoids the problem or what this has to
>> do with memory hotplug.
>>
>> For example, we should be able to tell why 256GB is the right number.
>> Maybe there's something specific in Xen you can reference?  Maybe an
>> example of what goes wrong with some details?
> Ping?  Is this change required to fix issues people are seeing?  If
> so, we either need to rework the changelog and get it merged, or
> revert the quirk as a whole.
>
> I tentatively applied the first two patches to for-linus, but I
> haven't asked Linus to pull them because I assumed we really needed
> all three.

This is not a fix but rather is a workaround. The problem is that Xen
dom0 may be running with less than all of the system memory and the
chunk of host memory that dom0 doesn't have is not exposed in e820 as
reserved. And so pci_amd_enable_64bit_bar() assumes that it can be used
for MMIO, with predictable results. Only trying to use very high
addresses limits chances that there is memory there.

The alternative is to revert f5775e0b6116b7e2425ccf535243b21768566d87.

I have been working on a proper fix but haven't been able to finish it yet.

-boris


>
> Bjorn
>
>>> Signed-off-by: Christian König <christian.koenig@amd.com>
>>> ---
>>>  arch/x86/pci/fixup.c | 2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/arch/x86/pci/fixup.c b/arch/x86/pci/fixup.c
>>> index c817ab85dc82..149adbc7f2a3 100644
>>> --- a/arch/x86/pci/fixup.c
>>> +++ b/arch/x86/pci/fixup.c
>>> @@ -701,7 +701,7 @@ static void pci_amd_enable_64bit_bar(struct pci_dev *dev)
>>>  	res->name = "PCI Bus 0000:00";
>>>  	res->flags = IORESOURCE_PREFETCH | IORESOURCE_MEM |
>>>  		IORESOURCE_MEM_64 | IORESOURCE_WINDOW;
>>> -	res->start = 0x100000000ull;
>>> +	res->start = 0xbd00000000ull;
>>>  	res->end = 0xfd00000000ull - 1;
>>>  
>>>  	/* Just grab the free area behind system memory for this */
>>> -- 
>>> 2.11.0
>>>

Re: [PATCH 3/3] x86/PCI: limit the size of the 64bit BAR to 256GB

[Boris Ostrovsky]

On 12/08/2017 12:56 PM, Bjorn Helgaas wrote:
> On Wed, Dec 06, 2017 at 01:51:18PM -0600, Bjorn Helgaas wrote:
>> On Wed, Nov 29, 2017 at 03:12:29PM +0100, Christian König wrote:
>>> This avoids problems with Xen which hides some memory resources from the
>>> OS and potentially also allows memory hotplug while this fixup is
>>> enabled.
>> The patch itself is OK, but the changelog doesn't say enough about
>> what the problem is.  I have no clue about what the Xen issue is or
>> why limiting the BAR to 256GB avoids the problem or what this has to
>> do with memory hotplug.
>>
>> For example, we should be able to tell why 256GB is the right number.
>> Maybe there's something specific in Xen you can reference?  Maybe an
>> example of what goes wrong with some details?
> Ping?  Is this change required to fix issues people are seeing?  If
> so, we either need to rework the changelog and get it merged, or
> revert the quirk as a whole.
>
> I tentatively applied the first two patches to for-linus, but I
> haven't asked Linus to pull them because I assumed we really needed
> all three.

This is not a fix but rather is a workaround. The problem is that Xen
dom0 may be running with less than all of the system memory and the
chunk of host memory that dom0 doesn't have is not exposed in e820 as
reserved. And so pci_amd_enable_64bit_bar() assumes that it can be used
for MMIO, with predictable results. Only trying to use very high
addresses limits chances that there is memory there.

The alternative is to revert f5775e0b6116b7e2425ccf535243b21768566d87.

I have been working on a proper fix but haven't been able to finish it yet.

-boris


>
> Bjorn
>
>>> Signed-off-by: Christian König <christian.koenig@amd.com>
>>> ---
>>>  arch/x86/pci/fixup.c | 2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/arch/x86/pci/fixup.c b/arch/x86/pci/fixup.c
>>> index c817ab85dc82..149adbc7f2a3 100644
>>> --- a/arch/x86/pci/fixup.c
>>> +++ b/arch/x86/pci/fixup.c
>>> @@ -701,7 +701,7 @@ static void pci_amd_enable_64bit_bar(struct pci_dev *dev)
>>>  	res->name = "PCI Bus 0000:00";
>>>  	res->flags = IORESOURCE_PREFETCH | IORESOURCE_MEM |
>>>  		IORESOURCE_MEM_64 | IORESOURCE_WINDOW;
>>> -	res->start = 0x100000000ull;
>>> +	res->start = 0xbd00000000ull;
>>>  	res->end = 0xfd00000000ull - 1;
>>>  
>>>  	/* Just grab the free area behind system memory for this */
>>> -- 
>>> 2.11.0
>>>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 3/3] x86/PCI: limit the size of the 64bit BAR to 256GB

[Christian König]

Am 08.12.2017 um 18:56 schrieb Bjorn Helgaas:
> On Wed, Dec 06, 2017 at 01:51:18PM -0600, Bjorn Helgaas wrote:
>> On Wed, Nov 29, 2017 at 03:12:29PM +0100, Christian König wrote:
>>> This avoids problems with Xen which hides some memory resources from the
>>> OS and potentially also allows memory hotplug while this fixup is
>>> enabled.
>> The patch itself is OK, but the changelog doesn't say enough about
>> what the problem is.  I have no clue about what the Xen issue is or
>> why limiting the BAR to 256GB avoids the problem or what this has to
>> do with memory hotplug.
>>
>> For example, we should be able to tell why 256GB is the right number.
>> Maybe there's something specific in Xen you can reference?  Maybe an
>> example of what goes wrong with some details?
> Ping?

Sorry for the delay, first been to busy and then got a bad cold and 
today is the first day I've got out of bed again.

> Is this change required to fix issues people are seeing?  If
> so, we either need to rework the changelog and get it merged, or
> revert the quirk as a whole.

It's just a precaution to eventually avoid problems, I will try to 
provided a patch with updated commit log tomorrow.

> I tentatively applied the first two patches to for-linus, but I
> haven't asked Linus to pull them because I assumed we really needed
> all three.

The first two patches are perfectly enough for now.

Thanks,
Christian.

>
> Bjorn
>
>>> Signed-off-by: Christian König <christian.koenig@amd.com>
>>> ---
>>>   arch/x86/pci/fixup.c | 2 +-
>>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/arch/x86/pci/fixup.c b/arch/x86/pci/fixup.c
>>> index c817ab85dc82..149adbc7f2a3 100644
>>> --- a/arch/x86/pci/fixup.c
>>> +++ b/arch/x86/pci/fixup.c
>>> @@ -701,7 +701,7 @@ static void pci_amd_enable_64bit_bar(struct pci_dev *dev)
>>>   	res->name = "PCI Bus 0000:00";
>>>   	res->flags = IORESOURCE_PREFETCH | IORESOURCE_MEM |
>>>   		IORESOURCE_MEM_64 | IORESOURCE_WINDOW;
>>> -	res->start = 0x100000000ull;
>>> +	res->start = 0xbd00000000ull;
>>>   	res->end = 0xfd00000000ull - 1;
>>>   
>>>   	/* Just grab the free area behind system memory for this */
>>> -- 
>>> 2.11.0
>>>