All of lore.kernel.org
 help / color / mirror / Atom feed
From: Boris Ostrovsky <boris.ostrovsky@oracle.com>
To: Igor Druzhinin <igor.druzhinin@citrix.com>,
	xen-devel@lists.xenproject.org, linux-kernel@vger.kernel.org
Cc: jgross@suse.com
Subject: Re: [PATCH] xen/pci: try to reserve MCFG areas earlier
Date: Sun, 8 Sep 2019 14:28:15 -0400	[thread overview]
Message-ID: <43b7da04-5c42-80d8-898b-470ee1c91ed2@oracle.com> (raw)
In-Reply-To: <e3114d56-51cd-b973-1ada-f6a60a7e99cc@citrix.com>

On 9/6/19 7:00 PM, Igor Druzhinin wrote:
>
> On 06/09/2019 23:30, Boris Ostrovsky wrote:
>> On 9/3/19 8:20 PM, Igor Druzhinin wrote:
>>> If MCFG area is not reserved in E820, Xen by default will defer its usage
>>> until Dom0 registers it explicitly after ACPI parser recognizes it as
>>> a reserved resource in DSDT. Having it reserved in E820 is not
>>> mandatory according to "PCI Firmware Specification, rev 3.2" (par. 4.1.2)
>>> and firmware is free to keep a hole E820 in that place. Xen doesn't know
>>> what exactly is inside this hole since it lacks full ACPI view of the
>>> platform therefore it's potentially harmful to access MCFG region
>>> without additional checks as some machines are known to provide
>>> inconsistent information on the size of the region.
>>>
>>> Now xen_mcfg_late() runs after acpi_init() which is too late as some basic
>>> PCI enumeration starts exactly there. Trying to register a device prior
>>> to MCFG reservation causes multiple problems with PCIe extended
>>> capability initializations in Xen (e.g. SR-IOV VF BAR sizing). There are
>>> no convenient hooks for us to subscribe to so try to register MCFG
>>> areas earlier upon the first invocation of xen_add_device(). 
>>
>> Where is MCFG parsed? pci_arch_init()?
> It happens twice:
> 1) first time early one in pci_arch_init() that is arch_initcall - that
> time pci_mmcfg_list will be freed immediately there because MCFG area is
> not reserved in E820;
> 2) second time late one in acpi_init() which is subsystem_initcall right
> before where PCI enumeration starts - this time ACPI tables will be
> checked for a reserved resource and pci_mmcfg_list will be finally
> populated.
>
> The problem is that on a system that doesn't have MCFG area reserved in
> E820 pci_mmcfg_list is empty before acpi_init() and our PCI hooks are
> called in the same place. So MCFG is still not in use by Xen at this
> point since we haven't reached our xen_mcfg_late().


Would it be possible for us to parse MCFG ourselves in pci_xen_init()? I
realize that we'd be doing this twice (or maybe even three times since
apparently both pci_arch_init()  and acpi_ini() do it).

-boris

WARNING: multiple messages have this Message-ID (diff)
From: Boris Ostrovsky <boris.ostrovsky@oracle.com>
To: Igor Druzhinin <igor.druzhinin@citrix.com>,
	xen-devel@lists.xenproject.org, linux-kernel@vger.kernel.org
Cc: jgross@suse.com
Subject: Re: [Xen-devel] [PATCH] xen/pci: try to reserve MCFG areas earlier
Date: Sun, 8 Sep 2019 14:28:15 -0400	[thread overview]
Message-ID: <43b7da04-5c42-80d8-898b-470ee1c91ed2@oracle.com> (raw)
In-Reply-To: <e3114d56-51cd-b973-1ada-f6a60a7e99cc@citrix.com>

On 9/6/19 7:00 PM, Igor Druzhinin wrote:
>
> On 06/09/2019 23:30, Boris Ostrovsky wrote:
>> On 9/3/19 8:20 PM, Igor Druzhinin wrote:
>>> If MCFG area is not reserved in E820, Xen by default will defer its usage
>>> until Dom0 registers it explicitly after ACPI parser recognizes it as
>>> a reserved resource in DSDT. Having it reserved in E820 is not
>>> mandatory according to "PCI Firmware Specification, rev 3.2" (par. 4.1.2)
>>> and firmware is free to keep a hole E820 in that place. Xen doesn't know
>>> what exactly is inside this hole since it lacks full ACPI view of the
>>> platform therefore it's potentially harmful to access MCFG region
>>> without additional checks as some machines are known to provide
>>> inconsistent information on the size of the region.
>>>
>>> Now xen_mcfg_late() runs after acpi_init() which is too late as some basic
>>> PCI enumeration starts exactly there. Trying to register a device prior
>>> to MCFG reservation causes multiple problems with PCIe extended
>>> capability initializations in Xen (e.g. SR-IOV VF BAR sizing). There are
>>> no convenient hooks for us to subscribe to so try to register MCFG
>>> areas earlier upon the first invocation of xen_add_device(). 
>>
>> Where is MCFG parsed? pci_arch_init()?
> It happens twice:
> 1) first time early one in pci_arch_init() that is arch_initcall - that
> time pci_mmcfg_list will be freed immediately there because MCFG area is
> not reserved in E820;
> 2) second time late one in acpi_init() which is subsystem_initcall right
> before where PCI enumeration starts - this time ACPI tables will be
> checked for a reserved resource and pci_mmcfg_list will be finally
> populated.
>
> The problem is that on a system that doesn't have MCFG area reserved in
> E820 pci_mmcfg_list is empty before acpi_init() and our PCI hooks are
> called in the same place. So MCFG is still not in use by Xen at this
> point since we haven't reached our xen_mcfg_late().


Would it be possible for us to parse MCFG ourselves in pci_xen_init()? I
realize that we'd be doing this twice (or maybe even three times since
apparently both pci_arch_init()  and acpi_ini() do it).

-boris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

  reply	other threads:[~2019-09-08 18:28 UTC|newest]

Thread overview: 44+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-04  0:20 [PATCH] xen/pci: try to reserve MCFG areas earlier Igor Druzhinin
2019-09-04  0:20 ` [Xen-devel] " Igor Druzhinin
2019-09-04  9:08 ` Jan Beulich
2019-09-04  9:08   ` [Xen-devel] " Jan Beulich
2019-09-04 11:36   ` Igor Druzhinin
2019-09-04 11:36     ` [Xen-devel] " Igor Druzhinin
2019-09-04 12:09     ` Jan Beulich
2019-09-04 12:09       ` [Xen-devel] " Jan Beulich
2019-09-06 22:30 ` Boris Ostrovsky
2019-09-06 22:30   ` [Xen-devel] " Boris Ostrovsky
2019-09-06 23:00   ` Igor Druzhinin
2019-09-06 23:00     ` [Xen-devel] " Igor Druzhinin
2019-09-08 18:28     ` Boris Ostrovsky [this message]
2019-09-08 18:28       ` Boris Ostrovsky
2019-09-08 21:11       ` Igor Druzhinin
2019-09-08 21:11         ` [Xen-devel] " Igor Druzhinin
2019-09-08 23:30         ` Boris Ostrovsky
2019-09-08 23:30           ` [Xen-devel] " Boris Ostrovsky
2019-09-08 23:37           ` Igor Druzhinin
2019-09-08 23:37             ` [Xen-devel] " Igor Druzhinin
2019-09-09 19:19             ` Boris Ostrovsky
2019-09-09 19:19               ` [Xen-devel] " Boris Ostrovsky
2019-09-09 21:48               ` Igor Druzhinin
2019-09-09 21:48                 ` [Xen-devel] " Igor Druzhinin
2019-09-10  1:47                 ` Boris Ostrovsky
2019-09-10  1:47                   ` Boris Ostrovsky
2019-09-10  9:46                   ` Igor Druzhinin
2019-09-10  9:46                     ` Igor Druzhinin
2019-09-10  9:55                     ` Jan Beulich
2019-09-10  9:55                       ` Jan Beulich
2019-09-10 10:08                       ` Igor Druzhinin
2019-09-10 10:08                         ` Igor Druzhinin
2019-09-10 17:48                     ` Boris Ostrovsky
2019-09-10 17:48                       ` Boris Ostrovsky
2019-09-10 20:36                       ` Igor Druzhinin
2019-09-10 20:36                         ` Igor Druzhinin
2019-09-10 21:19                         ` Boris Ostrovsky
2019-09-10 21:19                           ` Boris Ostrovsky
2019-09-11  1:15                           ` Igor Druzhinin
2019-09-11  1:15                             ` Igor Druzhinin
2019-09-11  9:13                             ` Jan Beulich
2019-09-11  9:13                               ` Jan Beulich
2019-09-12 17:33                             ` Boris Ostrovsky
2019-09-12 17:33                               ` Boris Ostrovsky

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=43b7da04-5c42-80d8-898b-470ee1c91ed2@oracle.com \
    --to=boris.ostrovsky@oracle.com \
    --cc=igor.druzhinin@citrix.com \
    --cc=jgross@suse.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=xen-devel@lists.xenproject.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.