* Re: [Qemu-devel] [SeaBIOS] [RFC v2 0/3] Support multiple pci domains in pci_device [not found] ` <20180827070406.t525gr43qpu7wpsj@sirius.home.kraxel.org> @ 2018-08-28 4:12 ` Zihan Yang 2018-08-28 5:30 ` Gerd Hoffmann 2018-08-28 5:37 ` Marcel Apfelbaum 0 siblings, 2 replies; 9+ messages in thread From: Zihan Yang @ 2018-08-28 4:12 UTC (permalink / raw) To: Gerd Hoffmann; +Cc: qemu-devel, Marcel Apfelbaum Gerd Hoffmann <kraxel@redhat.com> 于2018年8月27日周一 上午7:04写道: > > Hi, > > > > However, QEMU only binds port 0xcf8 and 0xcfc to > > > bus pcie.0. To avoid bus confliction, we should use other port pairs for > > > busses under new domains. > > > > I would skip support for IO based configuration and use only MMCONFIG > > for extra root buses. > > > > The question remains: how do we assign MMCONFIG space for > > each PCI domain. > > Allocation-wise it would be easiest to place them above 4G. Right after > memory, or after etc/reserved-memory-end (if that fw_cfg file is > present), where the 64bit pci bars would have been placed. Move the pci > bars up in address space to make room. > > Only problem is that seabios wouldn't be able to access mmconfig then. > > Placing them below 4G would work at least for a few pci domains. q35 > mmconfig bar is placed at 0xb0000000 -> 0xbfffffff, basically for > historical reasons. Old qemu versions had 2.75G low memory on q35 (up > to 0xafffffff), and I think old machine types still have that for live > migration compatibility reasons. Modern qemu uses 2G only, to make > gigabyte alignment work. > > 32bit pci bars are placed above 0xc0000000. The address space from 2G > to 2.75G (0x8000000 -> 0xafffffff) is unused on new machine types. > Enough room for three additional mmconfig bars (full size), so four > pci domains total if you add the q35 one. Maybe we can support 4 domains first before we come up with a better solution. But I'm not sure if four domains are enough for those who want too many devices? > cheers, > Gerd > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [SeaBIOS] [RFC v2 0/3] Support multiple pci domains in pci_device 2018-08-28 4:12 ` [Qemu-devel] [SeaBIOS] [RFC v2 0/3] Support multiple pci domains in pci_device Zihan Yang @ 2018-08-28 5:30 ` Gerd Hoffmann 2018-08-28 5:37 ` Marcel Apfelbaum 1 sibling, 0 replies; 9+ messages in thread From: Gerd Hoffmann @ 2018-08-28 5:30 UTC (permalink / raw) To: Zihan Yang; +Cc: qemu-devel, Marcel Apfelbaum On Tue, Aug 28, 2018 at 04:12:17AM +0000, Zihan Yang wrote: > Gerd Hoffmann <kraxel@redhat.com> 于2018年8月27日周一 上午7:04写道: > > > > Hi, > > > > > > However, QEMU only binds port 0xcf8 and 0xcfc to > > > > bus pcie.0. To avoid bus confliction, we should use other port pairs for > > > > busses under new domains. > > > > > > I would skip support for IO based configuration and use only MMCONFIG > > > for extra root buses. > > > > > > The question remains: how do we assign MMCONFIG space for > > > each PCI domain. > > > > Allocation-wise it would be easiest to place them above 4G. Right after > > memory, or after etc/reserved-memory-end (if that fw_cfg file is > > present), where the 64bit pci bars would have been placed. Move the pci > > bars up in address space to make room. > > > > Only problem is that seabios wouldn't be able to access mmconfig then. > > > > Placing them below 4G would work at least for a few pci domains. q35 > > mmconfig bar is placed at 0xb0000000 -> 0xbfffffff, basically for > > historical reasons. Old qemu versions had 2.75G low memory on q35 (up > > to 0xafffffff), and I think old machine types still have that for live > > migration compatibility reasons. Modern qemu uses 2G only, to make > > gigabyte alignment work. > > > > 32bit pci bars are placed above 0xc0000000. The address space from 2G > > to 2.75G (0x8000000 -> 0xafffffff) is unused on new machine types. > > Enough room for three additional mmconfig bars (full size), so four > > pci domains total if you add the q35 one. > > Maybe we can support 4 domains first before we come up > with a better solution. But I'm not sure if four domains are > enough for those who want too many devices? There is the option to use ovmf instead. Runs in long mode, so mapping and using mmconfig above 4G isn't a problem for ovmf. Given that alternative exists I think it is reasonable to declare only 4 pci domains being fully supported with seabios. We can even configure mmconfig (above 4G) for the 5th and above, seabios just would not enumerate the devices and map the pci bars for these domains and leave it to the guest os instead. cheers, Gerd ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [SeaBIOS] [RFC v2 0/3] Support multiple pci domains in pci_device 2018-08-28 4:12 ` [Qemu-devel] [SeaBIOS] [RFC v2 0/3] Support multiple pci domains in pci_device Zihan Yang 2018-08-28 5:30 ` Gerd Hoffmann @ 2018-08-28 5:37 ` Marcel Apfelbaum 2018-08-28 6:07 ` Gerd Hoffmann 1 sibling, 1 reply; 9+ messages in thread From: Marcel Apfelbaum @ 2018-08-28 5:37 UTC (permalink / raw) To: Zihan Yang, Gerd Hoffmann, Michael S. Tsirkin; +Cc: qemu-devel, seabios Hi Gerd On 08/28/2018 07:12 AM, Zihan Yang wrote: > Gerd Hoffmann <kraxel@redhat.com> 于2018年8月27日周一 上午7:04写道: >> Hi, >> >>>> However, QEMU only binds port 0xcf8 and 0xcfc to >>>> bus pcie.0. To avoid bus confliction, we should use other port pairs for >>>> busses under new domains. >>> I would skip support for IO based configuration and use only MMCONFIG >>> for extra root buses. >>> >>> The question remains: how do we assign MMCONFIG space for >>> each PCI domain. Thanks for your comments! >> Allocation-wise it would be easiest to place them above 4G. Right after >> memory, or after etc/reserved-memory-end (if that fw_cfg file is >> present), where the 64bit pci bars would have been placed. Move the pci >> bars up in address space to make room. >> >> Only problem is that seabios wouldn't be able to access mmconfig then. >> >> Placing them below 4G would work at least for a few pci domains. q35 >> mmconfig bar is placed at 0xb0000000 -> 0xbfffffff, basically for >> historical reasons. Old qemu versions had 2.75G low memory on q35 (up >> to 0xafffffff), and I think old machine types still have that for live >> migration compatibility reasons. Modern qemu uses 2G only, to make >> gigabyte alignment work. >> >> 32bit pci bars are placed above 0xc0000000. The address space from 2G >> to 2.75G (0x8000000 -> 0xafffffff) is unused on new machine types. >> Enough room for three additional mmconfig bars (full size), so four >> pci domains total if you add the q35 one. > Maybe we can support 4 domains first before we come up > with a better solution. But I'm not sure if four domains are > enough for those who want too many devices? (Adding Michael) Since we will not use all 256 buses of an extra PCI domain, I think this space will allow us to support more PCI domains. How will the flow look like ? 1. QEMU passes to SeaBIOS information of how many extra PCI domains needs, and how many buses per domain. How it will pass this info? A vendor specific capability, some PCI registers or modifying extra-pci-roots fw_cfg file? 2. SeaBIOS assigns the address for each PCI Domain and returns the information to QEMU. How it will do that? Some pxb-pcie registers? Or do we model the MMCFG like a PCI BAR? 3. Once QEMU gets the MMCFG addresses, it can answer to mmio configuration cycles. 4. SeaBIOS queries all PCI domains devices, computes and assigns IO/MEM resources (for PCI domains > 0 it will use MMCFG to configure the PCI devices) 5. QEMU uses the IO/MEM information to create the CRS for each extra PCI host bridge. 6. SeaBIOS gets the ACPI tables from QEMU and passes them to the guest OS. Thanks, Marcel >> cheers, >> Gerd >> ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [SeaBIOS] [RFC v2 0/3] Support multiple pci domains in pci_device 2018-08-28 5:37 ` Marcel Apfelbaum @ 2018-08-28 6:07 ` Gerd Hoffmann 2018-08-28 6:53 ` Marcel Apfelbaum 0 siblings, 1 reply; 9+ messages in thread From: Gerd Hoffmann @ 2018-08-28 6:07 UTC (permalink / raw) To: Marcel Apfelbaum; +Cc: Zihan Yang, Michael S. Tsirkin, qemu-devel, seabios Hi, > Since we will not use all 256 buses of an extra PCI domain, > I think this space will allow us to support more PCI domains. Depends on the use case I guess. If you just need many pcie devices this probably doesn't help. If you want them for numa support then yes, more domains with less devices each can be useful then. > How will the flow look like ? > > 1. QEMU passes to SeaBIOS information of how many extra > PCI domains needs, and how many buses per domain. > How it will pass this info? A vendor specific capability, > some PCI registers or modifying extra-pci-roots fw_cfg file? Where is the pxb-pcie device? 0000:$somewhere? Or $domain:00:00.0? > 2. SeaBIOS assigns the address for each PCI Domain and > returns the information to QEMU. > How it will do that? Some pxb-pcie registers? Or do we model > the MMCFG like a PCI BAR? If we can access pxb-pcie registers before configuring MMCFG then yes, we should use pxb-pcie registers for that. cheers, Gerd ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [SeaBIOS] [RFC v2 0/3] Support multiple pci domains in pci_device 2018-08-28 6:07 ` Gerd Hoffmann @ 2018-08-28 6:53 ` Marcel Apfelbaum 2018-08-28 10:14 ` Gerd Hoffmann 0 siblings, 1 reply; 9+ messages in thread From: Marcel Apfelbaum @ 2018-08-28 6:53 UTC (permalink / raw) To: Gerd Hoffmann; +Cc: Zihan Yang, Michael S. Tsirkin, qemu-devel, seabios Hi Gerd, On 08/28/2018 09:07 AM, Gerd Hoffmann wrote: > Hi, > >> Since we will not use all 256 buses of an extra PCI domain, >> I think this space will allow us to support more PCI domains. > Depends on the use case I guess. If you just need many pcie devices > this probably doesn't help. If you want them for numa support then yes, > more domains with less devices each can be useful then. We already support multiple NUMA nodes. We want more devices. Still, having 4x number of devices we previously supported is a good step forward. > >> How will the flow look like ? >> >> 1. QEMU passes to SeaBIOS information of how many extra >> PCI domains needs, and how many buses per domain. >> How it will pass this info? A vendor specific capability, >> some PCI registers or modifying extra-pci-roots fw_cfg file? > Where is the pxb-pcie device? 0000:$somewhere? Or $domain:00:00.0? 0000:$somewhere (On PCI domain 0) >> 2. SeaBIOS assigns the address for each PCI Domain >> and >> returns the information to QEMU. >> How it will do that? Some pxb-pcie registers? Or do we model >> the MMCFG like a PCI BAR? > If we can access pxb-pcie registers before configuring MMCFG then yes, > we should use pxb-pcie registers for that. Yes, we can. Thanks Gerd! Marcel > > cheers, > Gerd > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [SeaBIOS] [RFC v2 0/3] Support multiple pci domains in pci_device 2018-08-28 6:53 ` Marcel Apfelbaum @ 2018-08-28 10:14 ` Gerd Hoffmann 2018-08-28 17:02 ` Kevin O'Connor 0 siblings, 1 reply; 9+ messages in thread From: Gerd Hoffmann @ 2018-08-28 10:14 UTC (permalink / raw) To: Marcel Apfelbaum; +Cc: Zihan Yang, Michael S. Tsirkin, qemu-devel, seabios Hi, > > Where is the pxb-pcie device? 0000:$somewhere? Or $domain:00:00.0? > > 0000:$somewhere (On PCI domain 0) Cool, so we don't have an chicken-and-egg issue. > > If we can access pxb-pcie registers before configuring MMCFG then yes, > > we should use pxb-pcie registers for that. > > Yes, we can. Ok, so we can configure mmcfg as hidden pci bar, simliar to the q35 mmcfg. Any configuration hints can be passed as pci vendor capability (simliar to the bridge window size hints), if needed. cheers, Gerd ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [SeaBIOS] [RFC v2 0/3] Support multiple pci domains in pci_device 2018-08-28 10:14 ` Gerd Hoffmann @ 2018-08-28 17:02 ` Kevin O'Connor 2018-08-28 17:17 ` Marcel Apfelbaum 0 siblings, 1 reply; 9+ messages in thread From: Kevin O'Connor @ 2018-08-28 17:02 UTC (permalink / raw) To: Gerd Hoffmann Cc: Marcel Apfelbaum, seabios, qemu-devel, Zihan Yang, Michael S. Tsirkin On Tue, Aug 28, 2018 at 12:14:58PM +0200, Gerd Hoffmann wrote: > Hi, > > > > Where is the pxb-pcie device? 0000:$somewhere? Or $domain:00:00.0? > > > > 0000:$somewhere (On PCI domain 0) > > Cool, so we don't have an chicken-and-egg issue. > > > > If we can access pxb-pcie registers before configuring MMCFG then yes, > > > we should use pxb-pcie registers for that. > > > > Yes, we can. > > Ok, so we can configure mmcfg as hidden pci bar, simliar to the q35 > mmcfg. Any configuration hints can be passed as pci vendor capability > (simliar to the bridge window size hints), if needed. Just so I understand, the proposal is to have SeaBIOS search for pxb-pcie devices on the main PCI bus and allocate address space for each. (These devices would not be considered pci buses in the traditional sense.) Then SeaBIOS will traverse that address space (MMCFG) and allocate BARs (both address space and io space) for the PCI devices found in that address space. Finally, QEMU will take all those allocations and use it when generating the ACPI tables. Did I get that right? -Kevin ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [SeaBIOS] [RFC v2 0/3] Support multiple pci domains in pci_device 2018-08-28 17:02 ` Kevin O'Connor @ 2018-08-28 17:17 ` Marcel Apfelbaum 2018-08-28 17:45 ` Kevin O'Connor 0 siblings, 1 reply; 9+ messages in thread From: Marcel Apfelbaum @ 2018-08-28 17:17 UTC (permalink / raw) To: Kevin O'Connor, Gerd Hoffmann Cc: seabios, qemu-devel, Zihan Yang, Michael S. Tsirkin Hi Kevin, On 08/28/2018 08:02 PM, Kevin O'Connor wrote: > On Tue, Aug 28, 2018 at 12:14:58PM +0200, Gerd Hoffmann wrote: >> Hi, >> >>>> Where is the pxb-pcie device? 0000:$somewhere? Or $domain:00:00.0? >>> 0000:$somewhere (On PCI domain 0) >> Cool, so we don't have an chicken-and-egg issue. >> >>>> If we can access pxb-pcie registers before configuring MMCFG then yes, >>>> we should use pxb-pcie registers for that. >>> Yes, we can. >> Ok, so we can configure mmcfg as hidden pci bar, simliar to the q35 >> mmcfg. Any configuration hints can be passed as pci vendor capability >> (simliar to the bridge window size hints), if needed. > Just so I understand, the proposal is to have SeaBIOS search for > pxb-pcie devices on the main PCI bus and allocate address space for > each. (These devices would not be considered pci buses in the > traditional sense.) Then SeaBIOS will traverse that address space > (MMCFG) and allocate BARs (both address space and io space) for the > PCI devices found in that address space. Finally, QEMU will take all > those allocations and use it when generating the ACPI tables. > > Did I get that right? Yes, the pxb-pcie exposes a new PCI root bus, but we want it in a different PCI domain. This is done in order to remove the 256 PCI Express devices limitation on a PCI Express machine. Does the plan sounds sane? Thanks, Marcel > > -Kevin ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [SeaBIOS] [RFC v2 0/3] Support multiple pci domains in pci_device 2018-08-28 17:17 ` Marcel Apfelbaum @ 2018-08-28 17:45 ` Kevin O'Connor 0 siblings, 0 replies; 9+ messages in thread From: Kevin O'Connor @ 2018-08-28 17:45 UTC (permalink / raw) To: Marcel Apfelbaum Cc: Gerd Hoffmann, seabios, qemu-devel, Zihan Yang, Michael S. Tsirkin On Tue, Aug 28, 2018 at 08:17:19PM +0300, Marcel Apfelbaum wrote: > On 08/28/2018 08:02 PM, Kevin O'Connor wrote: > > On Tue, Aug 28, 2018 at 12:14:58PM +0200, Gerd Hoffmann wrote: > > > > > Where is the pxb-pcie device? 0000:$somewhere? Or $domain:00:00.0? > > > > 0000:$somewhere (On PCI domain 0) > > > Cool, so we don't have an chicken-and-egg issue. > > > > > > > > If we can access pxb-pcie registers before configuring MMCFG then yes, > > > > > we should use pxb-pcie registers for that. > > > > Yes, we can. > > > Ok, so we can configure mmcfg as hidden pci bar, simliar to the q35 > > > mmcfg. Any configuration hints can be passed as pci vendor capability > > > (simliar to the bridge window size hints), if needed. > > Just so I understand, the proposal is to have SeaBIOS search for > > pxb-pcie devices on the main PCI bus and allocate address space for > > each. (These devices would not be considered pci buses in the > > traditional sense.) Then SeaBIOS will traverse that address space > > (MMCFG) and allocate BARs (both address space and io space) for the > > PCI devices found in that address space. Finally, QEMU will take all > > those allocations and use it when generating the ACPI tables. > > > > Did I get that right? > > Yes, the pxb-pcie exposes a new PCI root bus, but we want it > in a different PCI domain. This is done in order to remove the > 256 PCI Express devices limitation on a PCI Express machine. > > Does the plan sounds sane? It sounds okay to me. Separately, we could "dust off" the SeaBIOS PAE patches if we want to place the address space allocations above 4GB. -Kevin ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2018-08-28 17:45 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <1533793434-7614-1-git-send-email-whois.zihan.yang@gmail.com> [not found] ` <ece2a3ff-678d-00b6-3fd6-660972c5beca@gmail.com> [not found] ` <20180827070406.t525gr43qpu7wpsj@sirius.home.kraxel.org> 2018-08-28 4:12 ` [Qemu-devel] [SeaBIOS] [RFC v2 0/3] Support multiple pci domains in pci_device Zihan Yang 2018-08-28 5:30 ` Gerd Hoffmann 2018-08-28 5:37 ` Marcel Apfelbaum 2018-08-28 6:07 ` Gerd Hoffmann 2018-08-28 6:53 ` Marcel Apfelbaum 2018-08-28 10:14 ` Gerd Hoffmann 2018-08-28 17:02 ` Kevin O'Connor 2018-08-28 17:17 ` Marcel Apfelbaum 2018-08-28 17:45 ` Kevin O'Connor
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.