* [Qemu-devel] [PATCH 0/6] Xen stubdom support @ 2019-03-11 18:02 Jason Andryuk 2019-03-11 18:02 ` [Qemu-devel] [PATCH 1/6] xen: Introduce -xen-stubdom option Jason Andryuk ` (5 more replies) 0 siblings, 6 replies; 14+ messages in thread From: Jason Andryuk @ 2019-03-11 18:02 UTC (permalink / raw) To: qemu-devel Cc: xen-devel, marmarek, Jason Andryuk, Stefano Stabellini, Anthony Perard, Paul Durrant, Michael S. Tsirkin, Marcel Apfelbaum, Paolo Bonzini, Richard Henderson, Eduardo Habkost Xen supports running QEMU in a dedicated service vm - a stub domain or stubdom. QEMU is then isolated outside of the privileged Domain-0. When running in a stubdom, there are a few changes needed for QEMU. On older Xen versions, the default ioreq server needs to have the stubdom's domid specified. The stubdom doesn't run PV backends, so that initialization code can be skipped. Stubdom's don't support MSI-X, so that PCI capability must be hidden from passed through devices. Stubdom mode is enabled by the new -xen-stubdom flag. Jason Andryuk (5): xen: Introduce -xen-stubdom option xen: Move xenstore initialization to common location xen: Skip backend initialization for stubdom xen: Set HVM_PARAM_DM_DOMAIN for stubdom on older Xen xen-pt: Hide MSI-X from xen stubdoms Simon Gaiser (1): xen-pt: Round pci regions sizes to XEN_PAGE_SIZE hw/i386/xen/xen-hvm.c | 22 ++++++++++++++++------ hw/xen/xen-legacy-backend.c | 8 -------- hw/xen/xen_pt.c | 10 +++++++--- hw/xen/xen_pt_config_init.c | 3 +++ include/hw/xen/xen.h | 6 ++++++ include/hw/xen/xen_common.h | 5 +++++ qemu-options.hx | 7 +++++++ vl.c | 8 ++++++++ 8 files changed, 52 insertions(+), 17 deletions(-) -- 2.20.1 ^ permalink raw reply [flat|nested] 14+ messages in thread
* [Qemu-devel] [PATCH 1/6] xen: Introduce -xen-stubdom option 2019-03-11 18:02 [Qemu-devel] [PATCH 0/6] Xen stubdom support Jason Andryuk @ 2019-03-11 18:02 ` Jason Andryuk 2019-03-11 18:06 ` Paolo Bonzini 2019-03-11 18:02 ` [Qemu-devel] [PATCH 2/6] xen: Move xenstore initialization to common location Jason Andryuk ` (4 subsequent siblings) 5 siblings, 1 reply; 14+ messages in thread From: Jason Andryuk @ 2019-03-11 18:02 UTC (permalink / raw) To: qemu-devel Cc: xen-devel, marmarek, Jason Andryuk, Stefano Stabellini, Anthony Perard, Paul Durrant, Paolo Bonzini With Xen, QEMU can run isolated in a dedicated service VM - a stubdom. There are a few differences when running in a stubdom compared to dom0. Add the -xen-stubdom option to select this mode at runtime. The default is off. Signed-off-by: Jason Andryuk <jandryuk@gmail.com> --- include/hw/xen/xen.h | 6 ++++++ qemu-options.hx | 7 +++++++ vl.c | 8 ++++++++ 3 files changed, 21 insertions(+) diff --git a/include/hw/xen/xen.h b/include/hw/xen/xen.h index ba039c146d..fed3611623 100644 --- a/include/hw/xen/xen.h +++ b/include/hw/xen/xen.h @@ -21,6 +21,7 @@ enum xen_mode { extern uint32_t xen_domid; extern enum xen_mode xen_mode; extern bool xen_domid_restrict; +extern bool xen_stubdom; extern bool xen_allowed; @@ -29,6 +30,11 @@ static inline bool xen_enabled(void) return xen_allowed; } +static inline bool xen_stubdom_enabled(void) +{ + return xen_stubdom; +} + int xen_pci_slot_get_pirq(PCIDevice *pci_dev, int irq_num); void xen_piix3_set_irq(void *opaque, int irq_num, int level); void xen_piix_pci_write_config_client(uint32_t address, uint32_t val, int len); diff --git a/qemu-options.hx b/qemu-options.hx index 1cf9aac1fe..ba56c3dd9a 100644 --- a/qemu-options.hx +++ b/qemu-options.hx @@ -3386,6 +3386,10 @@ DEF("xen-domid-restrict", 0, QEMU_OPTION_xen_domid_restrict, " to specified domain id. (Does not affect\n" " xenpv machine type).\n", QEMU_ARCH_ALL) +DEF("xen-stubdom", 0, QEMU_OPTION_xen_stubdom, + "-xen-stubdom specify QEMU is running in a stubdom, so certain\n" + " behavior changes. (Does not affect xenpv machine type).\n", + QEMU_ARCH_ALL) STEXI @item -xen-domid @var{id} @findex -xen-domid @@ -3396,6 +3400,9 @@ Attach to existing xen domain. libxl will use this when starting QEMU (XEN only). @findex -xen-domid-restrict Restrict set of available xen operations to specified domain id (XEN only). +@findex -xen-stubdom +@item -xen-stubdom +Run qemu in stubdom-mode (XEN only). ETEXI DEF("no-reboot", 0, QEMU_OPTION_no_reboot, \ diff --git a/vl.c b/vl.c index 4a350de5cd..0d04319d9b 100644 --- a/vl.c +++ b/vl.c @@ -206,6 +206,7 @@ bool xen_allowed; uint32_t xen_domid; enum xen_mode xen_mode = XEN_EMULATE; bool xen_domid_restrict; +bool xen_stubdom; static int has_defaults = 1; static int default_serial = 1; @@ -3796,6 +3797,13 @@ int main(int argc, char **argv, char **envp) } xen_domid_restrict = true; break; + case QEMU_OPTION_xen_stubdom: + if (!(xen_available())) { + error_report("Option not supported for this target"); + exit(1); + } + xen_stubdom = true; + break; case QEMU_OPTION_trace: g_free(trace_file); trace_file = trace_opt_parse(optarg); -- 2.20.1 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [PATCH 1/6] xen: Introduce -xen-stubdom option 2019-03-11 18:02 ` [Qemu-devel] [PATCH 1/6] xen: Introduce -xen-stubdom option Jason Andryuk @ 2019-03-11 18:06 ` Paolo Bonzini 2019-03-11 19:46 ` Jason Andryuk 0 siblings, 1 reply; 14+ messages in thread From: Paolo Bonzini @ 2019-03-11 18:06 UTC (permalink / raw) To: Jason Andryuk, qemu-devel Cc: xen-devel, marmarek, Stefano Stabellini, Anthony Perard, Paul Durrant On 11/03/19 19:02, Jason Andryuk wrote: > With Xen, QEMU can run isolated in a dedicated service VM - a stubdom. > There are a few differences when running in a stubdom compared to dom0. > Add the -xen-stubdom option to select this mode at runtime. The default > is off. This should be "-accel xen,stubdom=on". You should find examples for tcg that explain how to add a suboption to -accel. Paolo > Signed-off-by: Jason Andryuk <jandryuk@gmail.com> > --- > include/hw/xen/xen.h | 6 ++++++ > qemu-options.hx | 7 +++++++ > vl.c | 8 ++++++++ > 3 files changed, 21 insertions(+) > > diff --git a/include/hw/xen/xen.h b/include/hw/xen/xen.h > index ba039c146d..fed3611623 100644 > --- a/include/hw/xen/xen.h > +++ b/include/hw/xen/xen.h > @@ -21,6 +21,7 @@ enum xen_mode { > extern uint32_t xen_domid; > extern enum xen_mode xen_mode; > extern bool xen_domid_restrict; > +extern bool xen_stubdom; > > extern bool xen_allowed; > > @@ -29,6 +30,11 @@ static inline bool xen_enabled(void) > return xen_allowed; > } > > +static inline bool xen_stubdom_enabled(void) > +{ > + return xen_stubdom; > +} > + > int xen_pci_slot_get_pirq(PCIDevice *pci_dev, int irq_num); > void xen_piix3_set_irq(void *opaque, int irq_num, int level); > void xen_piix_pci_write_config_client(uint32_t address, uint32_t val, int len); > diff --git a/qemu-options.hx b/qemu-options.hx > index 1cf9aac1fe..ba56c3dd9a 100644 > --- a/qemu-options.hx > +++ b/qemu-options.hx > @@ -3386,6 +3386,10 @@ DEF("xen-domid-restrict", 0, QEMU_OPTION_xen_domid_restrict, > " to specified domain id. (Does not affect\n" > " xenpv machine type).\n", > QEMU_ARCH_ALL) > +DEF("xen-stubdom", 0, QEMU_OPTION_xen_stubdom, > + "-xen-stubdom specify QEMU is running in a stubdom, so certain\n" > + " behavior changes. (Does not affect xenpv machine type).\n", > + QEMU_ARCH_ALL) > STEXI > @item -xen-domid @var{id} > @findex -xen-domid > @@ -3396,6 +3400,9 @@ Attach to existing xen domain. > libxl will use this when starting QEMU (XEN only). > @findex -xen-domid-restrict > Restrict set of available xen operations to specified domain id (XEN only). > +@findex -xen-stubdom > +@item -xen-stubdom > +Run qemu in stubdom-mode (XEN only). > ETEXI > > DEF("no-reboot", 0, QEMU_OPTION_no_reboot, \ > diff --git a/vl.c b/vl.c > index 4a350de5cd..0d04319d9b 100644 > --- a/vl.c > +++ b/vl.c > @@ -206,6 +206,7 @@ bool xen_allowed; > uint32_t xen_domid; > enum xen_mode xen_mode = XEN_EMULATE; > bool xen_domid_restrict; > +bool xen_stubdom; > > static int has_defaults = 1; > static int default_serial = 1; > @@ -3796,6 +3797,13 @@ int main(int argc, char **argv, char **envp) > } > xen_domid_restrict = true; > break; > + case QEMU_OPTION_xen_stubdom: > + if (!(xen_available())) { > + error_report("Option not supported for this target"); > + exit(1); > + } > + xen_stubdom = true; > + break; > case QEMU_OPTION_trace: > g_free(trace_file); > trace_file = trace_opt_parse(optarg); > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [PATCH 1/6] xen: Introduce -xen-stubdom option 2019-03-11 18:06 ` Paolo Bonzini @ 2019-03-11 19:46 ` Jason Andryuk 0 siblings, 0 replies; 14+ messages in thread From: Jason Andryuk @ 2019-03-11 19:46 UTC (permalink / raw) To: Paolo Bonzini Cc: qemu-devel, xen-devel, Marek Marczykowski-Górecki, Stefano Stabellini, Anthony Perard, Paul Durrant On Mon, Mar 11, 2019 at 2:06 PM Paolo Bonzini <pbonzini@redhat.com> wrote: > > On 11/03/19 19:02, Jason Andryuk wrote: > > With Xen, QEMU can run isolated in a dedicated service VM - a stubdom. > > There are a few differences when running in a stubdom compared to dom0. > > Add the -xen-stubdom option to select this mode at runtime. The default > > is off. > > This should be "-accel xen,stubdom=on". You should find examples for > tcg that explain how to add a suboption to -accel. Thanks, Paolo. I'll re-work the option as you suggest. Regards, Jason ^ permalink raw reply [flat|nested] 14+ messages in thread
* [Qemu-devel] [PATCH 2/6] xen: Move xenstore initialization to common location 2019-03-11 18:02 [Qemu-devel] [PATCH 0/6] Xen stubdom support Jason Andryuk 2019-03-11 18:02 ` [Qemu-devel] [PATCH 1/6] xen: Introduce -xen-stubdom option Jason Andryuk @ 2019-03-11 18:02 ` Jason Andryuk 2019-03-11 18:02 ` [Qemu-devel] [PATCH 3/6] xen: Skip backend initialization for stubdom Jason Andryuk ` (3 subsequent siblings) 5 siblings, 0 replies; 14+ messages in thread From: Jason Andryuk @ 2019-03-11 18:02 UTC (permalink / raw) To: qemu-devel Cc: xen-devel, marmarek, Jason Andryuk, Stefano Stabellini, Anthony Perard, Paul Durrant, Paolo Bonzini, Richard Henderson, Eduardo Habkost, Michael S. Tsirkin, Marcel Apfelbaum For the xen stubdom case, we'll want xenstore initialized, but we'll want to skip the rest of xen_be_init. Move the initialization to xen_hvm_init so we can conditionalize calling xen_be_init. xs_domain_open() is deprecated for xs_open(0), so make the replacement as well. Signed-off-by: Jason Andryuk <jandryuk@gmail.com> --- hw/i386/xen/xen-hvm.c | 8 ++++++++ hw/xen/xen-legacy-backend.c | 8 -------- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/hw/i386/xen/xen-hvm.c b/hw/i386/xen/xen-hvm.c index 2939122e7c..c20c4b27f6 100644 --- a/hw/i386/xen/xen-hvm.c +++ b/hw/i386/xen/xen-hvm.c @@ -1487,6 +1487,14 @@ void xen_hvm_init(PCMachineState *pcms, MemoryRegion **ram_memory) xen_bus_init(); + xenstore = xs_open(0); + if (!xenstore) { + error_report("Can't connect to xenstored"); + goto err; + } + + qemu_set_fd_handler(xs_fileno(xenstore), xenstore_update, NULL, NULL); + /* Initialize backend core & drivers */ if (xen_be_init() != 0) { error_report("xen backend core setup failed"); diff --git a/hw/xen/xen-legacy-backend.c b/hw/xen/xen-legacy-backend.c index 36fd1e9b09..bdf2fa917f 100644 --- a/hw/xen/xen-legacy-backend.c +++ b/hw/xen/xen-legacy-backend.c @@ -683,14 +683,6 @@ int xen_be_init(void) { xengnttab_handle *gnttabdev; - xenstore = xs_daemon_open(); - if (!xenstore) { - xen_pv_printf(NULL, 0, "can't connect to xenstored\n"); - return -1; - } - - qemu_set_fd_handler(xs_fileno(xenstore), xenstore_update, NULL, NULL); - if (xen_xc == NULL || xen_fmem == NULL) { /* Check if xen_init() have been called */ goto err; -- 2.20.1 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* [Qemu-devel] [PATCH 3/6] xen: Skip backend initialization for stubdom 2019-03-11 18:02 [Qemu-devel] [PATCH 0/6] Xen stubdom support Jason Andryuk 2019-03-11 18:02 ` [Qemu-devel] [PATCH 1/6] xen: Introduce -xen-stubdom option Jason Andryuk 2019-03-11 18:02 ` [Qemu-devel] [PATCH 2/6] xen: Move xenstore initialization to common location Jason Andryuk @ 2019-03-11 18:02 ` Jason Andryuk 2019-03-11 18:02 ` [Qemu-devel] [PATCH 4/6] xen: Set HVM_PARAM_DM_DOMAIN for stubdom on older Xen Jason Andryuk ` (2 subsequent siblings) 5 siblings, 0 replies; 14+ messages in thread From: Jason Andryuk @ 2019-03-11 18:02 UTC (permalink / raw) To: qemu-devel Cc: xen-devel, marmarek, Jason Andryuk, Stefano Stabellini, Anthony Perard, Paul Durrant, Paolo Bonzini, Richard Henderson, Eduardo Habkost, Michael S. Tsirkin, Marcel Apfelbaum When QEMU is running in a stubdom, it does not provide any Paravirtualized backends. Those still run in dom0 or another driver domain. Therefore we skip backend initialization (xen_bus_init and xen_be_init) for the stubdom case. Original patch by Anthony PERARD <anthony.perard@citrix.com> Signed-off-by: Jason Andryuk <jandryuk@gmail.com> --- hw/i386/xen/xen-hvm.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/hw/i386/xen/xen-hvm.c b/hw/i386/xen/xen-hvm.c index c20c4b27f6..4b62f070cb 100644 --- a/hw/i386/xen/xen-hvm.c +++ b/hw/i386/xen/xen-hvm.c @@ -1485,8 +1485,6 @@ void xen_hvm_init(PCMachineState *pcms, MemoryRegion **ram_memory) QLIST_INIT(&state->dev_list); device_listener_register(&state->device_listener); - xen_bus_init(); - xenstore = xs_open(0); if (!xenstore) { error_report("Can't connect to xenstored"); @@ -1495,12 +1493,16 @@ void xen_hvm_init(PCMachineState *pcms, MemoryRegion **ram_memory) qemu_set_fd_handler(xs_fileno(xenstore), xenstore_update, NULL, NULL); - /* Initialize backend core & drivers */ - if (xen_be_init() != 0) { - error_report("xen backend core setup failed"); - goto err; + if (!xen_stubdom_enabled()) { + xen_bus_init(); + + /* Initialize backend core & drivers */ + if (xen_be_init() != 0) { + error_report("xen backend core setup failed"); + goto err; + } + xen_be_register_common(); } - xen_be_register_common(); QLIST_INIT(&xen_physmap); xen_read_physmap(state); -- 2.20.1 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* [Qemu-devel] [PATCH 4/6] xen: Set HVM_PARAM_DM_DOMAIN for stubdom on older Xen 2019-03-11 18:02 [Qemu-devel] [PATCH 0/6] Xen stubdom support Jason Andryuk ` (2 preceding siblings ...) 2019-03-11 18:02 ` [Qemu-devel] [PATCH 3/6] xen: Skip backend initialization for stubdom Jason Andryuk @ 2019-03-11 18:02 ` Jason Andryuk 2019-03-11 18:02 ` [Qemu-devel] [PATCH 5/6] xen-pt: Hide MSI-X from xen stubdoms Jason Andryuk 2019-03-11 18:02 ` [Qemu-devel] [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE Jason Andryuk 5 siblings, 0 replies; 14+ messages in thread From: Jason Andryuk @ 2019-03-11 18:02 UTC (permalink / raw) To: qemu-devel Cc: xen-devel, marmarek, Jason Andryuk, Stefano Stabellini, Anthony Perard, Paul Durrant When running in a stubdom, we have to inform the hypervisor that the stubdom and not dom0 is handling the device model. Explicitly created ioreq servers are fine, but a call to HVM_PARAM_DM_DOMAIN is needed for the default ioreq server. Xen 4.12 removes the default ioreq server. With that, Xen started returning an error when setting HVM_PARAM_DM_DOMAIN. Put the HVM_PARAM_DM_DOMAIN call in the version compatibility header. When we fallback to the default ioreq server, issue the call and don't bother to check the return value. Original patch by Anthony PERARD <anthony.perard@citrix.com> Signed-off-by: Jason Andryuk <jandryuk@gmail.com> --- include/hw/xen/xen_common.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/include/hw/xen/xen_common.h b/include/hw/xen/xen_common.h index 9a8155e172..f59f841a43 100644 --- a/include/hw/xen/xen_common.h +++ b/include/hw/xen/xen_common.h @@ -616,6 +616,11 @@ static inline void xen_create_ioreq_server(domid_t dom, *ioservid = 0; use_default_ioreq_server = true; + + if (xen_stubdom_enabled()) { + xc_hvm_param_set(xen_xc, xen_domid, HVM_PARAM_DM_DOMAIN, DOMID_SELF); + } + trace_xen_default_ioreq_server(); } -- 2.20.1 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* [Qemu-devel] [PATCH 5/6] xen-pt: Hide MSI-X from xen stubdoms 2019-03-11 18:02 [Qemu-devel] [PATCH 0/6] Xen stubdom support Jason Andryuk ` (3 preceding siblings ...) 2019-03-11 18:02 ` [Qemu-devel] [PATCH 4/6] xen: Set HVM_PARAM_DM_DOMAIN for stubdom on older Xen Jason Andryuk @ 2019-03-11 18:02 ` Jason Andryuk 2019-03-11 18:02 ` [Qemu-devel] [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE Jason Andryuk 5 siblings, 0 replies; 14+ messages in thread From: Jason Andryuk @ 2019-03-11 18:02 UTC (permalink / raw) To: qemu-devel Cc: xen-devel, marmarek, Jason Andryuk, James McKenzie, Stefano Stabellini, Anthony Perard, Paul Durrant MSI-X is not supported in Xen stubdoms, so it must be disabled. Use the existing xen_pt_hide_dev_cap to hide when running under -xen-stubdom. A compile-time patch was originally written by James McKenzie <james.mckenzie@bromium.com> Signed-off-by: Jason Andryuk <jandryuk@gmail.com> --- hw/xen/xen_pt_config_init.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/hw/xen/xen_pt_config_init.c b/hw/xen/xen_pt_config_init.c index 31ec5add1d..b827a493ea 100644 --- a/hw/xen/xen_pt_config_init.c +++ b/hw/xen/xen_pt_config_init.c @@ -54,6 +54,9 @@ static int xen_pt_hide_dev_cap(const XenHostPCIDevice *d, uint8_t grp_id) return 1; } break; + case PCI_CAP_ID_MSIX: + /* stubdoms don't support MSI-X so skip it. */ + return xen_stubdom_enabled(); } return 0; } -- 2.20.1 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* [Qemu-devel] [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE 2019-03-11 18:02 [Qemu-devel] [PATCH 0/6] Xen stubdom support Jason Andryuk ` (4 preceding siblings ...) 2019-03-11 18:02 ` [Qemu-devel] [PATCH 5/6] xen-pt: Hide MSI-X from xen stubdoms Jason Andryuk @ 2019-03-11 18:02 ` Jason Andryuk [not found] ` <ee0da83d3f054e72ae450437c8834d04@AMSPEX02CL02.citrite.net> 5 siblings, 1 reply; 14+ messages in thread From: Jason Andryuk @ 2019-03-11 18:02 UTC (permalink / raw) To: qemu-devel Cc: xen-devel, marmarek, Simon Gaiser, Jason Andryuk, Stefano Stabellini, Anthony Perard, Paul Durrant From: Simon Gaiser <simon@invisiblethingslab.com> If a pci memory region has a size < XEN_PAGE_SIZE it can get located at an address which is not page aligned. This breaks the memory mapping via xc_domain_memory_mapping since this function is page based and the "offset" is therefore lost. Without this patch you will see error like this in the stubdom log: [00:05.0] xen_pt_bar_read: Error: Should not read BAR through QEMU. @0x0000000000000004 QubesOS/qubes-issues#2849 Signed-off-by: Simon Gaiser <simon@invisiblethingslab.com> Signed-off-by: Jason Andryuk <jandryuk@gmail.com> --- hw/xen/xen_pt.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c index 5539d56c3a..7f680442ee 100644 --- a/hw/xen/xen_pt.c +++ b/hw/xen/xen_pt.c @@ -449,9 +449,10 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s, uint16_t *cmd) /* Register PIO/MMIO BARs */ for (i = 0; i < PCI_ROM_SLOT; i++) { XenHostPCIIORegion *r = &d->io_regions[i]; + pcibus_t r_size = r->size; uint8_t type; - if (r->base_addr == 0 || r->size == 0) { + if (r->base_addr == 0 || r_size == 0) { continue; } @@ -469,15 +470,18 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s, uint16_t *cmd) type |= PCI_BASE_ADDRESS_MEM_TYPE_64; } *cmd |= PCI_COMMAND_MEMORY; + + /* Round up to a full page for the hypercall. */ + r_size = (r_size + XC_PAGE_SIZE - 1) & XC_PAGE_MASK; } memory_region_init_io(&s->bar[i], OBJECT(s), &ops, &s->dev, - "xen-pci-pt-bar", r->size); + "xen-pci-pt-bar", r_size); pci_register_bar(&s->dev, i, type, &s->bar[i]); XEN_PT_LOG(&s->dev, "IO region %i registered (size=0x%08"PRIx64 " base_addr=0x%08"PRIx64" type: %#x)\n", - i, r->size, r->base_addr, type); + i, r_size, r->base_addr, type); } /* Register expansion ROM address */ -- 2.20.1 ^ permalink raw reply related [flat|nested] 14+ messages in thread
[parent not found: <ee0da83d3f054e72ae450437c8834d04@AMSPEX02CL02.citrite.net>]
[parent not found: <CAKf6xpujVs3RPJcb+2vqPZFcYwhdDcjbt=S_+awjPuPFpFHsPw@mail.gmail.com>]
[parent not found: <57dc1083d20a469785f05a2e5250a820@AMSPEX02CL02.citrite.net>]
[parent not found: <cd5f1ecc-576f-b7d0-3090-4f3e4faf6148@citrix.com>]
[parent not found: <CAKf6xpt4XYXn2xHJoVY_ibcaHSw-ED10V7ZGNKuDdkiJ93RS0A@mail.gmail.com>]
[parent not found: <20190322030936.fkiajz5ifgaejkd4@MacBook-Air-de-Roger.local>]
[parent not found: <CAKf6xptP_b-+FuscjsTK9G7pMeVS8drvA_t+xb5bdF2zxxmWfA@mail.gmail.com>]
* Re: [Xen-devel] [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE [not found] ` <CAKf6xptP_b-+FuscjsTK9G7pMeVS8drvA_t+xb5bdF2zxxmWfA@mail.gmail.com> @ 2020-01-13 19:01 ` Jason Andryuk 2020-01-14 10:04 ` Roger Pau Monné 0 siblings, 1 reply; 14+ messages in thread From: Jason Andryuk @ 2020-01-13 19:01 UTC (permalink / raw) To: Roger Pau Monné Cc: Stefano Stabellini, Andrew Cooper, marmarek, qemu-devel, Simon Gaiser, Paul Durrant, Anthony Perard, xen-devel On Fri, Mar 22, 2019 at 3:43 PM Jason Andryuk <jandryuk@gmail.com> wrote: > > On Thu, Mar 21, 2019 at 11:09 PM Roger Pau Monné <roger.pau@citrix.com> wrote: > > > > On Wed, Mar 20, 2019 at 01:28:47PM -0400, Jason Andryuk wrote: > > > On Fri, Mar 15, 2019 at 12:28 PM Andrew Cooper > > > <andrew.cooper3@citrix.com> wrote: > > > > > > > > On 15/03/2019 09:17, Paul Durrant wrote: > > > > >> -----Original Message----- > > > > >> From: Jason Andryuk [mailto:jandryuk@gmail.com] > > > > >> Sent: 14 March 2019 18:16 > > > > >> To: Paul Durrant <Paul.Durrant@citrix.com> > > > > >> Cc: qemu-devel@nongnu.org; xen-devel@lists.xenproject.org; marmarek@invisiblethingslab.com; Simon > > > > >> Gaiser <simon@invisiblethingslab.com>; Stefano Stabellini <sstabellini@kernel.org>; Anthony Perard > > > > >> <anthony.perard@citrix.com> > > > > >> Subject: Re: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE > > > > >> > > > > >> On Wed, Mar 13, 2019 at 11:09 AM Paul Durrant <Paul.Durrant@citrix.com> wrote: > > > > >>>> -----Original Message----- > > > > >>>> From: Jason Andryuk [mailto:jandryuk@gmail.com] > > > > >>>> Sent: 11 March 2019 18:02 > > > > >>>> To: qemu-devel@nongnu.org > > > > >>>> Cc: xen-devel@lists.xenproject.org; marmarek@invisiblethingslab.com; Simon Gaiser > > > > >>>> <simon@invisiblethingslab.com>; Jason Andryuk <jandryuk@gmail.com>; Stefano Stabellini > > > > >>>> <sstabellini@kernel.org>; Anthony Perard <anthony.perard@citrix.com>; Paul Durrant > > > > >>>> <Paul.Durrant@citrix.com> > > > > >>>> Subject: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE > > > > >>>> > > > > >>>> From: Simon Gaiser <simon@invisiblethingslab.com> > > > > >>>> > > > > >>>> If a pci memory region has a size < XEN_PAGE_SIZE it can get located at > > > > >>>> an address which is not page aligned. > > > > >>> IIRC the PCI spec says that the minimum memory region size should be at least 4k. Should we even be > > > > >> tolerating BARs smaller than that? > > > > >>> Paul > > > > >>> > > > > >> Hi, Paul. > > > > >> > > > > >> Simon found this, so it affects a real device. Simon, do you recall > > > > >> which device was affected? > > > > >> > > > > >> I think BARs only need to be power-of-two size and aligned, and 4k is > > > > >> not a minimum. 16bytes may be a minimum, but I don't know what the > > > > >> spec says. > > > > >> > > > > >> On an Ivy Bridge system, here are some of the devices with BARs smaller than 4K: > > > > >> 00:16.0 Communication controller: Intel Corporation 7 Series/C210 > > > > >> Series Chipset Family MEI Controller #1 (rev 04) > > > > >> Memory at d0735000 (64-bit, non-prefetchable) [disabled] [size=16] > > > > >> 00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset > > > > >> Family USB Enhanced Host Controller #1 (rev 04) (prog-if 20 [EHCI]) > > > > >> Memory at d0739000 (32-bit, non-prefetchable) [disabled] [size=1K] > > > > >> 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family > > > > >> SMBus Controller (rev 04) > > > > >> Memory at d0734000 (64-bit, non-prefetchable) [disabled] [size=256] > > > > >> 02:00.0 System peripheral: JMicron Technology Corp. SD/MMC Host > > > > >> Controller (rev 30) > > > > >> Memory at d0503000 (32-bit, non-prefetchable) [disabled] [size=256] > > > > >> > > > > >> These examples are all 4K aligned, so this is not an issue on this machine. > > > > >> > > > > >> Reviewing the code, I'm now wondering if the following in > > > > >> hw/xen/xen_pt.c:xen_pt_region_update is wrong: rc = > > > > >> xc_domain_memory_mapping(xen_xc, xen_domid, > > > > >> XEN_PFN(guest_addr + XC_PAGE_SIZE - 1), > > > > >> XEN_PFN(machine_addr + XC_PAGE_SIZE - 1), > > > > >> XEN_PFN(size + XC_PAGE_SIZE - 1), > > > > >> op); > > > > >> > > > > >> If a bar of size 0x100 is at 0xd0500800, then the machine_addr passed > > > > >> in would be 0xd0501000 which is past the actual location. Should the > > > > >> call arguments just be XEN_PFN(guest_addr) & XEN_PFN(machine_addr)? > > > > >> > > > > >> BARs smaller than a page would also be a problem if BARs for different > > > > >> devices shared the same page. > > > > > Exactly. We cannot pass them through with any degree of safety (not that passthrough of an arbitrary device is a particularly safe thing to do anyway). The xen-pt code would instead need to trap those BARs and perform the accesses to the real BAR itself. Ultimately though I think we should be retiring the xen-pt code in favour of a standalone emulator. > > > > > > > > It doesn't matter if the BAR is smaller than 4k, if there are holes next > > > > to it. > > > > > > > > Do we know what the case is in practice for these USB controllers? > > > > > > > > If the worst comes to the worst, we can re-enumerate the PCI bus to > > > > ensure that all bars smaller than 4k still have 4k alignment between > > > > them. That way we can safely pass them through even when they are smaller. > > > > > > Andrew, thanks for checking the spec on the minimum BAR size. > > > > > > Dropping the Round PCI region patch from QMEU, the guest HVM will have: > > > > > > 00:06.0 SD Host controller: Ricoh Co Ltd PCIe SDXC/MMC Host Controller (rev 07) > > > Memory at f2028800 (32-bit, non-prefetchable) [size=256] > > > 00:07.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host > > > Controller (rev 04) (prog-if 30 [XHCI]) > > > Memory at f2024000 (64-bit, non-prefetchable) [size=8K] > > > 00:08.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset > > > Family USB Enhanced Host Controller #2 (rev 05) (prog-if 20 [EHCI]) > > > Memory at f2028000 (32-bit, non-prefetchable) [size=1K] > > > 00:09.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset > > > Family USB Enhanced Host Controller #1 (rev 05) (prog-if 20 [EHCI]) > > > Memory at f2028400 (32-bit, non-prefetchable) [size=1K] > > > > > > 00:09.0, 00:08.0 & 00:06.0 all share the same page. Only 00:08.0 is > > > working. With some added debugging output, you'll see that the same > > > page* is used for three of the BARs. > > > > > > [00:06.0] mapping guest_addr 0xf2028800 gfn 0xf2028 to maddr > > > 0xe1a30000 mfn 0xe1a30 > > > [00:07.0] mapping guest_addr 0xf2024000 gfn 0xf2024 to maddr > > > 0xe0800000 mfn 0xe0800 > > > [00:09.0] mapping guest_addr 0xf2028400 gfn 0xf2028 to maddr > > > 0xe1900000 mfn 0xe1900 > > > [00:08.0] mapping guest_addr 0xf2028000 gfn 0xf2028 to maddr > > > 0xe1a2f000 mfn 0xe1a2f > > > > The patch below should prevent hvmloader from placing multiple BARs on > > the same page, could you give it a try? > > > > Note that this is not going to prevent the guest from moving those > > BARs around and place them in the same page, thus breaking the initial > > placement done by hvmloader. > > > > Thanks, Roger. > > Hi, Roger. > > I've minimally tested this. Yes, this patch seems to place small BARs > into separate pages. The linux stubdom and QEMU then use the spacing > as provided by hvmloader. Roger, Would you mind submitting this patch to Xen? Thanks, Jason > > > > ---8<--- > > diff --git a/tools/firmware/hvmloader/pci.c b/tools/firmware/hvmloader/pci.c > > index 0b708bf578..c433b34cd6 100644 > > --- a/tools/firmware/hvmloader/pci.c > > +++ b/tools/firmware/hvmloader/pci.c > > @@ -489,6 +489,10 @@ void pci_setup(void) > > > > resource->base = base; > > > > + if ( (bar_data & PCI_BASE_ADDRESS_SPACE) == > > + PCI_BASE_ADDRESS_SPACE_MEMORY ) > > + resource->base = ROUNDUP(resource->base, PAGE_SIZE); > > + > > pci_writel(devfn, bar_reg, bar_data); > > if (using_64bar) > > pci_writel(devfn, bar_reg + 4, bar_data_upper); > > diff --git a/tools/firmware/hvmloader/util.h b/tools/firmware/hvmloader/util.h > > index 7bca6418d2..b5554b5844 100644 > > --- a/tools/firmware/hvmloader/util.h > > +++ b/tools/firmware/hvmloader/util.h > > @@ -51,6 +51,8 @@ void __bug(char *file, int line) __attribute__((noreturn)); > > #define MB(mb) (mb##ULL << 20) > > #define GB(gb) (gb##ULL << 30) > > > > +#define ROUNDUP(x, a) (((x) + (a) - 1) & ~((a) - 1)) > > + > > static inline int test_bit(unsigned int b, const void *p) > > { > > return !!(((const uint8_t *)p)[b>>3] & (1u<<(b&7))); > > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Xen-devel] [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE 2020-01-13 19:01 ` [Xen-devel] " Jason Andryuk @ 2020-01-14 10:04 ` Roger Pau Monné 2020-01-14 14:41 ` Jason Andryuk 0 siblings, 1 reply; 14+ messages in thread From: Roger Pau Monné @ 2020-01-14 10:04 UTC (permalink / raw) To: Jason Andryuk Cc: Stefano Stabellini, Andrew Cooper, marmarek, qemu-devel, Simon Gaiser, Paul Durrant, Anthony Perard, xen-devel On Mon, Jan 13, 2020 at 02:01:47PM -0500, Jason Andryuk wrote: > On Fri, Mar 22, 2019 at 3:43 PM Jason Andryuk <jandryuk@gmail.com> wrote: > > > > On Thu, Mar 21, 2019 at 11:09 PM Roger Pau Monné <roger.pau@citrix.com> wrote: > > > > > > On Wed, Mar 20, 2019 at 01:28:47PM -0400, Jason Andryuk wrote: > > > > On Fri, Mar 15, 2019 at 12:28 PM Andrew Cooper > > > > <andrew.cooper3@citrix.com> wrote: > > > > > > > > > > On 15/03/2019 09:17, Paul Durrant wrote: > > > > > >> -----Original Message----- > > > > > >> From: Jason Andryuk [mailto:jandryuk@gmail.com] > > > > > >> Sent: 14 March 2019 18:16 > > > > > >> To: Paul Durrant <Paul.Durrant@citrix.com> > > > > > >> Cc: qemu-devel@nongnu.org; xen-devel@lists.xenproject.org; marmarek@invisiblethingslab.com; Simon > > > > > >> Gaiser <simon@invisiblethingslab.com>; Stefano Stabellini <sstabellini@kernel.org>; Anthony Perard > > > > > >> <anthony.perard@citrix.com> > > > > > >> Subject: Re: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE > > > > > >> > > > > > >> On Wed, Mar 13, 2019 at 11:09 AM Paul Durrant <Paul.Durrant@citrix.com> wrote: > > > > > >>>> -----Original Message----- > > > > > >>>> From: Jason Andryuk [mailto:jandryuk@gmail.com] > > > > > >>>> Sent: 11 March 2019 18:02 > > > > > >>>> To: qemu-devel@nongnu.org > > > > > >>>> Cc: xen-devel@lists.xenproject.org; marmarek@invisiblethingslab.com; Simon Gaiser > > > > > >>>> <simon@invisiblethingslab.com>; Jason Andryuk <jandryuk@gmail.com>; Stefano Stabellini > > > > > >>>> <sstabellini@kernel.org>; Anthony Perard <anthony.perard@citrix.com>; Paul Durrant > > > > > >>>> <Paul.Durrant@citrix.com> > > > > > >>>> Subject: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE > > > > > >>>> > > > > > >>>> From: Simon Gaiser <simon@invisiblethingslab.com> > > > > > >>>> > > > > > >>>> If a pci memory region has a size < XEN_PAGE_SIZE it can get located at > > > > > >>>> an address which is not page aligned. > > > > > >>> IIRC the PCI spec says that the minimum memory region size should be at least 4k. Should we even be > > > > > >> tolerating BARs smaller than that? > > > > > >>> Paul > > > > > >>> > > > > > >> Hi, Paul. > > > > > >> > > > > > >> Simon found this, so it affects a real device. Simon, do you recall > > > > > >> which device was affected? > > > > > >> > > > > > >> I think BARs only need to be power-of-two size and aligned, and 4k is > > > > > >> not a minimum. 16bytes may be a minimum, but I don't know what the > > > > > >> spec says. > > > > > >> > > > > > >> On an Ivy Bridge system, here are some of the devices with BARs smaller than 4K: > > > > > >> 00:16.0 Communication controller: Intel Corporation 7 Series/C210 > > > > > >> Series Chipset Family MEI Controller #1 (rev 04) > > > > > >> Memory at d0735000 (64-bit, non-prefetchable) [disabled] [size=16] > > > > > >> 00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset > > > > > >> Family USB Enhanced Host Controller #1 (rev 04) (prog-if 20 [EHCI]) > > > > > >> Memory at d0739000 (32-bit, non-prefetchable) [disabled] [size=1K] > > > > > >> 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family > > > > > >> SMBus Controller (rev 04) > > > > > >> Memory at d0734000 (64-bit, non-prefetchable) [disabled] [size=256] > > > > > >> 02:00.0 System peripheral: JMicron Technology Corp. SD/MMC Host > > > > > >> Controller (rev 30) > > > > > >> Memory at d0503000 (32-bit, non-prefetchable) [disabled] [size=256] > > > > > >> > > > > > >> These examples are all 4K aligned, so this is not an issue on this machine. > > > > > >> > > > > > >> Reviewing the code, I'm now wondering if the following in > > > > > >> hw/xen/xen_pt.c:xen_pt_region_update is wrong: rc = > > > > > >> xc_domain_memory_mapping(xen_xc, xen_domid, > > > > > >> XEN_PFN(guest_addr + XC_PAGE_SIZE - 1), > > > > > >> XEN_PFN(machine_addr + XC_PAGE_SIZE - 1), > > > > > >> XEN_PFN(size + XC_PAGE_SIZE - 1), > > > > > >> op); > > > > > >> > > > > > >> If a bar of size 0x100 is at 0xd0500800, then the machine_addr passed > > > > > >> in would be 0xd0501000 which is past the actual location. Should the > > > > > >> call arguments just be XEN_PFN(guest_addr) & XEN_PFN(machine_addr)? > > > > > >> > > > > > >> BARs smaller than a page would also be a problem if BARs for different > > > > > >> devices shared the same page. > > > > > > Exactly. We cannot pass them through with any degree of safety (not that passthrough of an arbitrary device is a particularly safe thing to do anyway). The xen-pt code would instead need to trap those BARs and perform the accesses to the real BAR itself. Ultimately though I think we should be retiring the xen-pt code in favour of a standalone emulator. > > > > > > > > > > It doesn't matter if the BAR is smaller than 4k, if there are holes next > > > > > to it. > > > > > > > > > > Do we know what the case is in practice for these USB controllers? > > > > > > > > > > If the worst comes to the worst, we can re-enumerate the PCI bus to > > > > > ensure that all bars smaller than 4k still have 4k alignment between > > > > > them. That way we can safely pass them through even when they are smaller. > > > > > > > > Andrew, thanks for checking the spec on the minimum BAR size. > > > > > > > > Dropping the Round PCI region patch from QMEU, the guest HVM will have: > > > > > > > > 00:06.0 SD Host controller: Ricoh Co Ltd PCIe SDXC/MMC Host Controller (rev 07) > > > > Memory at f2028800 (32-bit, non-prefetchable) [size=256] > > > > 00:07.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host > > > > Controller (rev 04) (prog-if 30 [XHCI]) > > > > Memory at f2024000 (64-bit, non-prefetchable) [size=8K] > > > > 00:08.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset > > > > Family USB Enhanced Host Controller #2 (rev 05) (prog-if 20 [EHCI]) > > > > Memory at f2028000 (32-bit, non-prefetchable) [size=1K] > > > > 00:09.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset > > > > Family USB Enhanced Host Controller #1 (rev 05) (prog-if 20 [EHCI]) > > > > Memory at f2028400 (32-bit, non-prefetchable) [size=1K] > > > > > > > > 00:09.0, 00:08.0 & 00:06.0 all share the same page. Only 00:08.0 is > > > > working. With some added debugging output, you'll see that the same > > > > page* is used for three of the BARs. > > > > > > > > [00:06.0] mapping guest_addr 0xf2028800 gfn 0xf2028 to maddr > > > > 0xe1a30000 mfn 0xe1a30 > > > > [00:07.0] mapping guest_addr 0xf2024000 gfn 0xf2024 to maddr > > > > 0xe0800000 mfn 0xe0800 > > > > [00:09.0] mapping guest_addr 0xf2028400 gfn 0xf2028 to maddr > > > > 0xe1900000 mfn 0xe1900 > > > > [00:08.0] mapping guest_addr 0xf2028000 gfn 0xf2028 to maddr > > > > 0xe1a2f000 mfn 0xe1a2f > > > > > > The patch below should prevent hvmloader from placing multiple BARs on > > > the same page, could you give it a try? > > > > > > Note that this is not going to prevent the guest from moving those > > > BARs around and place them in the same page, thus breaking the initial > > > placement done by hvmloader. > > > > > > Thanks, Roger. > > > > Hi, Roger. > > > > I've minimally tested this. Yes, this patch seems to place small BARs > > into separate pages. The linux stubdom and QEMU then use the spacing > > as provided by hvmloader. > > Roger, > > Would you mind submitting this patch to Xen? Hm, I'm half minded regarding this patch. It feels more like a bandaid than a proper solution. Mapping BARs not multiple of page-sizes is dangerous because AFAIK there's no entity that asserts there isn't any other BAR from a different device on the same page, and hence you might end up mapping some MMIO region from another device inadvertently. Anyway, I can formally submit the patch since it's no worse than what's currently done, but I would clearly state this is not safe in it's current state. Thanks, Roger. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Xen-devel] [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE 2020-01-14 10:04 ` Roger Pau Monné @ 2020-01-14 14:41 ` Jason Andryuk 2020-01-14 18:04 ` Roger Pau Monné 0 siblings, 1 reply; 14+ messages in thread From: Jason Andryuk @ 2020-01-14 14:41 UTC (permalink / raw) To: Roger Pau Monné Cc: Stefano Stabellini, Andrew Cooper, Paul Durrant, marmarek, qemu-devel, Simon Gaiser, Anthony Perard, xen-devel On Tue, Jan 14, 2020 at 5:04 AM Roger Pau Monné <roger.pau@citrix.com> wrote: > > On Mon, Jan 13, 2020 at 02:01:47PM -0500, Jason Andryuk wrote: > > On Fri, Mar 22, 2019 at 3:43 PM Jason Andryuk <jandryuk@gmail.com> wrote: > > > > > > On Thu, Mar 21, 2019 at 11:09 PM Roger Pau Monné <roger.pau@citrix.com> wrote: > > > > > > > > On Wed, Mar 20, 2019 at 01:28:47PM -0400, Jason Andryuk wrote: > > > > > On Fri, Mar 15, 2019 at 12:28 PM Andrew Cooper > > > > > <andrew.cooper3@citrix.com> wrote: > > > > > > > > > > > > On 15/03/2019 09:17, Paul Durrant wrote: > > > > > > >> -----Original Message----- > > > > > > >> From: Jason Andryuk [mailto:jandryuk@gmail.com] > > > > > > >> Sent: 14 March 2019 18:16 > > > > > > >> To: Paul Durrant <Paul.Durrant@citrix.com> > > > > > > >> Cc: qemu-devel@nongnu.org; xen-devel@lists.xenproject.org; marmarek@invisiblethingslab.com; Simon > > > > > > >> Gaiser <simon@invisiblethingslab.com>; Stefano Stabellini <sstabellini@kernel.org>; Anthony Perard > > > > > > >> <anthony.perard@citrix.com> > > > > > > >> Subject: Re: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE > > > > > > >> > > > > > > >> On Wed, Mar 13, 2019 at 11:09 AM Paul Durrant <Paul.Durrant@citrix.com> wrote: > > > > > > >>>> -----Original Message----- > > > > > > >>>> From: Jason Andryuk [mailto:jandryuk@gmail.com] > > > > > > >>>> Sent: 11 March 2019 18:02 > > > > > > >>>> To: qemu-devel@nongnu.org > > > > > > >>>> Cc: xen-devel@lists.xenproject.org; marmarek@invisiblethingslab.com; Simon Gaiser > > > > > > >>>> <simon@invisiblethingslab.com>; Jason Andryuk <jandryuk@gmail.com>; Stefano Stabellini > > > > > > >>>> <sstabellini@kernel.org>; Anthony Perard <anthony.perard@citrix.com>; Paul Durrant > > > > > > >>>> <Paul.Durrant@citrix.com> > > > > > > >>>> Subject: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE > > > > > > >>>> > > > > > > >>>> From: Simon Gaiser <simon@invisiblethingslab.com> > > > > > > >>>> > > > > > > >>>> If a pci memory region has a size < XEN_PAGE_SIZE it can get located at > > > > > > >>>> an address which is not page aligned. > > > > > > >>> IIRC the PCI spec says that the minimum memory region size should be at least 4k. Should we even be > > > > > > >> tolerating BARs smaller than that? > > > > > > >>> Paul > > > > > > >>> > > > > > > >> Hi, Paul. > > > > > > >> > > > > > > >> Simon found this, so it affects a real device. Simon, do you recall > > > > > > >> which device was affected? > > > > > > >> > > > > > > >> I think BARs only need to be power-of-two size and aligned, and 4k is > > > > > > >> not a minimum. 16bytes may be a minimum, but I don't know what the > > > > > > >> spec says. > > > > > > >> > > > > > > >> On an Ivy Bridge system, here are some of the devices with BARs smaller than 4K: > > > > > > >> 00:16.0 Communication controller: Intel Corporation 7 Series/C210 > > > > > > >> Series Chipset Family MEI Controller #1 (rev 04) > > > > > > >> Memory at d0735000 (64-bit, non-prefetchable) [disabled] [size=16] > > > > > > >> 00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset > > > > > > >> Family USB Enhanced Host Controller #1 (rev 04) (prog-if 20 [EHCI]) > > > > > > >> Memory at d0739000 (32-bit, non-prefetchable) [disabled] [size=1K] > > > > > > >> 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family > > > > > > >> SMBus Controller (rev 04) > > > > > > >> Memory at d0734000 (64-bit, non-prefetchable) [disabled] [size=256] > > > > > > >> 02:00.0 System peripheral: JMicron Technology Corp. SD/MMC Host > > > > > > >> Controller (rev 30) > > > > > > >> Memory at d0503000 (32-bit, non-prefetchable) [disabled] [size=256] > > > > > > >> > > > > > > >> These examples are all 4K aligned, so this is not an issue on this machine. > > > > > > >> > > > > > > >> Reviewing the code, I'm now wondering if the following in > > > > > > >> hw/xen/xen_pt.c:xen_pt_region_update is wrong: rc = > > > > > > >> xc_domain_memory_mapping(xen_xc, xen_domid, > > > > > > >> XEN_PFN(guest_addr + XC_PAGE_SIZE - 1), > > > > > > >> XEN_PFN(machine_addr + XC_PAGE_SIZE - 1), > > > > > > >> XEN_PFN(size + XC_PAGE_SIZE - 1), > > > > > > >> op); > > > > > > >> > > > > > > >> If a bar of size 0x100 is at 0xd0500800, then the machine_addr passed > > > > > > >> in would be 0xd0501000 which is past the actual location. Should the > > > > > > >> call arguments just be XEN_PFN(guest_addr) & XEN_PFN(machine_addr)? > > > > > > >> > > > > > > >> BARs smaller than a page would also be a problem if BARs for different > > > > > > >> devices shared the same page. > > > > > > > Exactly. We cannot pass them through with any degree of safety (not that passthrough of an arbitrary device is a particularly safe thing to do anyway). The xen-pt code would instead need to trap those BARs and perform the accesses to the real BAR itself. Ultimately though I think we should be retiring the xen-pt code in favour of a standalone emulator. > > > > > > > > > > > > It doesn't matter if the BAR is smaller than 4k, if there are holes next > > > > > > to it. > > > > > > > > > > > > Do we know what the case is in practice for these USB controllers? > > > > > > > > > > > > If the worst comes to the worst, we can re-enumerate the PCI bus to > > > > > > ensure that all bars smaller than 4k still have 4k alignment between > > > > > > them. That way we can safely pass them through even when they are smaller. > > > > > > > > > > Andrew, thanks for checking the spec on the minimum BAR size. > > > > > > > > > > Dropping the Round PCI region patch from QMEU, the guest HVM will have: > > > > > > > > > > 00:06.0 SD Host controller: Ricoh Co Ltd PCIe SDXC/MMC Host Controller (rev 07) > > > > > Memory at f2028800 (32-bit, non-prefetchable) [size=256] > > > > > 00:07.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host > > > > > Controller (rev 04) (prog-if 30 [XHCI]) > > > > > Memory at f2024000 (64-bit, non-prefetchable) [size=8K] > > > > > 00:08.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset > > > > > Family USB Enhanced Host Controller #2 (rev 05) (prog-if 20 [EHCI]) > > > > > Memory at f2028000 (32-bit, non-prefetchable) [size=1K] > > > > > 00:09.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset > > > > > Family USB Enhanced Host Controller #1 (rev 05) (prog-if 20 [EHCI]) > > > > > Memory at f2028400 (32-bit, non-prefetchable) [size=1K] > > > > > > > > > > 00:09.0, 00:08.0 & 00:06.0 all share the same page. Only 00:08.0 is > > > > > working. With some added debugging output, you'll see that the same > > > > > page* is used for three of the BARs. > > > > > > > > > > [00:06.0] mapping guest_addr 0xf2028800 gfn 0xf2028 to maddr > > > > > 0xe1a30000 mfn 0xe1a30 > > > > > [00:07.0] mapping guest_addr 0xf2024000 gfn 0xf2024 to maddr > > > > > 0xe0800000 mfn 0xe0800 > > > > > [00:09.0] mapping guest_addr 0xf2028400 gfn 0xf2028 to maddr > > > > > 0xe1900000 mfn 0xe1900 > > > > > [00:08.0] mapping guest_addr 0xf2028000 gfn 0xf2028 to maddr > > > > > 0xe1a2f000 mfn 0xe1a2f > > > > > > > > The patch below should prevent hvmloader from placing multiple BARs on > > > > the same page, could you give it a try? > > > > > > > > Note that this is not going to prevent the guest from moving those > > > > BARs around and place them in the same page, thus breaking the initial > > > > placement done by hvmloader. > > > > > > > > Thanks, Roger. > > > > > > Hi, Roger. > > > > > > I've minimally tested this. Yes, this patch seems to place small BARs > > > into separate pages. The linux stubdom and QEMU then use the spacing > > > as provided by hvmloader. > > > > Roger, > > > > Would you mind submitting this patch to Xen? > > Hm, I'm half minded regarding this patch. It feels more like a bandaid > than a proper solution. Mapping BARs not multiple of page-sizes is > dangerous because AFAIK there's no entity that asserts there isn't any > other BAR from a different device on the same page, and hence you > might end up mapping some MMIO region from another device > inadvertently. We have the guest, linux stubdom with qemu, & dom0. Are you concerned that all of them need a minimum of page alignment? Linux PCI subsytem has an option resource_alignment that can be applied to either a single device or all devices. Booting with pci=resource_aligment=4096 will align each device to a page. Do you think pciback should force resource_alignment=4096 for dom0? Are there other MMIO ranges to be concerned about adjacent to BARs? On my one test machine with a BAR smaller than 4096, the firmware already sets an alignment of 4096. Linux dom0 seems to keep the firmware BAR alignment by default. > Anyway, I can formally submit the patch since it's no worse than > what's currently done, but I would clearly state this is not safe in > it's current state. Regards, Jason ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Xen-devel] [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE 2020-01-14 14:41 ` Jason Andryuk @ 2020-01-14 18:04 ` Roger Pau Monné 2020-01-15 8:33 ` Durrant, Paul 0 siblings, 1 reply; 14+ messages in thread From: Roger Pau Monné @ 2020-01-14 18:04 UTC (permalink / raw) To: Jason Andryuk Cc: Stefano Stabellini, Andrew Cooper, Paul Durrant, marmarek, qemu-devel, Simon Gaiser, Anthony Perard, xen-devel On Tue, Jan 14, 2020 at 09:41:46AM -0500, Jason Andryuk wrote: > On Tue, Jan 14, 2020 at 5:04 AM Roger Pau Monné <roger.pau@citrix.com> wrote: > > > > On Mon, Jan 13, 2020 at 02:01:47PM -0500, Jason Andryuk wrote: > > > On Fri, Mar 22, 2019 at 3:43 PM Jason Andryuk <jandryuk@gmail.com> wrote: > > > > > > > > On Thu, Mar 21, 2019 at 11:09 PM Roger Pau Monné <roger.pau@citrix.com> wrote: > > > > > > > > > > The patch below should prevent hvmloader from placing multiple BARs on > > > > > the same page, could you give it a try? > > > > > > > > > > Note that this is not going to prevent the guest from moving those > > > > > BARs around and place them in the same page, thus breaking the initial > > > > > placement done by hvmloader. > > > > > > > > > > Thanks, Roger. > > > > > > > > Hi, Roger. > > > > > > > > I've minimally tested this. Yes, this patch seems to place small BARs > > > > into separate pages. The linux stubdom and QEMU then use the spacing > > > > as provided by hvmloader. > > > > > > Roger, > > > > > > Would you mind submitting this patch to Xen? > > > > Hm, I'm half minded regarding this patch. It feels more like a bandaid > > than a proper solution. Mapping BARs not multiple of page-sizes is > > dangerous because AFAIK there's no entity that asserts there isn't any > > other BAR from a different device on the same page, and hence you > > might end up mapping some MMIO region from another device > > inadvertently. > > We have the guest, linux stubdom with qemu, & dom0. Are you concerned > that all of them need a minimum of page alignment? No, not really. The hardware domain (dom0 in normal deployments) should be the one that makes sure there are no BARs sharing physical pages. > Linux PCI subsytem has an option resource_alignment that can be > applied to either a single device or all devices. Booting with > pci=resource_aligment=4096 will align each device to a page. Do you > think pciback should force resource_alignment=4096 for dom0? Ideally Xen should keep track of the BARs position and size and refuse to passthrough devices that have BARs sharing a page with other devices BARs. > Are > there other MMIO ranges to be concerned about adjacent to BARs? IIRC you can have two BARs of different devices in the same 4K page, BARs are only aligned to it's size, so BARs smaller than 4K are not required to be page aligned. > On my one test machine with a BAR smaller than 4096, the firmware > already sets an alignment of 4096. Linux dom0 seems to keep the > firmware BAR alignment by default. The PCI spec recommend BARs to be sized to a multiple of a page size, but sadly that's not a mandatory requirement. Will submit the patch now, thanks for the ping, I completely forgot about this TBH. Roger. ^ permalink raw reply [flat|nested] 14+ messages in thread
* RE: [Xen-devel] [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE 2020-01-14 18:04 ` Roger Pau Monné @ 2020-01-15 8:33 ` Durrant, Paul 0 siblings, 0 replies; 14+ messages in thread From: Durrant, Paul @ 2020-01-15 8:33 UTC (permalink / raw) To: Roger Pau Monné, Jason Andryuk Cc: Andrew Cooper, Stefano Stabellini, marmarek, qemu-devel, Simon Gaiser, Anthony Perard, xen-devel > -----Original Message----- > > > Linux PCI subsytem has an option resource_alignment that can be > > applied to either a single device or all devices. Booting with > > pci=resource_aligment=4096 will align each device to a page. Do you > > think pciback should force resource_alignment=4096 for dom0? > That sounds like a good idea. > Ideally Xen should keep track of the BARs position and size and refuse > to passthrough devices that have BARs sharing a page with other > devices BARs. > > > Are > > there other MMIO ranges to be concerned about adjacent to BARs? > > IIRC you can have two BARs of different devices in the same 4K page, > BARs are only aligned to it's size, so BARs smaller than 4K are not > required to be page aligned. If we had a notion of assignment groups for this, as well as devices sharing requester id, then Xen would not need to refuse pass-through, it would just require that all devices sharing the page were passed through as a unit. Paul ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2020-01-15 13:24 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-03-11 18:02 [Qemu-devel] [PATCH 0/6] Xen stubdom support Jason Andryuk 2019-03-11 18:02 ` [Qemu-devel] [PATCH 1/6] xen: Introduce -xen-stubdom option Jason Andryuk 2019-03-11 18:06 ` Paolo Bonzini 2019-03-11 19:46 ` Jason Andryuk 2019-03-11 18:02 ` [Qemu-devel] [PATCH 2/6] xen: Move xenstore initialization to common location Jason Andryuk 2019-03-11 18:02 ` [Qemu-devel] [PATCH 3/6] xen: Skip backend initialization for stubdom Jason Andryuk 2019-03-11 18:02 ` [Qemu-devel] [PATCH 4/6] xen: Set HVM_PARAM_DM_DOMAIN for stubdom on older Xen Jason Andryuk 2019-03-11 18:02 ` [Qemu-devel] [PATCH 5/6] xen-pt: Hide MSI-X from xen stubdoms Jason Andryuk 2019-03-11 18:02 ` [Qemu-devel] [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE Jason Andryuk [not found] ` <ee0da83d3f054e72ae450437c8834d04@AMSPEX02CL02.citrite.net> [not found] ` <CAKf6xpujVs3RPJcb+2vqPZFcYwhdDcjbt=S_+awjPuPFpFHsPw@mail.gmail.com> [not found] ` <57dc1083d20a469785f05a2e5250a820@AMSPEX02CL02.citrite.net> [not found] ` <cd5f1ecc-576f-b7d0-3090-4f3e4faf6148@citrix.com> [not found] ` <CAKf6xpt4XYXn2xHJoVY_ibcaHSw-ED10V7ZGNKuDdkiJ93RS0A@mail.gmail.com> [not found] ` <20190322030936.fkiajz5ifgaejkd4@MacBook-Air-de-Roger.local> [not found] ` <CAKf6xptP_b-+FuscjsTK9G7pMeVS8drvA_t+xb5bdF2zxxmWfA@mail.gmail.com> 2020-01-13 19:01 ` [Xen-devel] " Jason Andryuk 2020-01-14 10:04 ` Roger Pau Monné 2020-01-14 14:41 ` Jason Andryuk 2020-01-14 18:04 ` Roger Pau Monné 2020-01-15 8:33 ` Durrant, Paul
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).