[PATCH 0/3] Support xen-driven USB3 debug capability

[PATCH 0/3] Support xen-driven USB3 debug capability

[Connor Davis]

Hi all,

This goal of this series is to allow the USB3 debug capability (DbC) to be
safely used by xen while linux runs as dom0. The first patch prevents
the early DbC driver from using an already-running DbC. The second
exports xen_dbgp_reset_prep and xen_dbgp_external_startup functions when
CONFIG_XEN_DOM0 is enabled so they may be used by the xHCI driver.
The last uses those functions to notify xen of unsafe periods (e.g. reset
and D3hot transition).

Thanks,
Connor

--
Connor Davis (3):
  usb: early: Avoid using DbC if already enabled
  xen: Export dbgp functions when CONFIG_XEN_DOM0 is enabled
  usb: xhci: Notify xen when DbC is unsafe to use

 drivers/usb/early/xhci-dbc.c   | 10 ++++++
 drivers/usb/host/xhci-dbgcap.h |  6 ++++
 drivers/usb/host/xhci.c        | 57 ++++++++++++++++++++++++++++++++++
 drivers/usb/host/xhci.h        |  1 +
 drivers/xen/dbgp.c             |  2 +-
 5 files changed, 75 insertions(+), 1 deletion(-)


base-commit: 88b06399c9c766c283e070b022b5ceafa4f63f19
--
2.31.1

[PATCH 1/3] usb: early: Avoid using DbC if already enabled

[Connor Davis]

Check if the debug capability is enabled in early_xdbc_parse_parameter,
and if it is, return with an error. This avoids collisions with whatever
enabled the DbC prior to linux starting.

Signed-off-by: Connor Davis <connojdavis@gmail.com>
---
 drivers/usb/early/xhci-dbc.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/drivers/usb/early/xhci-dbc.c b/drivers/usb/early/xhci-dbc.c
index be4ecbabdd58..ca67fddc2d36 100644
--- a/drivers/usb/early/xhci-dbc.c
+++ b/drivers/usb/early/xhci-dbc.c
@@ -642,6 +642,16 @@ int __init early_xdbc_parse_parameter(char *s)
 	}
 	xdbc.xdbc_reg = (struct xdbc_regs __iomem *)(xdbc.xhci_base + offset);

+	if (readl(&xdbc.xdbc_reg->control) & CTRL_DBC_ENABLE) {
+		pr_notice("xhci debug capability already in use\n");
+		early_iounmap(xdbc.xhci_base, xdbc.xhci_length);
+		xdbc.xdbc_reg = NULL;
+		xdbc.xhci_base = NULL;
+		xdbc.xhci_length = 0;
+
+		return -ENODEV;
+	}
+
 	return 0;
 }

--
2.31.1

[PATCH 2/3] xen: Export dbgp functions when CONFIG_XEN_DOM0 is enabled

[Connor Davis]

Export xen_dbgp_reset_prep and xen_dbgp_external_startup
when CONFIG_XEN_DOM0 is defined. This allows use of these symbols
even if CONFIG_EARLY_PRINK_DBGP is defined.

Signed-off-by: Connor Davis <connojdavis@gmail.com>
---
 drivers/xen/dbgp.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/xen/dbgp.c b/drivers/xen/dbgp.c
index cfb5de31d860..fef32dd1a5dc 100644
--- a/drivers/xen/dbgp.c
+++ b/drivers/xen/dbgp.c
@@ -44,7 +44,7 @@ int xen_dbgp_external_startup(struct usb_hcd *hcd)
 	return xen_dbgp_op(hcd, PHYSDEVOP_DBGP_RESET_DONE);
 }

-#ifndef CONFIG_EARLY_PRINTK_DBGP
+#if defined(CONFIG_XEN_DOM0) || !defined(CONFIG_EARLY_PRINTK_DBGP)
 #include <linux/export.h>
 EXPORT_SYMBOL_GPL(xen_dbgp_reset_prep);
 EXPORT_SYMBOL_GPL(xen_dbgp_external_startup);
--
2.31.1

Re: [PATCH 2/3] xen: Export dbgp functions when CONFIG_XEN_DOM0 is enabled

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 335 bytes --]

On 12.05.21 02:18, Connor Davis wrote:
> Export xen_dbgp_reset_prep and xen_dbgp_external_startup
> when CONFIG_XEN_DOM0 is defined. This allows use of these symbols
> even if CONFIG_EARLY_PRINK_DBGP is defined.
> 
> Signed-off-by: Connor Davis <connojdavis@gmail.com>

Acked-by: Juergen Gross <jgross@suse.com>


Juergen


[-- Attachment #1.1.2: OpenPGP_0xB0DE9DD628BF132F.asc --]
[-- Type: application/pgp-keys, Size: 3135 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [PATCH 2/3] xen: Export dbgp functions when CONFIG_XEN_DOM0 is enabled

[Connor Davis]

On May 12, 21, Juergen Gross wrote:
> On 12.05.21 02:18, Connor Davis wrote:
> > Export xen_dbgp_reset_prep and xen_dbgp_external_startup
> > when CONFIG_XEN_DOM0 is defined. This allows use of these symbols
> > even if CONFIG_EARLY_PRINK_DBGP is defined.
> >
> > Signed-off-by: Connor Davis <connojdavis@gmail.com>
>
> Acked-by: Juergen Gross <jgross@suse.com>

Thank you.

- Connor

Re: [PATCH 2/3] xen: Export dbgp functions when CONFIG_XEN_DOM0 is enabled

[Boris Ostrovsky]

On 5/11/21 8:18 PM, Connor Davis wrote:
> Export xen_dbgp_reset_prep and xen_dbgp_external_startup
> when CONFIG_XEN_DOM0 is defined. This allows use of these symbols
> even if CONFIG_EARLY_PRINK_DBGP is defined.
>
> Signed-off-by: Connor Davis <connojdavis@gmail.com>
> ---
>  drivers/xen/dbgp.c | 2 +-


Unrelated to your patch but since you are fixing things around that file --- should we return -EPERM in xen_dbgp_op() when !xen_initial_domain()?



-boris

Re: [PATCH 2/3] xen: Export dbgp functions when CONFIG_XEN_DOM0 is enabled

[Connor Davis]

On May 12, 21, Boris Ostrovsky wrote:
>
> On 5/11/21 8:18 PM, Connor Davis wrote:
> > Export xen_dbgp_reset_prep and xen_dbgp_external_startup
> > when CONFIG_XEN_DOM0 is defined. This allows use of these symbols
> > even if CONFIG_EARLY_PRINK_DBGP is defined.
> >
> > Signed-off-by: Connor Davis <connojdavis@gmail.com>
> > ---
> >  drivers/xen/dbgp.c | 2 +-
>
>
> Unrelated to your patch but since you are fixing things around that file --- should we return -EPERM in xen_dbgp_op() when !xen_initial_domain()?

Yeah looks like it. That would make patch 3 simpler too.
Do you want me to add a patch that fixes that up?

>
> -boris
>

Thanks,
Connor

Re: [PATCH 2/3] xen: Export dbgp functions when CONFIG_XEN_DOM0 is enabled

[Boris Ostrovsky]

On 5/12/21 10:58 AM, Connor Davis wrote:
> On May 12, 21, Boris Ostrovsky wrote:
>> Unrelated to your patch but since you are fixing things around that file --- should we return -EPERM in xen_dbgp_op() when !xen_initial_domain()?
> Yeah looks like it. That would make patch 3 simpler too.
> Do you want me to add a patch that fixes that up?


Yes, please.


-boris

[PATCH 3/3] usb: xhci: Notify xen when DbC is unsafe to use

[Connor Davis]

When running as a dom0 guest on Xen, check if the USB3 debug
capability is enabled before xHCI reset, suspend, and resume. If it
is, call xen_dbgp_reset_prep() to notify Xen that it is unsafe to touch
MMIO registers until the next xen_dbgp_external_startup().

This notification allows Xen to avoid undefined behavior resulting
from MMIO access when the host controller's CNR bit is set or when
the device transitions to D3hot.

Signed-off-by: Connor Davis <connojdavis@gmail.com>
---
 drivers/usb/host/xhci-dbgcap.h |  6 ++++
 drivers/usb/host/xhci.c        | 57 ++++++++++++++++++++++++++++++++++
 drivers/usb/host/xhci.h        |  1 +
 3 files changed, 64 insertions(+)

diff --git a/drivers/usb/host/xhci-dbgcap.h b/drivers/usb/host/xhci-dbgcap.h
index c70b78d504eb..24784b82a840 100644
--- a/drivers/usb/host/xhci-dbgcap.h
+++ b/drivers/usb/host/xhci-dbgcap.h
@@ -227,4 +227,10 @@ static inline int xhci_dbc_resume(struct xhci_hcd *xhci)
 	return 0;
 }
 #endif /* CONFIG_USB_XHCI_DBGCAP */
+
+#ifdef CONFIG_XEN_DOM0
+int xen_dbgp_reset_prep(struct usb_hcd *hcd);
+int xen_dbgp_external_startup(struct usb_hcd *hcd);
+#endif /* CONFIG_XEN_DOM0 */
+
 #endif /* __LINUX_XHCI_DBGCAP_H */
diff --git a/drivers/usb/host/xhci.c b/drivers/usb/host/xhci.c
index ca9385d22f68..afe44169183f 100644
--- a/drivers/usb/host/xhci.c
+++ b/drivers/usb/host/xhci.c
@@ -37,6 +37,57 @@ static unsigned long long quirks;
 module_param(quirks, ullong, S_IRUGO);
 MODULE_PARM_DESC(quirks, "Bit flags for quirks to be enabled as default");

+#ifdef CONFIG_XEN_DOM0
+#include <xen/xen.h>
+
+static void xhci_dbc_external_reset_prep(struct xhci_hcd *xhci)
+{
+	struct dbc_regs __iomem *regs;
+	void __iomem		*base;
+	int			dbc_cap;
+
+	if (!xen_initial_domain())
+		return;
+
+	base = &xhci->cap_regs->hc_capbase;
+	dbc_cap = xhci_find_next_ext_cap(base, 0, XHCI_EXT_CAPS_DEBUG);
+
+	if (!dbc_cap)
+		return;
+
+	xhci->external_dbc = 0;
+	regs = base + dbc_cap;
+
+	if (readl(&regs->control) & DBC_CTRL_DBC_ENABLE) {
+		if (xen_dbgp_reset_prep(xhci_to_hcd(xhci)))
+			xhci_dbg_trace(xhci, trace_xhci_dbg_init,
+					"// Failed to reset external DBC");
+		else {
+			xhci->external_dbc = 1;
+			xhci_dbg_trace(xhci, trace_xhci_dbg_init,
+					"// Completed reset of external DBC");
+		}
+	}
+}
+
+static void xhci_dbc_external_reset_done(struct xhci_hcd *xhci)
+{
+	if (!xen_initial_domain() || !xhci->external_dbc)
+		return;
+
+	if (xen_dbgp_external_startup(xhci_to_hcd(xhci)))
+		xhci->external_dbc = 0;
+}
+#else
+static void xhci_dbc_external_reset_prep(struct xhci_hcd *xhci)
+{
+}
+
+static void xhci_dbc_external_reset_done(struct xhci_hcd *xhci)
+{
+}
+#endif
+
 static bool td_on_ring(struct xhci_td *td, struct xhci_ring *ring)
 {
 	struct xhci_segment *seg = ring->first_seg;
@@ -180,6 +231,8 @@ int xhci_reset(struct xhci_hcd *xhci)
 		return 0;
 	}

+	xhci_dbc_external_reset_prep(xhci);
+
 	xhci_dbg_trace(xhci, trace_xhci_dbg_init, "// Reset the HC");
 	command = readl(&xhci->op_regs->command);
 	command |= CMD_RESET;
@@ -211,6 +264,8 @@ int xhci_reset(struct xhci_hcd *xhci)
 	 */
 	ret = xhci_handshake(&xhci->op_regs->status,
 			STS_CNR, 0, 10 * 1000 * 1000);
+	if (!ret)
+		xhci_dbc_external_reset_done(xhci);

 	xhci->usb2_rhub.bus_state.port_c_suspend = 0;
 	xhci->usb2_rhub.bus_state.suspended_ports = 0;
@@ -991,6 +1046,7 @@ int xhci_suspend(struct xhci_hcd *xhci, bool do_wakeup)
 		return 0;

 	xhci_dbc_suspend(xhci);
+	xhci_dbc_external_reset_prep(xhci);

 	/* Don't poll the roothubs on bus suspend. */
 	xhci_dbg(xhci, "%s: stopping port polling.\n", __func__);
@@ -1225,6 +1281,7 @@ int xhci_resume(struct xhci_hcd *xhci, bool hibernated)
 	spin_unlock_irq(&xhci->lock);

 	xhci_dbc_resume(xhci);
+	xhci_dbc_external_reset_done(xhci);

  done:
 	if (retval == 0) {
diff --git a/drivers/usb/host/xhci.h b/drivers/usb/host/xhci.h
index 2595a8f057c4..61d8efc9eef2 100644
--- a/drivers/usb/host/xhci.h
+++ b/drivers/usb/host/xhci.h
@@ -1920,6 +1920,7 @@ struct xhci_hcd {
 	struct list_head	regset_list;

 	void			*dbc;
+	int			external_dbc;
 	/* platform-specific data -- must come last */
 	unsigned long		priv[] __aligned(sizeof(s64));
 };
--
2.31.1

Re: [PATCH 3/3] usb: xhci: Notify xen when DbC is unsafe to use

[Greg Kroah-Hartman]

On Tue, May 11, 2021 at 06:18:21PM -0600, Connor Davis wrote:
> When running as a dom0 guest on Xen, check if the USB3 debug
> capability is enabled before xHCI reset, suspend, and resume. If it
> is, call xen_dbgp_reset_prep() to notify Xen that it is unsafe to touch
> MMIO registers until the next xen_dbgp_external_startup().
> 
> This notification allows Xen to avoid undefined behavior resulting
> from MMIO access when the host controller's CNR bit is set or when
> the device transitions to D3hot.
> 
> Signed-off-by: Connor Davis <connojdavis@gmail.com>
> ---
>  drivers/usb/host/xhci-dbgcap.h |  6 ++++
>  drivers/usb/host/xhci.c        | 57 ++++++++++++++++++++++++++++++++++
>  drivers/usb/host/xhci.h        |  1 +
>  3 files changed, 64 insertions(+)
> 
> diff --git a/drivers/usb/host/xhci-dbgcap.h b/drivers/usb/host/xhci-dbgcap.h
> index c70b78d504eb..24784b82a840 100644
> --- a/drivers/usb/host/xhci-dbgcap.h
> +++ b/drivers/usb/host/xhci-dbgcap.h
> @@ -227,4 +227,10 @@ static inline int xhci_dbc_resume(struct xhci_hcd *xhci)
>  	return 0;
>  }
>  #endif /* CONFIG_USB_XHCI_DBGCAP */
> +
> +#ifdef CONFIG_XEN_DOM0
> +int xen_dbgp_reset_prep(struct usb_hcd *hcd);
> +int xen_dbgp_external_startup(struct usb_hcd *hcd);
> +#endif /* CONFIG_XEN_DOM0 */
> +
>  #endif /* __LINUX_XHCI_DBGCAP_H */
> diff --git a/drivers/usb/host/xhci.c b/drivers/usb/host/xhci.c
> index ca9385d22f68..afe44169183f 100644
> --- a/drivers/usb/host/xhci.c
> +++ b/drivers/usb/host/xhci.c
> @@ -37,6 +37,57 @@ static unsigned long long quirks;
>  module_param(quirks, ullong, S_IRUGO);
>  MODULE_PARM_DESC(quirks, "Bit flags for quirks to be enabled as default");
> 
> +#ifdef CONFIG_XEN_DOM0
> +#include <xen/xen.h>

<snip>

Can't this #ifdef stuff go into a .h file?

thanks,

greg k-h

Re: [PATCH 3/3] usb: xhci: Notify xen when DbC is unsafe to use

[Connor Davis]

On May 12, 21, Greg Kroah-Hartman wrote:
> On Tue, May 11, 2021 at 06:18:21PM -0600, Connor Davis wrote:
> > When running as a dom0 guest on Xen, check if the USB3 debug
> > capability is enabled before xHCI reset, suspend, and resume. If it
> > is, call xen_dbgp_reset_prep() to notify Xen that it is unsafe to touch
> > MMIO registers until the next xen_dbgp_external_startup().
> >
> > This notification allows Xen to avoid undefined behavior resulting
> > from MMIO access when the host controller's CNR bit is set or when
> > the device transitions to D3hot.
> >
> > Signed-off-by: Connor Davis <connojdavis@gmail.com>
> > ---
> >  drivers/usb/host/xhci-dbgcap.h |  6 ++++
> >  drivers/usb/host/xhci.c        | 57 ++++++++++++++++++++++++++++++++++
> >  drivers/usb/host/xhci.h        |  1 +
> >  3 files changed, 64 insertions(+)
> >
> > diff --git a/drivers/usb/host/xhci-dbgcap.h b/drivers/usb/host/xhci-dbgcap.h
> > index c70b78d504eb..24784b82a840 100644
> > --- a/drivers/usb/host/xhci-dbgcap.h
> > +++ b/drivers/usb/host/xhci-dbgcap.h
> > @@ -227,4 +227,10 @@ static inline int xhci_dbc_resume(struct xhci_hcd *xhci)
> >  	return 0;
> >  }
> >  #endif /* CONFIG_USB_XHCI_DBGCAP */
> > +
> > +#ifdef CONFIG_XEN_DOM0
> > +int xen_dbgp_reset_prep(struct usb_hcd *hcd);
> > +int xen_dbgp_external_startup(struct usb_hcd *hcd);
> > +#endif /* CONFIG_XEN_DOM0 */
> > +
> >  #endif /* __LINUX_XHCI_DBGCAP_H */
> > diff --git a/drivers/usb/host/xhci.c b/drivers/usb/host/xhci.c
> > index ca9385d22f68..afe44169183f 100644
> > --- a/drivers/usb/host/xhci.c
> > +++ b/drivers/usb/host/xhci.c
> > @@ -37,6 +37,57 @@ static unsigned long long quirks;
> >  module_param(quirks, ullong, S_IRUGO);
> >  MODULE_PARM_DESC(quirks, "Bit flags for quirks to be enabled as default");
> >
> > +#ifdef CONFIG_XEN_DOM0
> > +#include <xen/xen.h>
>
> <snip>
>
> Can't this #ifdef stuff go into a .h file?
>

Yep, will clean that up in v2.

> thanks,
>
> greg k-h

Thanks,
Connor