[PATCH v5 0/5] Fix controller halt and endxfer timeout issues

[PATCH v5 0/5] Fix controller halt and endxfer timeout issues

[Wesley Cheng]

Changes in v5:
- Rebased series on usb-testing from patch #5 onwards.

Changes in v4:
- Split the increase timeout patch into separate patches. #1 for the
gadget suspend/resume locking changes #2 for the increased timeout
- Modified msleep to usleep_range w/ an interval of 1-2ms and a max
timeout of 4s.

Changes in v3:
- Modified the msleep() duration to ~2s versus ~10s due to the minimum
mdelay() value.
- Removed patch to modify DEP flags during dwc3_stop_active_transfer().
This was not required after fixing the logic to allow EP xfercomplete
events to be handled on EP0.
- Added some changes to account for a cable disconnect scenario, where
dwc3_gadget_pullup() would not be executed to stop active transfers.
Needed to add some logic to the disconnect interrupt to ensure that we
cleanup/restart any pending SETUP transaction, so that we can clear the
EP0 delayed stop status. (if pending)
- Added patch to ensure that we don't proceed with umapping buffers
until the endxfer was actually sent.

Changes in v2:
- Moved msleep() to before reading status register for halted state
- Fixed kernel bot errors
- Clearing DEP flags in __dwc3_stop_active_transfers()
- Added Suggested-by tags and link references to previous discussions

This patch series addresses some issues seen while testing with the latest
soft disconnect implementation where EP events are allowed to process while
the controller halt is occurring.

#1
Since routines can now interweave, we can see that the soft disconnect can
occur while conndone is being serviced.  This leads to a controller halt
timeout, as the soft disconnect clears the DEP flags, for which conndone
interrupt handler will issue a __dwc3_ep_enable(ep0), that leads to
re-issuing the set ep config command for every endpoint.

#2
Function drivers can ask for a delayed_status phase, while it processes the
received SETUP packet.  This can lead to large delays when handling the
soft disconnect routine.  To improve the timing, forcefully send the status
phase, as we are going to disconnect from the host.

#3
Ensure that local interrupts are left enabled, so that EP0 events can be
processed while the soft disconnect/dequeue is happening.

#4
Since EP0 events can occur during controller halt, it may increase the time
needed for the controller to fully stop.

#5
Account for cable disconnect scenarios where nothing may cause the endxfer
retry if DWC3_EP_DELAY_STOP is set.

#6
Avoid unmapping pending USB requests that were never stopped.  This would
lead to a potential SMMU fault.
Wesley Cheng (5):
  usb: dwc3: Avoid unmapping USB requests if endxfer is not complete
  usb: dwc3: Remove DWC3 locking during gadget suspend/resume
  usb: dwc3: Increase DWC3 controller halt timeout
  usb: dwc3: gadget: Skip waiting for CMDACT cleared during endxfer
  usb: dwc3: gadget: Submit endxfer command if delayed during disconnect

 drivers/usb/dwc3/core.c   |  4 ----
 drivers/usb/dwc3/core.h   |  3 +++
 drivers/usb/dwc3/ep0.c    |  5 ++++-
 drivers/usb/dwc3/gadget.c | 31 +++++++++++++++++++++++++++----
 4 files changed, 34 insertions(+), 9 deletions(-)

[PATCH v5 1/5] usb: dwc3: Avoid unmapping USB requests if endxfer is not complete

[Wesley Cheng]

If DWC3_EP_DELAYED_STOP is set during stop active transfers, then do not
continue attempting to unmap request buffers during dwc3_remove_requests().
This can lead to SMMU faults, as the controller has not stopped the
processing of the TRB.  Defer this sequence to the EP0 out start, which
ensures that there are no pending SETUP transactions before issuing the
endxfer.

Reviewed-by: Thinh Nguyen <Thinh.Nguyen@synopsys.com>
Signed-off-by: Wesley Cheng <quic_wcheng@quicinc.com>
---
 drivers/usb/dwc3/core.h   | 3 +++
 drivers/usb/dwc3/ep0.c    | 5 ++++-
 drivers/usb/dwc3/gadget.c | 6 +++++-
 3 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/drivers/usb/dwc3/core.h b/drivers/usb/dwc3/core.h
index 4fe4287dc934..7327e5767df9 100644
--- a/drivers/usb/dwc3/core.h
+++ b/drivers/usb/dwc3/core.h
@@ -1560,6 +1560,7 @@ int dwc3_send_gadget_ep_cmd(struct dwc3_ep *dep, unsigned int cmd,
 int dwc3_send_gadget_generic_command(struct dwc3 *dwc, unsigned int cmd,
 		u32 param);
 void dwc3_gadget_clear_tx_fifos(struct dwc3 *dwc);
+void dwc3_remove_requests(struct dwc3 *dwc, struct dwc3_ep *dep);
 #else
 static inline int dwc3_gadget_init(struct dwc3 *dwc)
 { return 0; }
@@ -1581,6 +1582,8 @@ static inline int dwc3_send_gadget_generic_command(struct dwc3 *dwc,
 { return 0; }
 static inline void dwc3_gadget_clear_tx_fifos(struct dwc3 *dwc)
 { }
+static inline void dwc3_remove_requests(struct dwc3 *dwc, struct dwc3_ep *dep)
+{ }
 #endif
 
 #if IS_ENABLED(CONFIG_USB_DWC3_DUAL_ROLE)
diff --git a/drivers/usb/dwc3/ep0.c b/drivers/usb/dwc3/ep0.c
index 33cee0089609..fbb154a5ee1f 100644
--- a/drivers/usb/dwc3/ep0.c
+++ b/drivers/usb/dwc3/ep0.c
@@ -293,7 +293,10 @@ void dwc3_ep0_out_start(struct dwc3 *dwc)
 			continue;
 
 		dwc3_ep->flags &= ~DWC3_EP_DELAY_STOP;
-		dwc3_stop_active_transfer(dwc3_ep, true, true);
+		if (dwc->connected)
+			dwc3_stop_active_transfer(dwc3_ep, true, true);
+		else
+			dwc3_remove_requests(dwc, dwc3_ep);
 	}
 }
 
diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
index d6c0cb79ace3..6f2491fc109e 100644
--- a/drivers/usb/dwc3/gadget.c
+++ b/drivers/usb/dwc3/gadget.c
@@ -965,12 +965,16 @@ static int __dwc3_gadget_ep_enable(struct dwc3_ep *dep, unsigned int action)
 	return 0;
 }
 
-static void dwc3_remove_requests(struct dwc3 *dwc, struct dwc3_ep *dep, int status)
+void dwc3_remove_requests(struct dwc3 *dwc, struct dwc3_ep *dep, int status)
 {
 	struct dwc3_request		*req;
 
 	dwc3_stop_active_transfer(dep, true, false);
 
+	/* If endxfer is delayed, avoid unmapping requests */
+	if (dep->flags & DWC3_EP_DELAY_STOP)
+		return;
+
 	/* - giveback all requests to gadget driver */
 	while (!list_empty(&dep->started_list)) {
 		req = next_request(&dep->started_list);

Re: [PATCH v5 1/5] usb: dwc3: Avoid unmapping USB requests if endxfer is not complete

[kernel test robot]

Hi Wesley,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on usb/usb-testing]
[also build test ERROR on next-20220831]
[cannot apply to linus/master v6.0-rc3]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Wesley-Cheng/Fix-controller-halt-and-endxfer-timeout-issues/20220901-023750
base:   https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing
config: x86_64-randconfig-a012 (https://download.01.org/0day-ci/archive/20220901/202209011138.FP2kL0sl-lkp@intel.com/config)
compiler: clang version 14.0.6 (https://github.com/llvm/llvm-project f28c006a5895fc0e329fe15fead81e37457cb1d1)
reproduce (this is a W=1 build):
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # https://github.com/intel-lab-lkp/linux/commit/18bece9b81c07fbd2dbcec20ef8cc7e56d1ebf35
        git remote add linux-review https://github.com/intel-lab-lkp/linux
        git fetch --no-tags linux-review Wesley-Cheng/Fix-controller-halt-and-endxfer-timeout-issues/20220901-023750
        git checkout 18bece9b81c07fbd2dbcec20ef8cc7e56d1ebf35
        # save the config file
        mkdir build_dir && cp config build_dir/.config
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=x86_64 SHELL=/bin/bash drivers/usb/dwc3/

If you fix the issue, kindly add following tag where applicable
Reported-by: kernel test robot <lkp@intel.com>

All errors (new ones prefixed by >>):

>> drivers/usb/dwc3/gadget.c:968:6: error: conflicting types for 'dwc3_remove_requests'
   void dwc3_remove_requests(struct dwc3 *dwc, struct dwc3_ep *dep, int status)
        ^
   drivers/usb/dwc3/core.h:1563:6: note: previous declaration is here
   void dwc3_remove_requests(struct dwc3 *dwc, struct dwc3_ep *dep);
        ^
>> drivers/usb/dwc3/gadget.c:1029:33: error: too many arguments to function call, expected 2, have 3
           dwc3_remove_requests(dwc, dep, -ECONNRESET);
           ~~~~~~~~~~~~~~~~~~~~           ^~~~~~~~~~~
   drivers/usb/dwc3/core.h:1563:6: note: 'dwc3_remove_requests' declared here
   void dwc3_remove_requests(struct dwc3 *dwc, struct dwc3_ep *dep);
        ^
   drivers/usb/dwc3/gadget.c:2347:34: error: too many arguments to function call, expected 2, have 3
                   dwc3_remove_requests(dwc, dep, -ESHUTDOWN);
                   ~~~~~~~~~~~~~~~~~~~~           ^~~~~~~~~~
   drivers/usb/dwc3/core.h:1563:6: note: 'dwc3_remove_requests' declared here
   void dwc3_remove_requests(struct dwc3 *dwc, struct dwc3_ep *dep);
        ^
   3 errors generated.


vim +/dwc3_remove_requests +968 drivers/usb/dwc3/gadget.c

   967	
 > 968	void dwc3_remove_requests(struct dwc3 *dwc, struct dwc3_ep *dep, int status)
   969	{
   970		struct dwc3_request		*req;
   971	
   972		dwc3_stop_active_transfer(dep, true, false);
   973	
   974		/* If endxfer is delayed, avoid unmapping requests */
   975		if (dep->flags & DWC3_EP_DELAY_STOP)
   976			return;
   977	
   978		/* - giveback all requests to gadget driver */
   979		while (!list_empty(&dep->started_list)) {
   980			req = next_request(&dep->started_list);
   981	
   982			dwc3_gadget_giveback(dep, req, status);
   983		}
   984	
   985		while (!list_empty(&dep->pending_list)) {
   986			req = next_request(&dep->pending_list);
   987	
   988			dwc3_gadget_giveback(dep, req, status);
   989		}
   990	
   991		while (!list_empty(&dep->cancelled_list)) {
   992			req = next_request(&dep->cancelled_list);
   993	
   994			dwc3_gadget_giveback(dep, req, status);
   995		}
   996	}
   997	
   998	/**
   999	 * __dwc3_gadget_ep_disable - disables a hw endpoint
  1000	 * @dep: the endpoint to disable
  1001	 *
  1002	 * This function undoes what __dwc3_gadget_ep_enable did and also removes
  1003	 * requests which are currently being processed by the hardware and those which
  1004	 * are not yet scheduled.
  1005	 *
  1006	 * Caller should take care of locking.
  1007	 */
  1008	static int __dwc3_gadget_ep_disable(struct dwc3_ep *dep)
  1009	{
  1010		struct dwc3		*dwc = dep->dwc;
  1011		u32			reg;
  1012	
  1013		trace_dwc3_gadget_ep_disable(dep);
  1014	
  1015		/* make sure HW endpoint isn't stalled */
  1016		if (dep->flags & DWC3_EP_STALL)
  1017			__dwc3_gadget_ep_set_halt(dep, 0, false);
  1018	
  1019		reg = dwc3_readl(dwc->regs, DWC3_DALEPENA);
  1020		reg &= ~DWC3_DALEPENA_EP(dep->number);
  1021		dwc3_writel(dwc->regs, DWC3_DALEPENA, reg);
  1022	
  1023		/* Clear out the ep descriptors for non-ep0 */
  1024		if (dep->number > 1) {
  1025			dep->endpoint.comp_desc = NULL;
  1026			dep->endpoint.desc = NULL;
  1027		}
  1028	
> 1029		dwc3_remove_requests(dwc, dep, -ECONNRESET);
  1030	
  1031		dep->stream_capable = false;
  1032		dep->type = 0;
  1033		dep->flags &= DWC3_EP_TXFIFO_RESIZED;
  1034	
  1035		return 0;
  1036	}
  1037	

-- 
0-DAY CI Kernel Test Service
https://01.org/lkp

Re: [PATCH v5 1/5] usb: dwc3: Avoid unmapping USB requests if endxfer is not complete

[Greg KH]

On Wed, Aug 31, 2022 at 11:32:38AM -0700, Wesley Cheng wrote:
> If DWC3_EP_DELAYED_STOP is set during stop active transfers, then do not
> continue attempting to unmap request buffers during dwc3_remove_requests().
> This can lead to SMMU faults, as the controller has not stopped the
> processing of the TRB.  Defer this sequence to the EP0 out start, which
> ensures that there are no pending SETUP transactions before issuing the
> endxfer.
> 
> Reviewed-by: Thinh Nguyen <Thinh.Nguyen@synopsys.com>
> Signed-off-by: Wesley Cheng <quic_wcheng@quicinc.com>
> ---
>  drivers/usb/dwc3/core.h   | 3 +++
>  drivers/usb/dwc3/ep0.c    | 5 ++++-
>  drivers/usb/dwc3/gadget.c | 6 +++++-
>  3 files changed, 12 insertions(+), 2 deletions(-)

As the kernel-test robot reported, this patch doesn't even build.  How
did you test it?

greg k-h

Re: [PATCH v5 1/5] usb: dwc3: Avoid unmapping USB requests if endxfer is not complete

[Wesley Cheng]

On 9/1/2022 7:10 AM, Greg KH wrote:
> On Wed, Aug 31, 2022 at 11:32:38AM -0700, Wesley Cheng wrote:
>> If DWC3_EP_DELAYED_STOP is set during stop active transfers, then do not
>> continue attempting to unmap request buffers during dwc3_remove_requests().
>> This can lead to SMMU faults, as the controller has not stopped the
>> processing of the TRB.  Defer this sequence to the EP0 out start, which
>> ensures that there are no pending SETUP transactions before issuing the
>> endxfer.
>>
>> Reviewed-by: Thinh Nguyen <Thinh.Nguyen@synopsys.com>
>> Signed-off-by: Wesley Cheng <quic_wcheng@quicinc.com>
>> ---
>>   drivers/usb/dwc3/core.h   | 3 +++
>>   drivers/usb/dwc3/ep0.c    | 5 ++++-
>>   drivers/usb/dwc3/gadget.c | 6 +++++-
>>   3 files changed, 12 insertions(+), 2 deletions(-)
> 
> As the kernel-test robot reported, this patch doesn't even build.  How
> did you test it?
> 
Sorry Greg, didn't get a chance to test it before submitting, but fixed 
the build errors in v6.  I had initially missed the argument changes to 
the APIs.

V6 works as expected.

Thanks
Wesley Cheng

[PATCH v5 2/5] usb: dwc3: Remove DWC3 locking during gadget suspend/resume

[Wesley Cheng]

Remove the need for making dwc3_gadget_suspend() and dwc3_gadget_resume()
to be called in a spinlock, as dwc3_gadget_run_stop() could potentially
take some time to complete.

Reviewed-by: Thinh Nguyen <Thinh.Nguyen@synopsys.com>
Signed-off-by: Wesley Cheng <quic_wcheng@quicinc.com>
---
 drivers/usb/dwc3/core.c   | 4 ----
 drivers/usb/dwc3/gadget.c | 5 +++++
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/drivers/usb/dwc3/core.c b/drivers/usb/dwc3/core.c
index c5c238ab3083..23e123a1ab5f 100644
--- a/drivers/usb/dwc3/core.c
+++ b/drivers/usb/dwc3/core.c
@@ -1976,9 +1976,7 @@ static int dwc3_suspend_common(struct dwc3 *dwc, pm_message_t msg)
 	case DWC3_GCTL_PRTCAP_DEVICE:
 		if (pm_runtime_suspended(dwc->dev))
 			break;
-		spin_lock_irqsave(&dwc->lock, flags);
 		dwc3_gadget_suspend(dwc);
-		spin_unlock_irqrestore(&dwc->lock, flags);
 		synchronize_irq(dwc->irq_gadget);
 		dwc3_core_exit(dwc);
 		break;
@@ -2039,9 +2037,7 @@ static int dwc3_resume_common(struct dwc3 *dwc, pm_message_t msg)
 			return ret;
 
 		dwc3_set_prtcap(dwc, DWC3_GCTL_PRTCAP_DEVICE);
-		spin_lock_irqsave(&dwc->lock, flags);
 		dwc3_gadget_resume(dwc);
-		spin_unlock_irqrestore(&dwc->lock, flags);
 		break;
 	case DWC3_GCTL_PRTCAP_HOST:
 		if (!PMSG_IS_AUTO(msg) && !device_can_wakeup(dwc->dev)) {
diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
index 6f2491fc109e..08281c4f47e6 100644
--- a/drivers/usb/dwc3/gadget.c
+++ b/drivers/usb/dwc3/gadget.c
@@ -4520,12 +4520,17 @@ void dwc3_gadget_exit(struct dwc3 *dwc)
 
 int dwc3_gadget_suspend(struct dwc3 *dwc)
 {
+	unsigned long flags;
+
 	if (!dwc->gadget_driver)
 		return 0;
 
 	dwc3_gadget_run_stop(dwc, false, false);
+
+	spin_lock_irqsave(&dwc->lock, flags);
 	dwc3_disconnect_gadget(dwc);
 	__dwc3_gadget_stop(dwc);
+	spin_unlock_irqrestore(&dwc->lock, flags);
 
 	return 0;
 }

[PATCH v5 3/5] usb: dwc3: Increase DWC3 controller halt timeout

[Wesley Cheng]

Since EP0 transactions need to be completed before the controller halt
sequence is finished, this may take some time depending on the host and the
enabled functions.  Increase the controller halt timeout, so that we give
the controller sufficient time to handle EP0 transfers.

Reviewed-by: Thinh Nguyen <Thinh.Nguyen@synopsys.com>
Signed-off-by: Wesley Cheng <quic_wcheng@quicinc.com>
---
 drivers/usb/dwc3/gadget.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
index 08281c4f47e6..c9310645701d 100644
--- a/drivers/usb/dwc3/gadget.c
+++ b/drivers/usb/dwc3/gadget.c
@@ -2444,7 +2444,7 @@ static void __dwc3_gadget_set_speed(struct dwc3 *dwc)
 static int dwc3_gadget_run_stop(struct dwc3 *dwc, int is_on, int suspend)
 {
 	u32			reg;
-	u32			timeout = 500;
+	u32			timeout = 2000;
 
 	if (pm_runtime_suspended(dwc->dev))
 		return 0;
@@ -2477,6 +2477,7 @@ static int dwc3_gadget_run_stop(struct dwc3 *dwc, int is_on, int suspend)
 	dwc3_gadget_dctl_write_safe(dwc, reg);
 
 	do {
+		usleep_range(1000, 2000);
 		reg = dwc3_readl(dwc->regs, DWC3_DSTS);
 		reg &= DWC3_DSTS_DEVCTRLHLT;
 	} while (--timeout && !(!is_on ^ !reg));

[PATCH v5 4/5] usb: dwc3: gadget: Skip waiting for CMDACT cleared during endxfer

[Wesley Cheng]

For endxfer commands that do not require an endpoint complete interrupt,
avoid having to wait for the command active bit to clear.  This allows for
EP0 events to continue to be handled, which allows for the controller to
complete it.  Otherwise, it is known that the endxfer command will fail if
there is a pending SETUP token that needs to be read.

Suggested-by: Thinh Nguyen <Thinh.Nguyen@synopsys.com>
Reviewed-by: Thinh Nguyen <Thinh.Nguyen@synopsys.com>
Signed-off-by: Wesley Cheng <quic_wcheng@quicinc.com>
---
 drivers/usb/dwc3/gadget.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
index c9310645701d..4721561e4ba5 100644
--- a/drivers/usb/dwc3/gadget.c
+++ b/drivers/usb/dwc3/gadget.c
@@ -366,7 +366,9 @@ int dwc3_send_gadget_ep_cmd(struct dwc3_ep *dep, unsigned int cmd,
 
 	dwc3_writel(dep->regs, DWC3_DEPCMD, cmd);
 
-	if (!(cmd & DWC3_DEPCMD_CMDACT)) {
+	if (!(cmd & DWC3_DEPCMD_CMDACT) ||
+		(DWC3_DEPCMD_CMD(cmd) == DWC3_DEPCMD_ENDTRANSFER &&
+		!(cmd & DWC3_DEPCMD_CMDIOC))) {
 		ret = 0;
 		goto skip_status;
 	}

[PATCH v5 5/5] usb: dwc3: gadget: Submit endxfer command if delayed during disconnect

[Wesley Cheng]

During a cable disconnect sequence, if ep0state is not in the SETUP phase,
then nothing will trigger any pending end transfer commands.  Force
stopping of any pending SETUP transaction, and move back to the SETUP
phase.

Reviewed-by: Thinh Nguyen <Thinh.Nguyen@synopsys.com>
Signed-off-by: Wesley Cheng <quic_wcheng@quicinc.com>
---
 drivers/usb/dwc3/gadget.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
index 4721561e4ba5..515980bf61e2 100644
--- a/drivers/usb/dwc3/gadget.c
+++ b/drivers/usb/dwc3/gadget.c
@@ -3776,13 +3776,24 @@ static void dwc3_gadget_disconnect_interrupt(struct dwc3 *dwc)
 	reg &= ~DWC3_DCTL_INITU2ENA;
 	dwc3_gadget_dctl_write_safe(dwc, reg);
 
+	dwc->connected = false;
+
 	dwc3_disconnect_gadget(dwc);
 
 	dwc->gadget->speed = USB_SPEED_UNKNOWN;
 	dwc->setup_packet_pending = false;
 	usb_gadget_set_state(dwc->gadget, USB_STATE_NOTATTACHED);
 
-	dwc->connected = false;
+	if (dwc->ep0state != EP0_SETUP_PHASE) {
+		unsigned int    dir;
+
+		dir = !!dwc->ep0_expect_in;
+		if (dwc->ep0state == EP0_DATA_PHASE)
+			dwc3_ep0_end_control_data(dwc, dwc->eps[dir]);
+		else
+			dwc3_ep0_end_control_data(dwc, dwc->eps[!dir]);
+		dwc3_ep0_stall_and_restart(dwc);
+	}
 }
 
 static void dwc3_gadget_reset_interrupt(struct dwc3 *dwc)