linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
@ 2021-06-17  9:58 Wesley Cheng
  2021-06-17  9:58 ` [PATCH v10 1/6] usb: gadget: udc: core: Introduce check_config to verify USB configuration Wesley Cheng
                   ` (6 more replies)
  0 siblings, 7 replies; 73+ messages in thread
From: Wesley Cheng @ 2021-06-17  9:58 UTC (permalink / raw)
  To: balbi, gregkh, robh+dt, agross, bjorn.andersson, frowand.list
  Cc: linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	fntoth, heikki.krogerus, andy.shevchenko, Wesley Cheng

Changes in V10:
 - Fixed compilation errors in config where OF is not used (error due to
   unknown symbol for of_add_property()).  Add of_add_property() stub.
 - Fixed compilation warning for incorrect argument being passed to dwc3_mdwidth

Changes in V9:
 - Fixed incorrect patch in series.  Removed changes in DTSI, as dwc3-qcom will
   add the property by default from the kernel.

Changes in V8:
 - Rebased to usb-testing
 - Using devm_kzalloc for adding txfifo property in dwc3-qcom
 - Removed DWC3 QCOM ACPI property for enabling the txfifo resize

Changes in V7:
 - Added a new property tx-fifo-max-num for limiting how much fifo space the
   resizing logic can allocate for endpoints with large burst values.  This
   can differ across platforms, and tie in closely with overall system latency.
 - Added recommended checks for DWC32.
 - Added changes to set the tx-fifo-resize property from dwc3-qcom by default
   instead of modifying the current DTSI files.
 - Added comments on all APIs/variables introduced.
 - Updated the DWC3 YAML to include a better description of the tx-fifo-resize
   property and added an entry for tx-fifo-max-num.

Changes in V6:
 - Rebased patches to usb-testing.
 - Renamed to PATCH series instead of RFC.
 - Checking for fs_descriptors instead of ss_descriptors for determining the
   endpoint count for a particular configuration.
 - Re-ordered patch series to fix patch dependencies.

Changes in V5:
 - Added check_config() logic, which is used to communicate the number of EPs
   used in a particular configuration.  Based on this, the DWC3 gadget driver
   has the ability to know the maximum number of eps utilized in all configs.
   This helps reduce unnecessary allocation to unused eps, and will catch fifo
   allocation issues at bind() time.
 - Fixed variable declaration to single line per variable, and reverse xmas.
 - Created a helper for fifo clearing, which is used by ep0.c

Changes in V4:
 - Removed struct dwc3* as an argument for dwc3_gadget_resize_tx_fifos()
 - Removed WARN_ON(1) in case we run out of fifo space
 
Changes in V3:
 - Removed "Reviewed-by" tags
 - Renamed series back to RFC
 - Modified logic to ensure that fifo_size is reset if we pass the minimum
   threshold.  Tested with binding multiple FDs requesting 6 FIFOs.

Changes in V2:
 - Modified TXFIFO resizing logic to ensure that each EP is reserved a
   FIFO.
 - Removed dev_dbg() prints and fixed typos from patches
 - Added some more description on the dt-bindings commit message

Currently, there is no functionality to allow for resizing the TXFIFOs, and
relying on the HW default setting for the TXFIFO depth.  In most cases, the
HW default is probably sufficient, but for USB compositions that contain
multiple functions that require EP bursting, the default settings
might not be enough.  Also to note, the current SW will assign an EP to a
function driver w/o checking to see if the TXFIFO size for that particular
EP is large enough. (this is a problem if there are multiple HW defined
values for the TXFIFO size)

It is mentioned in the SNPS databook that a minimum of TX FIFO depth = 3
is required for an EP that supports bursting.  Otherwise, there may be
frequent occurences of bursts ending.  For high bandwidth functions,
such as data tethering (protocols that support data aggregation), mass
storage, and media transfer protocol (over FFS), the bMaxBurst value can be
large, and a bigger TXFIFO depth may prove to be beneficial in terms of USB
throughput. (which can be associated to system access latency, etc...)  It
allows for a more consistent burst of traffic, w/o any interruptions, as
data is readily available in the FIFO.

With testing done using the mass storage function driver, the results show
that with a larger TXFIFO depth, the bandwidth increased significantly.

Test Parameters:
 - Platform: Qualcomm SM8150
 - bMaxBurst = 6
 - USB req size = 256kB
 - Num of USB reqs = 16
 - USB Speed = Super-Speed
 - Function Driver: Mass Storage (w/ ramdisk)
 - Test Application: CrystalDiskMark

Results:

TXFIFO Depth = 3 max packets

Test Case | Data Size | AVG tput (in MB/s)
-------------------------------------------
Sequential|1 GB x     | 
Read      |9 loops    | 193.60
	  |           | 195.86
          |           | 184.77
          |           | 193.60
-------------------------------------------

TXFIFO Depth = 6 max packets

Test Case | Data Size | AVG tput (in MB/s)
-------------------------------------------
Sequential|1 GB x     | 
Read      |9 loops    | 287.35
	  |           | 304.94
          |           | 289.64
          |           | 293.61
-------------------------------------------

Wesley Cheng (6):
  usb: gadget: udc: core: Introduce check_config to verify USB
    configuration
  usb: gadget: configfs: Check USB configuration before adding
  usb: dwc3: Resize TX FIFOs to meet EP bursting requirements
  of: Add stub for of_add_property()
  usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by default
  dt-bindings: usb: dwc3: Update dwc3 TX fifo properties

 .../devicetree/bindings/usb/snps,dwc3.yaml         |  15 +-
 drivers/usb/dwc3/core.c                            |   9 +
 drivers/usb/dwc3/core.h                            |  15 ++
 drivers/usb/dwc3/dwc3-qcom.c                       |   9 +
 drivers/usb/dwc3/ep0.c                             |   2 +
 drivers/usb/dwc3/gadget.c                          | 212 +++++++++++++++++++++
 drivers/usb/gadget/configfs.c                      |  22 +++
 drivers/usb/gadget/udc/core.c                      |  25 +++
 include/linux/of.h                                 |   5 +
 include/linux/usb/gadget.h                         |   5 +
 10 files changed, 317 insertions(+), 2 deletions(-)

-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project


^ permalink raw reply	[flat|nested] 73+ messages in thread

* [PATCH v10 1/6] usb: gadget: udc: core: Introduce check_config to verify USB configuration
  2021-06-17  9:58 [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting Wesley Cheng
@ 2021-06-17  9:58 ` Wesley Cheng
  2021-06-17 11:09   ` Greg KH
  2021-06-17  9:58 ` [PATCH v10 2/6] usb: gadget: configfs: Check USB configuration before adding Wesley Cheng
                   ` (5 subsequent siblings)
  6 siblings, 1 reply; 73+ messages in thread
From: Wesley Cheng @ 2021-06-17  9:58 UTC (permalink / raw)
  To: balbi, gregkh, robh+dt, agross, bjorn.andersson, frowand.list
  Cc: linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	fntoth, heikki.krogerus, andy.shevchenko, Wesley Cheng

Some UDCs may have constraints on how many high bandwidth endpoints it can
support in a certain configuration.  This API allows for the composite
driver to pass down the total number of endpoints to the UDC so it can verify
it has the required resources to support the configuration.

Signed-off-by: Wesley Cheng <wcheng@codeaurora.org>
---
 drivers/usb/gadget/udc/core.c | 25 +++++++++++++++++++++++++
 include/linux/usb/gadget.h    |  5 +++++
 2 files changed, 30 insertions(+)

diff --git a/drivers/usb/gadget/udc/core.c b/drivers/usb/gadget/udc/core.c
index b7f0b1e..e33ae2d 100644
--- a/drivers/usb/gadget/udc/core.c
+++ b/drivers/usb/gadget/udc/core.c
@@ -1003,6 +1003,31 @@ int usb_gadget_ep_match_desc(struct usb_gadget *gadget,
 }
 EXPORT_SYMBOL_GPL(usb_gadget_ep_match_desc);
 
+/**
+ * usb_gadget_check_config - checks if the UDC can support the number of eps
+ * @gadget: controller to check the USB configuration
+ * @ep_map: bitmap of endpoints being requested by a USB configuration
+ *
+ * Ensure that a UDC is able to support the number of endpoints within a USB
+ * configuration, and that there are no resource limitations to support all
+ * requested eps.
+ *
+ * Returns zero on success, else a negative errno.
+ */
+int usb_gadget_check_config(struct usb_gadget *gadget, unsigned long ep_map)
+{
+	int ret = 0;
+
+	if (!gadget->ops->check_config)
+		goto out;
+
+	ret = gadget->ops->check_config(gadget, ep_map);
+
+out:
+	return ret;
+}
+EXPORT_SYMBOL_GPL(usb_gadget_check_config);
+
 /* ------------------------------------------------------------------------- */
 
 static void usb_gadget_state_work(struct work_struct *work)
diff --git a/include/linux/usb/gadget.h b/include/linux/usb/gadget.h
index 75c7538..fcde3b3 100644
--- a/include/linux/usb/gadget.h
+++ b/include/linux/usb/gadget.h
@@ -329,6 +329,7 @@ struct usb_gadget_ops {
 	struct usb_ep *(*match_ep)(struct usb_gadget *,
 			struct usb_endpoint_descriptor *,
 			struct usb_ss_ep_comp_descriptor *);
+	int	(*check_config)(struct usb_gadget *gadget, unsigned long ep_map);
 };
 
 /**
@@ -608,6 +609,7 @@ int usb_gadget_connect(struct usb_gadget *gadget);
 int usb_gadget_disconnect(struct usb_gadget *gadget);
 int usb_gadget_deactivate(struct usb_gadget *gadget);
 int usb_gadget_activate(struct usb_gadget *gadget);
+int usb_gadget_check_config(struct usb_gadget *gadget, unsigned long ep_map);
 #else
 static inline int usb_gadget_frame_number(struct usb_gadget *gadget)
 { return 0; }
@@ -631,6 +633,9 @@ static inline int usb_gadget_deactivate(struct usb_gadget *gadget)
 { return 0; }
 static inline int usb_gadget_activate(struct usb_gadget *gadget)
 { return 0; }
+static inline int usb_gadget_check_config(struct usb_gadget *gadget,
+					  unsigned long ep_map)
+{ return 0; }
 #endif /* CONFIG_USB_GADGET */
 
 /*-------------------------------------------------------------------------*/
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project


^ permalink raw reply	[flat|nested] 73+ messages in thread

* [PATCH v10 2/6] usb: gadget: configfs: Check USB configuration before adding
  2021-06-17  9:58 [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting Wesley Cheng
  2021-06-17  9:58 ` [PATCH v10 1/6] usb: gadget: udc: core: Introduce check_config to verify USB configuration Wesley Cheng
@ 2021-06-17  9:58 ` Wesley Cheng
  2021-06-17 11:07   ` Greg KH
  2021-06-17  9:58 ` [PATCH v10 3/6] usb: dwc3: Resize TX FIFOs to meet EP bursting requirements Wesley Cheng
                   ` (4 subsequent siblings)
  6 siblings, 1 reply; 73+ messages in thread
From: Wesley Cheng @ 2021-06-17  9:58 UTC (permalink / raw)
  To: balbi, gregkh, robh+dt, agross, bjorn.andersson, frowand.list
  Cc: linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	fntoth, heikki.krogerus, andy.shevchenko, Wesley Cheng

Ensure that the USB gadget is able to support the configuration being
added based on the number of endpoints required from all interfaces.  This
is for accounting for any bandwidth or space limitations.

Signed-off-by: Wesley Cheng <wcheng@codeaurora.org>
---
 drivers/usb/gadget/configfs.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/drivers/usb/gadget/configfs.c b/drivers/usb/gadget/configfs.c
index 15a607c..76b9983 100644
--- a/drivers/usb/gadget/configfs.c
+++ b/drivers/usb/gadget/configfs.c
@@ -1374,6 +1374,7 @@ static int configfs_composite_bind(struct usb_gadget *gadget,
 		struct usb_function *f;
 		struct usb_function *tmp;
 		struct gadget_config_name *cn;
+		unsigned long ep_map = 0;
 
 		if (gadget_is_otg(gadget))
 			c->descriptors = otg_desc;
@@ -1403,7 +1404,28 @@ static int configfs_composite_bind(struct usb_gadget *gadget,
 				list_add(&f->list, &cfg->func_list);
 				goto err_purge_funcs;
 			}
+			if (f->fs_descriptors) {
+				struct usb_descriptor_header **d;
+
+				d = f->fs_descriptors;
+				for (; *d; ++d) {
+					struct usb_endpoint_descriptor *ep;
+					int addr;
+
+					if ((*d)->bDescriptorType != USB_DT_ENDPOINT)
+						continue;
+
+					ep = (struct usb_endpoint_descriptor *)*d;
+					addr = ((ep->bEndpointAddress & 0x80) >> 3) |
+						(ep->bEndpointAddress & 0x0f);
+					set_bit(addr, &ep_map);
+				}
+			}
 		}
+		ret = usb_gadget_check_config(cdev->gadget, ep_map);
+		if (ret)
+			goto err_purge_funcs;
+
 		usb_ep_autoconfig_reset(cdev->gadget);
 	}
 	if (cdev->use_os_string) {
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project


^ permalink raw reply	[flat|nested] 73+ messages in thread

* [PATCH v10 3/6] usb: dwc3: Resize TX FIFOs to meet EP bursting requirements
  2021-06-17  9:58 [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting Wesley Cheng
  2021-06-17  9:58 ` [PATCH v10 1/6] usb: gadget: udc: core: Introduce check_config to verify USB configuration Wesley Cheng
  2021-06-17  9:58 ` [PATCH v10 2/6] usb: gadget: configfs: Check USB configuration before adding Wesley Cheng
@ 2021-06-17  9:58 ` Wesley Cheng
  2021-06-17 11:10   ` Greg KH
                     ` (2 more replies)
  2021-06-17  9:58 ` [PATCH v10 4/6] of: Add stub for of_add_property() Wesley Cheng
                   ` (3 subsequent siblings)
  6 siblings, 3 replies; 73+ messages in thread
From: Wesley Cheng @ 2021-06-17  9:58 UTC (permalink / raw)
  To: balbi, gregkh, robh+dt, agross, bjorn.andersson, frowand.list
  Cc: linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	fntoth, heikki.krogerus, andy.shevchenko, Wesley Cheng

Some devices have USB compositions which may require multiple endpoints
that support EP bursting.  HW defined TX FIFO sizes may not always be
sufficient for these compositions.  By utilizing flexible TX FIFO
allocation, this allows for endpoints to request the required FIFO depth to
achieve higher bandwidth.  With some higher bMaxBurst configurations, using
a larger TX FIFO size results in better TX throughput.

By introducing the check_config() callback, the resizing logic can fetch
the maximum number of endpoints used in the USB composition (can contain
multiple configurations), which helps ensure that the resizing logic can
fulfill the configuration(s), or return an error to the gadget layer
otherwise during bind time.

Signed-off-by: Wesley Cheng <wcheng@codeaurora.org>
---
 drivers/usb/dwc3/core.c   |   9 ++
 drivers/usb/dwc3/core.h   |  15 ++++
 drivers/usb/dwc3/ep0.c    |   2 +
 drivers/usb/dwc3/gadget.c | 212 ++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 238 insertions(+)

diff --git a/drivers/usb/dwc3/core.c b/drivers/usb/dwc3/core.c
index e0a8e79..a7bcdb9d 100644
--- a/drivers/usb/dwc3/core.c
+++ b/drivers/usb/dwc3/core.c
@@ -1267,6 +1267,7 @@ static void dwc3_get_properties(struct dwc3 *dwc)
 	u8			rx_max_burst_prd;
 	u8			tx_thr_num_pkt_prd;
 	u8			tx_max_burst_prd;
+	u8			tx_fifo_resize_max_num;
 	const char		*usb_psy_name;
 	int			ret;
 
@@ -1282,6 +1283,8 @@ static void dwc3_get_properties(struct dwc3 *dwc)
 	 */
 	hird_threshold = 12;
 
+	tx_fifo_resize_max_num = 6;
+
 	dwc->maximum_speed = usb_get_maximum_speed(dev);
 	dwc->max_ssp_rate = usb_get_maximum_ssp_rate(dev);
 	dwc->dr_mode = usb_get_dr_mode(dev);
@@ -1325,6 +1328,10 @@ static void dwc3_get_properties(struct dwc3 *dwc)
 				&tx_thr_num_pkt_prd);
 	device_property_read_u8(dev, "snps,tx-max-burst-prd",
 				&tx_max_burst_prd);
+	dwc->do_fifo_resize = device_property_read_bool(dev,
+							"tx-fifo-resize");
+	device_property_read_u8(dev, "tx-fifo-max-num",
+				&tx_fifo_resize_max_num);
 
 	dwc->disable_scramble_quirk = device_property_read_bool(dev,
 				"snps,disable_scramble_quirk");
@@ -1390,6 +1397,8 @@ static void dwc3_get_properties(struct dwc3 *dwc)
 	dwc->tx_max_burst_prd = tx_max_burst_prd;
 
 	dwc->imod_interval = 0;
+
+	dwc->tx_fifo_resize_max_num = tx_fifo_resize_max_num;
 }
 
 /* check whether the core supports IMOD */
diff --git a/drivers/usb/dwc3/core.h b/drivers/usb/dwc3/core.h
index dccdf13..dd985ba 100644
--- a/drivers/usb/dwc3/core.h
+++ b/drivers/usb/dwc3/core.h
@@ -1023,6 +1023,7 @@ struct dwc3_scratchpad_array {
  * @rx_max_burst_prd: max periodic ESS receive burst size
  * @tx_thr_num_pkt_prd: periodic ESS transmit packet count
  * @tx_max_burst_prd: max periodic ESS transmit burst size
+ * @tx_fifo_resize_max_num: max number of fifos allocated during txfifo resize
  * @hsphy_interface: "utmi" or "ulpi"
  * @connected: true when we're connected to a host, false otherwise
  * @delayed_status: true when gadget driver asks for delayed status
@@ -1079,6 +1080,11 @@ struct dwc3_scratchpad_array {
  * @dis_split_quirk: set to disable split boundary.
  * @imod_interval: set the interrupt moderation interval in 250ns
  *			increments or 0 to disable.
+ * @max_cfg_eps: current max number of IN eps used across all USB configs.
+ * @last_fifo_depth: last fifo depth used to determine next fifo ram start
+ *		     address.
+ * @num_ep_resized: carries the current number endpoints which have had its tx
+ *		    fifo resized.
  */
 struct dwc3 {
 	struct work_struct	drd_work;
@@ -1233,6 +1239,7 @@ struct dwc3 {
 	u8			rx_max_burst_prd;
 	u8			tx_thr_num_pkt_prd;
 	u8			tx_max_burst_prd;
+	u8			tx_fifo_resize_max_num;
 
 	const char		*hsphy_interface;
 
@@ -1246,6 +1253,7 @@ struct dwc3 {
 	unsigned		is_utmi_l1_suspend:1;
 	unsigned		is_fpga:1;
 	unsigned		pending_events:1;
+	unsigned		do_fifo_resize:1;
 	unsigned		pullups_connected:1;
 	unsigned		setup_packet_pending:1;
 	unsigned		three_stage_setup:1;
@@ -1281,6 +1289,10 @@ struct dwc3 {
 	unsigned		dis_split_quirk:1;
 
 	u16			imod_interval;
+
+	int			max_cfg_eps;
+	int			last_fifo_depth;
+	int			num_ep_resized;
 };
 
 #define INCRX_BURST_MODE 0
@@ -1512,6 +1524,7 @@ int dwc3_send_gadget_ep_cmd(struct dwc3_ep *dep, unsigned int cmd,
 		struct dwc3_gadget_ep_cmd_params *params);
 int dwc3_send_gadget_generic_command(struct dwc3 *dwc, unsigned int cmd,
 		u32 param);
+void dwc3_gadget_clear_tx_fifos(struct dwc3 *dwc);
 #else
 static inline int dwc3_gadget_init(struct dwc3 *dwc)
 { return 0; }
@@ -1531,6 +1544,8 @@ static inline int dwc3_send_gadget_ep_cmd(struct dwc3_ep *dep, unsigned int cmd,
 static inline int dwc3_send_gadget_generic_command(struct dwc3 *dwc,
 		int cmd, u32 param)
 { return 0; }
+static inline void dwc3_gadget_clear_tx_fifos(struct dwc3 *dwc)
+{ }
 #endif
 
 #if IS_ENABLED(CONFIG_USB_DWC3_DUAL_ROLE)
diff --git a/drivers/usb/dwc3/ep0.c b/drivers/usb/dwc3/ep0.c
index 3cd2942..d28d085 100644
--- a/drivers/usb/dwc3/ep0.c
+++ b/drivers/usb/dwc3/ep0.c
@@ -619,6 +619,8 @@ static int dwc3_ep0_set_config(struct dwc3 *dwc, struct usb_ctrlrequest *ctrl)
 		return -EINVAL;
 
 	case USB_STATE_ADDRESS:
+		dwc3_gadget_clear_tx_fifos(dwc);
+
 		ret = dwc3_ep0_delegate_req(dwc, ctrl);
 		/* if the cfg matches and the cfg is non zero */
 		if (cfg && (!ret || (ret == USB_GADGET_DELAYED_STATUS))) {
diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
index af6d7f1..8855481a 100644
--- a/drivers/usb/dwc3/gadget.c
+++ b/drivers/usb/dwc3/gadget.c
@@ -632,6 +632,179 @@ static void dwc3_stop_active_transfer(struct dwc3_ep *dep, bool force,
 		bool interrupt);
 
 /**
+ * dwc3_gadget_calc_tx_fifo_size - calculates the txfifo size value
+ * @dwc: pointer to the DWC3 context
+ * @nfifos: number of fifos to calculate for
+ *
+ * Calculates the size value based on the equation below:
+ *
+ * fifo_size = mult * ((max_packet + mdwidth)/mdwidth + 1) + 1
+ *
+ * The max packet size is set to 1024, as the txfifo requirements mainly apply
+ * to super speed USB use cases.  However, it is safe to overestimate the fifo
+ * allocations for other scenarios, i.e. high speed USB.
+ */
+static int dwc3_gadget_calc_tx_fifo_size(struct dwc3 *dwc, int mult)
+{
+	int max_packet = 1024;
+	int fifo_size;
+	int mdwidth;
+
+	mdwidth = dwc3_mdwidth(dwc);
+
+	/* MDWIDTH is represented in bits, we need it in bytes */
+	mdwidth >>= 3;
+
+	fifo_size = mult * ((max_packet + mdwidth) / mdwidth) + 1;
+	return fifo_size;
+}
+
+/**
+ * dwc3_gadget_clear_tx_fifo_size - Clears txfifo allocation
+ * @dwc: pointer to the DWC3 context
+ *
+ * Iterates through all the endpoint registers and clears the previous txfifo
+ * allocations.
+ */
+void dwc3_gadget_clear_tx_fifos(struct dwc3 *dwc)
+{
+	struct dwc3_ep *dep;
+	int fifo_depth;
+	int size;
+	int num;
+
+	if (!dwc->do_fifo_resize)
+		return;
+
+	/* Read ep0IN related TXFIFO size */
+	dep = dwc->eps[1];
+	size = dwc3_readl(dwc->regs, DWC3_GTXFIFOSIZ(0));
+	if (DWC3_IP_IS(DWC3))
+		fifo_depth = DWC3_GTXFIFOSIZ_TXFDEP(size);
+	else
+		fifo_depth = DWC31_GTXFIFOSIZ_TXFDEP(size);
+
+	dwc->last_fifo_depth = fifo_depth;
+	/* Clear existing TXFIFO for all IN eps except ep0 */
+	for (num = 3; num < min_t(int, dwc->num_eps, DWC3_ENDPOINTS_NUM);
+	     num += 2) {
+		dep = dwc->eps[num];
+		/* Don't change TXFRAMNUM on usb31 version */
+		size = DWC3_IP_IS(DWC3) ? 0 :
+			dwc3_readl(dwc->regs, DWC3_GTXFIFOSIZ(num >> 1)) &
+				   DWC31_GTXFIFOSIZ_TXFRAMNUM;
+
+		dwc3_writel(dwc->regs, DWC3_GTXFIFOSIZ(num >> 1), size);
+	}
+	dwc->num_ep_resized = 0;
+}
+
+/*
+ * dwc3_gadget_resize_tx_fifos - reallocate fifo spaces for current use-case
+ * @dwc: pointer to our context structure
+ *
+ * This function will a best effort FIFO allocation in order
+ * to improve FIFO usage and throughput, while still allowing
+ * us to enable as many endpoints as possible.
+ *
+ * Keep in mind that this operation will be highly dependent
+ * on the configured size for RAM1 - which contains TxFifo -,
+ * the amount of endpoints enabled on coreConsultant tool, and
+ * the width of the Master Bus.
+ *
+ * In general, FIFO depths are represented with the following equation:
+ *
+ * fifo_size = mult * ((max_packet + mdwidth)/mdwidth + 1) + 1
+ *
+ * Conversions can be done to the equation to derive the number of packets that
+ * will fit to a particular FIFO size value.
+ */
+static int dwc3_gadget_resize_tx_fifos(struct dwc3_ep *dep)
+{
+	struct dwc3 *dwc = dep->dwc;
+	int fifo_0_start;
+	int ram1_depth;
+	int fifo_size;
+	int min_depth;
+	int num_in_ep;
+	int remaining;
+	int num_fifos = 1;
+	int fifo;
+	int tmp;
+
+	if (!dwc->do_fifo_resize)
+		return 0;
+
+	/* resize IN endpoints except ep0 */
+	if (!usb_endpoint_dir_in(dep->endpoint.desc) || dep->number <= 1)
+		return 0;
+
+	ram1_depth = DWC3_RAM1_DEPTH(dwc->hwparams.hwparams7);
+
+	if ((dep->endpoint.maxburst > 1 &&
+	     usb_endpoint_xfer_bulk(dep->endpoint.desc)) ||
+	    usb_endpoint_xfer_isoc(dep->endpoint.desc))
+		num_fifos = 3;
+
+	if (dep->endpoint.maxburst > 6 &&
+	    usb_endpoint_xfer_bulk(dep->endpoint.desc) && DWC3_IP_IS(DWC31))
+		num_fifos = dwc->tx_fifo_resize_max_num;
+
+	/* FIFO size for a single buffer */
+	fifo = dwc3_gadget_calc_tx_fifo_size(dwc, 1);
+
+	/* Calculate the number of remaining EPs w/o any FIFO */
+	num_in_ep = dwc->max_cfg_eps;
+	num_in_ep -= dwc->num_ep_resized;
+
+	/* Reserve at least one FIFO for the number of IN EPs */
+	min_depth = num_in_ep * (fifo + 1);
+	remaining = ram1_depth - min_depth - dwc->last_fifo_depth;
+	remaining = max_t(int, 0, remaining);
+	/*
+	 * We've already reserved 1 FIFO per EP, so check what we can fit in
+	 * addition to it.  If there is not enough remaining space, allocate
+	 * all the remaining space to the EP.
+	 */
+	fifo_size = (num_fifos - 1) * fifo;
+	if (remaining < fifo_size)
+		fifo_size = remaining;
+
+	fifo_size += fifo;
+	/* Last increment according to the TX FIFO size equation */
+	fifo_size++;
+
+	/* Check if TXFIFOs start at non-zero addr */
+	tmp = dwc3_readl(dwc->regs, DWC3_GTXFIFOSIZ(0));
+	fifo_0_start = DWC3_GTXFIFOSIZ_TXFSTADDR(tmp);
+
+	fifo_size |= (fifo_0_start + (dwc->last_fifo_depth << 16));
+	if (DWC3_IP_IS(DWC3))
+		dwc->last_fifo_depth += DWC3_GTXFIFOSIZ_TXFDEP(fifo_size);
+	else
+		dwc->last_fifo_depth += DWC31_GTXFIFOSIZ_TXFDEP(fifo_size);
+
+	/* Check fifo size allocation doesn't exceed available RAM size. */
+	if (dwc->last_fifo_depth >= ram1_depth) {
+		dev_err(dwc->dev, "Fifosize(%d) > RAM size(%d) %s depth:%d\n",
+			dwc->last_fifo_depth, ram1_depth,
+			dep->endpoint.name, fifo_size);
+		if (DWC3_IP_IS(DWC3))
+			fifo_size = DWC3_GTXFIFOSIZ_TXFDEP(fifo_size);
+		else
+			fifo_size = DWC31_GTXFIFOSIZ_TXFDEP(fifo_size);
+
+		dwc->last_fifo_depth -= fifo_size;
+		return -ENOMEM;
+	}
+
+	dwc3_writel(dwc->regs, DWC3_GTXFIFOSIZ(dep->number >> 1), fifo_size);
+	dwc->num_ep_resized++;
+
+	return 0;
+}
+
+/**
  * __dwc3_gadget_ep_enable - initializes a hw endpoint
  * @dep: endpoint to be initialized
  * @action: one of INIT, MODIFY or RESTORE
@@ -648,6 +821,10 @@ static int __dwc3_gadget_ep_enable(struct dwc3_ep *dep, unsigned int action)
 	int			ret;
 
 	if (!(dep->flags & DWC3_EP_ENABLED)) {
+		ret = dwc3_gadget_resize_tx_fifos(dep);
+		if (ret)
+			return ret;
+
 		ret = dwc3_gadget_start_config(dep);
 		if (ret)
 			return ret;
@@ -2498,6 +2675,7 @@ static int dwc3_gadget_stop(struct usb_gadget *g)
 
 	spin_lock_irqsave(&dwc->lock, flags);
 	dwc->gadget_driver	= NULL;
+	dwc->max_cfg_eps = 0;
 	spin_unlock_irqrestore(&dwc->lock, flags);
 
 	free_irq(dwc->irq_gadget, dwc->ev_buf);
@@ -2585,6 +2763,39 @@ static int dwc3_gadget_vbus_draw(struct usb_gadget *g, unsigned int mA)
 	return ret;
 }
 
+static int dwc3_gadget_check_config(struct usb_gadget *g, unsigned long ep_map)
+{
+	struct dwc3 *dwc = gadget_to_dwc(g);
+	unsigned long in_ep_map;
+	int fifo_size = 0;
+	int ram1_depth;
+	int ep_num;
+
+	if (!dwc->do_fifo_resize)
+		return 0;
+
+	/* Only interested in the IN endpoints */
+	in_ep_map = ep_map >> 16;
+	ep_num = hweight_long(in_ep_map);
+
+	if (ep_num <= dwc->max_cfg_eps)
+		return 0;
+
+	/* Update the max number of eps in the composition */
+	dwc->max_cfg_eps = ep_num;
+
+	fifo_size = dwc3_gadget_calc_tx_fifo_size(dwc, dwc->max_cfg_eps);
+	/* Based on the equation, increment by one for every ep */
+	fifo_size += dwc->max_cfg_eps;
+
+	/* Check if we can fit a single fifo per endpoint */
+	ram1_depth = DWC3_RAM1_DEPTH(dwc->hwparams.hwparams7);
+	if (fifo_size > ram1_depth)
+		return -ENOMEM;
+
+	return 0;
+}
+
 static const struct usb_gadget_ops dwc3_gadget_ops = {
 	.get_frame		= dwc3_gadget_get_frame,
 	.wakeup			= dwc3_gadget_wakeup,
@@ -2596,6 +2807,7 @@ static const struct usb_gadget_ops dwc3_gadget_ops = {
 	.udc_set_ssp_rate	= dwc3_gadget_set_ssp_rate,
 	.get_config_params	= dwc3_gadget_config_params,
 	.vbus_draw		= dwc3_gadget_vbus_draw,
+	.check_config		= dwc3_gadget_check_config,
 };
 
 /* -------------------------------------------------------------------------- */
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project


^ permalink raw reply	[flat|nested] 73+ messages in thread

* [PATCH v10 4/6] of: Add stub for of_add_property()
  2021-06-17  9:58 [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting Wesley Cheng
                   ` (2 preceding siblings ...)
  2021-06-17  9:58 ` [PATCH v10 3/6] usb: dwc3: Resize TX FIFOs to meet EP bursting requirements Wesley Cheng
@ 2021-06-17  9:58 ` Wesley Cheng
  2021-06-17  9:58 ` [PATCH v10 5/6] usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by default Wesley Cheng
                   ` (2 subsequent siblings)
  6 siblings, 0 replies; 73+ messages in thread
From: Wesley Cheng @ 2021-06-17  9:58 UTC (permalink / raw)
  To: balbi, gregkh, robh+dt, agross, bjorn.andersson, frowand.list
  Cc: linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	fntoth, heikki.krogerus, andy.shevchenko, Wesley Cheng

If building with OF Kconfig disabled, this can lead to errors for
drivers utilizing of_add_property().  Add a stub for the add API, as
it exists for the remove variant as well, and to avoid compliation
issues.

Signed-off-by: Wesley Cheng <wcheng@codeaurora.org>
---
 include/linux/of.h | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/include/linux/of.h b/include/linux/of.h
index d8db8d3..abd39a4 100644
--- a/include/linux/of.h
+++ b/include/linux/of.h
@@ -946,6 +946,11 @@ static inline int of_machine_is_compatible(const char *compat)
 	return 0;
 }
 
+static inline int of_add_property(struct device_node *np, struct property *prop)
+{
+	return 0;
+}
+
 static inline int of_remove_property(struct device_node *np, struct property *prop)
 {
 	return 0;
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project


^ permalink raw reply	[flat|nested] 73+ messages in thread

* [PATCH v10 5/6] usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by default
  2021-06-17  9:58 [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting Wesley Cheng
                   ` (3 preceding siblings ...)
  2021-06-17  9:58 ` [PATCH v10 4/6] of: Add stub for of_add_property() Wesley Cheng
@ 2021-06-17  9:58 ` Wesley Cheng
  2021-06-17 11:12   ` Greg KH
  2021-06-17 18:04   ` kernel test robot
  2021-06-17  9:58 ` [PATCH v10 6/6] dt-bindings: usb: dwc3: Update dwc3 TX fifo properties Wesley Cheng
  2021-06-17 21:01 ` [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting Ferry Toth
  6 siblings, 2 replies; 73+ messages in thread
From: Wesley Cheng @ 2021-06-17  9:58 UTC (permalink / raw)
  To: balbi, gregkh, robh+dt, agross, bjorn.andersson, frowand.list
  Cc: linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	fntoth, heikki.krogerus, andy.shevchenko, Wesley Cheng

In order to take advantage of the TX fifo resizing logic, manually add
these properties to the DWC3 child node by default.  This will allow
the DWC3 gadget to resize the TX fifos for the IN endpoints, which
help with performance.

Signed-off-by: Wesley Cheng <wcheng@codeaurora.org>
---
 drivers/usb/dwc3/dwc3-qcom.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/usb/dwc3/dwc3-qcom.c b/drivers/usb/dwc3/dwc3-qcom.c
index 49e6ca9..44e0eaa1 100644
--- a/drivers/usb/dwc3/dwc3-qcom.c
+++ b/drivers/usb/dwc3/dwc3-qcom.c
@@ -645,6 +645,7 @@ static int dwc3_qcom_of_register_core(struct platform_device *pdev)
 	struct dwc3_qcom	*qcom = platform_get_drvdata(pdev);
 	struct device_node	*np = pdev->dev.of_node, *dwc3_np;
 	struct device		*dev = &pdev->dev;
+	struct property		*prop;
 	int			ret;
 
 	dwc3_np = of_get_compatible_child(np, "snps,dwc3");
@@ -653,6 +654,14 @@ static int dwc3_qcom_of_register_core(struct platform_device *pdev)
 		return -ENODEV;
 	}
 
+	prop = devm_kzalloc(dev, sizeof(*prop), GFP_KERNEL);
+	if (prop) {
+		prop->name = "tx-fifo-resize";
+		ret = of_add_property(dwc3_np, prop);
+		if (ret < 0)
+			dev_info(dev, "unable to add tx-fifo-resize prop\n");
+	}
+
 	ret = of_platform_populate(np, NULL, NULL, dev);
 	if (ret) {
 		dev_err(dev, "failed to register dwc3 core - %d\n", ret);
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project


^ permalink raw reply	[flat|nested] 73+ messages in thread

* [PATCH v10 6/6] dt-bindings: usb: dwc3: Update dwc3 TX fifo properties
  2021-06-17  9:58 [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting Wesley Cheng
                   ` (4 preceding siblings ...)
  2021-06-17  9:58 ` [PATCH v10 5/6] usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by default Wesley Cheng
@ 2021-06-17  9:58 ` Wesley Cheng
  2021-06-17 21:01 ` [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting Ferry Toth
  6 siblings, 0 replies; 73+ messages in thread
From: Wesley Cheng @ 2021-06-17  9:58 UTC (permalink / raw)
  To: balbi, gregkh, robh+dt, agross, bjorn.andersson, frowand.list
  Cc: linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	fntoth, heikki.krogerus, andy.shevchenko, Wesley Cheng

Update the tx-fifo-resize property with a better description, while
adding the tx-fifo-max-num, which is a new parameter allowing
adjustments for the maximum number of packets the txfifo resizing logic
can account for while resizing the endpoints.

Reviewed-by: Rob Herring <robh@kernel.org>
Signed-off-by: Wesley Cheng <wcheng@codeaurora.org>
---
 Documentation/devicetree/bindings/usb/snps,dwc3.yaml | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/Documentation/devicetree/bindings/usb/snps,dwc3.yaml b/Documentation/devicetree/bindings/usb/snps,dwc3.yaml
index 41416fb..078fb78 100644
--- a/Documentation/devicetree/bindings/usb/snps,dwc3.yaml
+++ b/Documentation/devicetree/bindings/usb/snps,dwc3.yaml
@@ -289,10 +289,21 @@ properties:
     maximum: 16
 
   tx-fifo-resize:
-    description: Determines if the FIFO *has* to be reallocated
-    deprecated: true
+    description: Determines if the TX fifos can be dynamically resized depending
+      on the number of IN endpoints used and if bursting is supported.  This
+      may help improve bandwidth on platforms with higher system latencies, as
+      increased fifo space allows for the controller to prefetch data into its
+      internal memory.
     type: boolean
 
+  tx-fifo-max-num:
+    description: Specifies the max number of packets the txfifo resizing logic
+      can account for when higher endpoint bursting is used. (bMaxBurst > 6) The
+      higher the number, the more fifo space the txfifo resizing logic will
+      allocate for that endpoint.
+    $ref: /schemas/types.yaml#/definitions/uint8
+    minimum: 3
+
   snps,incr-burst-type-adjustment:
     description:
       Value for INCR burst type of GSBUSCFG0 register, undefined length INCR
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project


^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 2/6] usb: gadget: configfs: Check USB configuration before adding
  2021-06-17  9:58 ` [PATCH v10 2/6] usb: gadget: configfs: Check USB configuration before adding Wesley Cheng
@ 2021-06-17 11:07   ` Greg KH
  2021-06-22  5:27     ` Wesley Cheng
  0 siblings, 1 reply; 73+ messages in thread
From: Greg KH @ 2021-06-17 11:07 UTC (permalink / raw)
  To: Wesley Cheng
  Cc: balbi, robh+dt, agross, bjorn.andersson, frowand.list, linux-usb,
	linux-kernel, devicetree, linux-arm-msm, jackp, fntoth,
	heikki.krogerus, andy.shevchenko

On Thu, Jun 17, 2021 at 02:58:15AM -0700, Wesley Cheng wrote:
> Ensure that the USB gadget is able to support the configuration being
> added based on the number of endpoints required from all interfaces.  This
> is for accounting for any bandwidth or space limitations.
> 
> Signed-off-by: Wesley Cheng <wcheng@codeaurora.org>
> ---
>  drivers/usb/gadget/configfs.c | 22 ++++++++++++++++++++++
>  1 file changed, 22 insertions(+)
> 
> diff --git a/drivers/usb/gadget/configfs.c b/drivers/usb/gadget/configfs.c
> index 15a607c..76b9983 100644
> --- a/drivers/usb/gadget/configfs.c
> +++ b/drivers/usb/gadget/configfs.c
> @@ -1374,6 +1374,7 @@ static int configfs_composite_bind(struct usb_gadget *gadget,
>  		struct usb_function *f;
>  		struct usb_function *tmp;
>  		struct gadget_config_name *cn;
> +		unsigned long ep_map = 0;
>  
>  		if (gadget_is_otg(gadget))
>  			c->descriptors = otg_desc;
> @@ -1403,7 +1404,28 @@ static int configfs_composite_bind(struct usb_gadget *gadget,
>  				list_add(&f->list, &cfg->func_list);
>  				goto err_purge_funcs;
>  			}
> +			if (f->fs_descriptors) {
> +				struct usb_descriptor_header **d;
> +
> +				d = f->fs_descriptors;
> +				for (; *d; ++d) {

With this check, there really is not a need to check for
f->fs_descriptors above in the if statement, right?

> +					struct usb_endpoint_descriptor *ep;
> +					int addr;
> +
> +					if ((*d)->bDescriptorType != USB_DT_ENDPOINT)
> +						continue;
> +
> +					ep = (struct usb_endpoint_descriptor *)*d;
> +					addr = ((ep->bEndpointAddress & 0x80) >> 3) |
> +						(ep->bEndpointAddress & 0x0f);

Don't we have direction macros for this type of check?

> +					set_bit(addr, &ep_map);
> +				}

What is this loop trying to do?  Please document it as I can not figure
it out at all.

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 1/6] usb: gadget: udc: core: Introduce check_config to verify USB configuration
  2021-06-17  9:58 ` [PATCH v10 1/6] usb: gadget: udc: core: Introduce check_config to verify USB configuration Wesley Cheng
@ 2021-06-17 11:09   ` Greg KH
  2021-06-22  5:27     ` Wesley Cheng
  0 siblings, 1 reply; 73+ messages in thread
From: Greg KH @ 2021-06-17 11:09 UTC (permalink / raw)
  To: Wesley Cheng
  Cc: balbi, robh+dt, agross, bjorn.andersson, frowand.list, linux-usb,
	linux-kernel, devicetree, linux-arm-msm, jackp, fntoth,
	heikki.krogerus, andy.shevchenko

On Thu, Jun 17, 2021 at 02:58:14AM -0700, Wesley Cheng wrote:
> Some UDCs may have constraints on how many high bandwidth endpoints it can
> support in a certain configuration.  This API allows for the composite
> driver to pass down the total number of endpoints to the UDC so it can verify
> it has the required resources to support the configuration.
> 
> Signed-off-by: Wesley Cheng <wcheng@codeaurora.org>
> ---
>  drivers/usb/gadget/udc/core.c | 25 +++++++++++++++++++++++++
>  include/linux/usb/gadget.h    |  5 +++++
>  2 files changed, 30 insertions(+)
> 
> diff --git a/drivers/usb/gadget/udc/core.c b/drivers/usb/gadget/udc/core.c
> index b7f0b1e..e33ae2d 100644
> --- a/drivers/usb/gadget/udc/core.c
> +++ b/drivers/usb/gadget/udc/core.c
> @@ -1003,6 +1003,31 @@ int usb_gadget_ep_match_desc(struct usb_gadget *gadget,
>  }
>  EXPORT_SYMBOL_GPL(usb_gadget_ep_match_desc);
>  
> +/**
> + * usb_gadget_check_config - checks if the UDC can support the number of eps
> + * @gadget: controller to check the USB configuration
> + * @ep_map: bitmap of endpoints being requested by a USB configuration

Will a u64 really hold all of the possible endpoints?

Why make it odd like this, why not just provide a list like we do in the
USB core with the structure that USB drivers use?  What can a driver do
with a bitmap only?


thanks,

greg k-h

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 3/6] usb: dwc3: Resize TX FIFOs to meet EP bursting requirements
  2021-06-17  9:58 ` [PATCH v10 3/6] usb: dwc3: Resize TX FIFOs to meet EP bursting requirements Wesley Cheng
@ 2021-06-17 11:10   ` Greg KH
  2021-06-22  5:27     ` Wesley Cheng
  2021-06-17 17:56   ` kernel test robot
  2021-06-17 18:01   ` kernel test robot
  2 siblings, 1 reply; 73+ messages in thread
From: Greg KH @ 2021-06-17 11:10 UTC (permalink / raw)
  To: Wesley Cheng
  Cc: balbi, robh+dt, agross, bjorn.andersson, frowand.list, linux-usb,
	linux-kernel, devicetree, linux-arm-msm, jackp, fntoth,
	heikki.krogerus, andy.shevchenko

On Thu, Jun 17, 2021 at 02:58:16AM -0700, Wesley Cheng wrote:
> +static int dwc3_gadget_check_config(struct usb_gadget *g, unsigned long ep_map)
> +{
> +	struct dwc3 *dwc = gadget_to_dwc(g);
> +	unsigned long in_ep_map;
> +	int fifo_size = 0;
> +	int ram1_depth;
> +	int ep_num;
> +
> +	if (!dwc->do_fifo_resize)
> +		return 0;
> +
> +	/* Only interested in the IN endpoints */
> +	in_ep_map = ep_map >> 16;

Wait, this "map" is split up into 16/16 somehow?  So it's only 32bits
big?

Where did you document this map structure?  Why is it needed at all, you
have the gadget, don't you have access to the full list of endpoints
here as well?

confused,

greg k-h

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 5/6] usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by default
  2021-06-17  9:58 ` [PATCH v10 5/6] usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by default Wesley Cheng
@ 2021-06-17 11:12   ` Greg KH
  2021-06-17 18:04   ` kernel test robot
  1 sibling, 0 replies; 73+ messages in thread
From: Greg KH @ 2021-06-17 11:12 UTC (permalink / raw)
  To: Wesley Cheng
  Cc: balbi, robh+dt, agross, bjorn.andersson, frowand.list, linux-usb,
	linux-kernel, devicetree, linux-arm-msm, jackp, fntoth,
	heikki.krogerus, andy.shevchenko

On Thu, Jun 17, 2021 at 02:58:18AM -0700, Wesley Cheng wrote:
> In order to take advantage of the TX fifo resizing logic, manually add
> these properties to the DWC3 child node by default.  This will allow
> the DWC3 gadget to resize the TX fifos for the IN endpoints, which
> help with performance.
> 
> Signed-off-by: Wesley Cheng <wcheng@codeaurora.org>
> ---
>  drivers/usb/dwc3/dwc3-qcom.c | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/drivers/usb/dwc3/dwc3-qcom.c b/drivers/usb/dwc3/dwc3-qcom.c
> index 49e6ca9..44e0eaa1 100644
> --- a/drivers/usb/dwc3/dwc3-qcom.c
> +++ b/drivers/usb/dwc3/dwc3-qcom.c
> @@ -645,6 +645,7 @@ static int dwc3_qcom_of_register_core(struct platform_device *pdev)
>  	struct dwc3_qcom	*qcom = platform_get_drvdata(pdev);
>  	struct device_node	*np = pdev->dev.of_node, *dwc3_np;
>  	struct device		*dev = &pdev->dev;
> +	struct property		*prop;
>  	int			ret;
>  
>  	dwc3_np = of_get_compatible_child(np, "snps,dwc3");
> @@ -653,6 +654,14 @@ static int dwc3_qcom_of_register_core(struct platform_device *pdev)
>  		return -ENODEV;
>  	}
>  
> +	prop = devm_kzalloc(dev, sizeof(*prop), GFP_KERNEL);
> +	if (prop) {
> +		prop->name = "tx-fifo-resize";
> +		ret = of_add_property(dwc3_np, prop);
> +		if (ret < 0)
> +			dev_info(dev, "unable to add tx-fifo-resize prop\n");

dev_err()?

And you do not error out properly?


> +	}

No failure if prop is NULL?  This check looks backwards to me, error
paths are in the if () statement, not "all is good".

This feels backwards.

greg k-h

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 3/6] usb: dwc3: Resize TX FIFOs to meet EP bursting requirements
  2021-06-17  9:58 ` [PATCH v10 3/6] usb: dwc3: Resize TX FIFOs to meet EP bursting requirements Wesley Cheng
  2021-06-17 11:10   ` Greg KH
@ 2021-06-17 17:56   ` kernel test robot
  2021-06-17 18:01   ` kernel test robot
  2 siblings, 0 replies; 73+ messages in thread
From: kernel test robot @ 2021-06-17 17:56 UTC (permalink / raw)
  To: Wesley Cheng, balbi, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list
  Cc: kbuild-all, clang-built-linux, linux-usb, linux-kernel,
	devicetree, linux-arm-msm

[-- Attachment #1: Type: text/plain, Size: 3763 bytes --]

Hi Wesley,

Thank you for the patch! Perhaps something to improve:

[auto build test WARNING on usb/usb-testing]
[also build test WARNING on robh/for-next v5.13-rc6 next-20210617]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Wesley-Cheng/Re-introduce-TX-FIFO-resize-for-larger-EP-bursting/20210617-180037
base:   https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing
config: x86_64-randconfig-a011-20210617 (attached as .config)
compiler: clang version 13.0.0 (https://github.com/llvm/llvm-project 64720f57bea6a6bf033feef4a5751ab9c0c3b401)
reproduce (this is a W=1 build):
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # install x86_64 cross compiling tool for clang build
        # apt-get install binutils-x86-64-linux-gnu
        # https://github.com/0day-ci/linux/commit/94892083cf17e46a29d4ef33a044af04854162e6
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Wesley-Cheng/Re-introduce-TX-FIFO-resize-for-larger-EP-bursting/20210617-180037
        git checkout 94892083cf17e46a29d4ef33a044af04854162e6
        # save the attached .config to linux build tree
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=x86_64 

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>

All warnings (new ones prefixed by >>):

>> drivers/usb/dwc3/gadget.c:671:18: warning: variable 'dep' set but not used [-Wunused-but-set-variable]
           struct dwc3_ep *dep;
                           ^
   1 warning generated.
--
   drivers/usb/dwc3/gadget.c:648: warning: Function parameter or member 'mult' not described in 'dwc3_gadget_calc_tx_fifo_size'
   drivers/usb/dwc3/gadget.c:648: warning: Excess function parameter 'nfifos' description in 'dwc3_gadget_calc_tx_fifo_size'
>> drivers/usb/dwc3/gadget.c:670: warning: expecting prototype for dwc3_gadget_clear_tx_fifo_size(). Prototype was for dwc3_gadget_clear_tx_fifos() instead


vim +/dep +671 drivers/usb/dwc3/gadget.c

   661	
   662	/**
   663	 * dwc3_gadget_clear_tx_fifo_size - Clears txfifo allocation
   664	 * @dwc: pointer to the DWC3 context
   665	 *
   666	 * Iterates through all the endpoint registers and clears the previous txfifo
   667	 * allocations.
   668	 */
   669	void dwc3_gadget_clear_tx_fifos(struct dwc3 *dwc)
 > 670	{
 > 671		struct dwc3_ep *dep;
   672		int fifo_depth;
   673		int size;
   674		int num;
   675	
   676		if (!dwc->do_fifo_resize)
   677			return;
   678	
   679		/* Read ep0IN related TXFIFO size */
   680		dep = dwc->eps[1];
   681		size = dwc3_readl(dwc->regs, DWC3_GTXFIFOSIZ(0));
   682		if (DWC3_IP_IS(DWC3))
   683			fifo_depth = DWC3_GTXFIFOSIZ_TXFDEP(size);
   684		else
   685			fifo_depth = DWC31_GTXFIFOSIZ_TXFDEP(size);
   686	
   687		dwc->last_fifo_depth = fifo_depth;
   688		/* Clear existing TXFIFO for all IN eps except ep0 */
   689		for (num = 3; num < min_t(int, dwc->num_eps, DWC3_ENDPOINTS_NUM);
   690		     num += 2) {
   691			dep = dwc->eps[num];
   692			/* Don't change TXFRAMNUM on usb31 version */
   693			size = DWC3_IP_IS(DWC3) ? 0 :
   694				dwc3_readl(dwc->regs, DWC3_GTXFIFOSIZ(num >> 1)) &
   695					   DWC31_GTXFIFOSIZ_TXFRAMNUM;
   696	
   697			dwc3_writel(dwc->regs, DWC3_GTXFIFOSIZ(num >> 1), size);
   698		}
   699		dwc->num_ep_resized = 0;
   700	}
   701	

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 34458 bytes --]

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 3/6] usb: dwc3: Resize TX FIFOs to meet EP bursting requirements
  2021-06-17  9:58 ` [PATCH v10 3/6] usb: dwc3: Resize TX FIFOs to meet EP bursting requirements Wesley Cheng
  2021-06-17 11:10   ` Greg KH
  2021-06-17 17:56   ` kernel test robot
@ 2021-06-17 18:01   ` kernel test robot
  2 siblings, 0 replies; 73+ messages in thread
From: kernel test robot @ 2021-06-17 18:01 UTC (permalink / raw)
  To: Wesley Cheng, balbi, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list
  Cc: kbuild-all, linux-usb, linux-kernel, devicetree, linux-arm-msm

[-- Attachment #1: Type: text/plain, Size: 3656 bytes --]

Hi Wesley,

Thank you for the patch! Perhaps something to improve:

[auto build test WARNING on usb/usb-testing]
[also build test WARNING on robh/for-next v5.13-rc6 next-20210617]
[cannot apply to balbi-usb/testing/next]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Wesley-Cheng/Re-introduce-TX-FIFO-resize-for-larger-EP-bursting/20210617-180037
base:   https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing
config: m68k-allmodconfig (attached as .config)
compiler: m68k-linux-gcc (GCC) 9.3.0
reproduce (this is a W=1 build):
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # https://github.com/0day-ci/linux/commit/94892083cf17e46a29d4ef33a044af04854162e6
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Wesley-Cheng/Re-introduce-TX-FIFO-resize-for-larger-EP-bursting/20210617-180037
        git checkout 94892083cf17e46a29d4ef33a044af04854162e6
        # save the attached .config to linux build tree
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=m68k 

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>

All warnings (new ones prefixed by >>):

   drivers/usb/dwc3/gadget.c: In function 'dwc3_gadget_clear_tx_fifos':
>> drivers/usb/dwc3/gadget.c:671:18: warning: variable 'dep' set but not used [-Wunused-but-set-variable]
     671 |  struct dwc3_ep *dep;
         |                  ^~~
--
   drivers/usb/dwc3/gadget.c:648: warning: Function parameter or member 'mult' not described in 'dwc3_gadget_calc_tx_fifo_size'
   drivers/usb/dwc3/gadget.c:648: warning: Excess function parameter 'nfifos' description in 'dwc3_gadget_calc_tx_fifo_size'
>> drivers/usb/dwc3/gadget.c:670: warning: expecting prototype for dwc3_gadget_clear_tx_fifo_size(). Prototype was for dwc3_gadget_clear_tx_fifos() instead


vim +/dep +671 drivers/usb/dwc3/gadget.c

   661	
   662	/**
   663	 * dwc3_gadget_clear_tx_fifo_size - Clears txfifo allocation
   664	 * @dwc: pointer to the DWC3 context
   665	 *
   666	 * Iterates through all the endpoint registers and clears the previous txfifo
   667	 * allocations.
   668	 */
   669	void dwc3_gadget_clear_tx_fifos(struct dwc3 *dwc)
 > 670	{
 > 671		struct dwc3_ep *dep;
   672		int fifo_depth;
   673		int size;
   674		int num;
   675	
   676		if (!dwc->do_fifo_resize)
   677			return;
   678	
   679		/* Read ep0IN related TXFIFO size */
   680		dep = dwc->eps[1];
   681		size = dwc3_readl(dwc->regs, DWC3_GTXFIFOSIZ(0));
   682		if (DWC3_IP_IS(DWC3))
   683			fifo_depth = DWC3_GTXFIFOSIZ_TXFDEP(size);
   684		else
   685			fifo_depth = DWC31_GTXFIFOSIZ_TXFDEP(size);
   686	
   687		dwc->last_fifo_depth = fifo_depth;
   688		/* Clear existing TXFIFO for all IN eps except ep0 */
   689		for (num = 3; num < min_t(int, dwc->num_eps, DWC3_ENDPOINTS_NUM);
   690		     num += 2) {
   691			dep = dwc->eps[num];
   692			/* Don't change TXFRAMNUM on usb31 version */
   693			size = DWC3_IP_IS(DWC3) ? 0 :
   694				dwc3_readl(dwc->regs, DWC3_GTXFIFOSIZ(num >> 1)) &
   695					   DWC31_GTXFIFOSIZ_TXFRAMNUM;
   696	
   697			dwc3_writel(dwc->regs, DWC3_GTXFIFOSIZ(num >> 1), size);
   698		}
   699		dwc->num_ep_resized = 0;
   700	}
   701	

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 60446 bytes --]

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 5/6] usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by default
  2021-06-17  9:58 ` [PATCH v10 5/6] usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by default Wesley Cheng
  2021-06-17 11:12   ` Greg KH
@ 2021-06-17 18:04   ` kernel test robot
  2021-06-18  4:35     ` Jack Pham
  1 sibling, 1 reply; 73+ messages in thread
From: kernel test robot @ 2021-06-17 18:04 UTC (permalink / raw)
  To: Wesley Cheng, balbi, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list
  Cc: kbuild-all, linux-usb, linux-kernel, devicetree, linux-arm-msm

[-- Attachment #1: Type: text/plain, Size: 1848 bytes --]

Hi Wesley,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on usb/usb-testing]
[also build test ERROR on robh/for-next v5.13-rc6 next-20210617]
[cannot apply to balbi-usb/testing/next]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Wesley-Cheng/Re-introduce-TX-FIFO-resize-for-larger-EP-bursting/20210617-180037
base:   https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing
config: csky-randconfig-s031-20210617 (attached as .config)
compiler: csky-linux-gcc (GCC) 9.3.0
reproduce:
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # apt-get install sparse
        # sparse version: v0.6.3-341-g8af24329-dirty
        # https://github.com/0day-ci/linux/commit/54b6eb3d44464c519a4546e9a2c3041eeced1009
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Wesley-Cheng/Re-introduce-TX-FIFO-resize-for-larger-EP-bursting/20210617-180037
        git checkout 54b6eb3d44464c519a4546e9a2c3041eeced1009
        # save the attached .config to linux build tree
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' W=1 ARCH=csky 

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>

All errors (new ones prefixed by >>, old ones prefixed by <<):

>> ERROR: modpost: "of_add_property" [drivers/usb/dwc3/dwc3-qcom.ko] undefined!

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 37762 bytes --]

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-06-17  9:58 [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting Wesley Cheng
                   ` (5 preceding siblings ...)
  2021-06-17  9:58 ` [PATCH v10 6/6] dt-bindings: usb: dwc3: Update dwc3 TX fifo properties Wesley Cheng
@ 2021-06-17 21:01 ` Ferry Toth
  2021-06-17 21:48   ` Wesley Cheng
  6 siblings, 1 reply; 73+ messages in thread
From: Ferry Toth @ 2021-06-17 21:01 UTC (permalink / raw)
  To: Wesley Cheng, balbi, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list
  Cc: linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	heikki.krogerus, andy.shevchenko

Hi

Op 17-06-2021 om 11:58 schreef Wesley Cheng:
> Changes in V10:
>   - Fixed compilation errors in config where OF is not used (error due to
>     unknown symbol for of_add_property()).  Add of_add_property() stub.
>   - Fixed compilation warning for incorrect argument being passed to dwc3_mdwidth
This fixes the OOPS I had in V9. I do not see any change in performance 
on Merrifield though.
> Changes in V9:
>   - Fixed incorrect patch in series.  Removed changes in DTSI, as dwc3-qcom will
>     add the property by default from the kernel.
>
> Changes in V8:
>   - Rebased to usb-testing
>   - Using devm_kzalloc for adding txfifo property in dwc3-qcom
>   - Removed DWC3 QCOM ACPI property for enabling the txfifo resize
>
> Changes in V7:
>   - Added a new property tx-fifo-max-num for limiting how much fifo space the
>     resizing logic can allocate for endpoints with large burst values.  This
>     can differ across platforms, and tie in closely with overall system latency.
>   - Added recommended checks for DWC32.
>   - Added changes to set the tx-fifo-resize property from dwc3-qcom by default
>     instead of modifying the current DTSI files.
>   - Added comments on all APIs/variables introduced.
>   - Updated the DWC3 YAML to include a better description of the tx-fifo-resize
>     property and added an entry for tx-fifo-max-num.
>
> Changes in V6:
>   - Rebased patches to usb-testing.
>   - Renamed to PATCH series instead of RFC.
>   - Checking for fs_descriptors instead of ss_descriptors for determining the
>     endpoint count for a particular configuration.
>   - Re-ordered patch series to fix patch dependencies.
>
> Changes in V5:
>   - Added check_config() logic, which is used to communicate the number of EPs
>     used in a particular configuration.  Based on this, the DWC3 gadget driver
>     has the ability to know the maximum number of eps utilized in all configs.
>     This helps reduce unnecessary allocation to unused eps, and will catch fifo
>     allocation issues at bind() time.
>   - Fixed variable declaration to single line per variable, and reverse xmas.
>   - Created a helper for fifo clearing, which is used by ep0.c
>
> Changes in V4:
>   - Removed struct dwc3* as an argument for dwc3_gadget_resize_tx_fifos()
>   - Removed WARN_ON(1) in case we run out of fifo space
>   
> Changes in V3:
>   - Removed "Reviewed-by" tags
>   - Renamed series back to RFC
>   - Modified logic to ensure that fifo_size is reset if we pass the minimum
>     threshold.  Tested with binding multiple FDs requesting 6 FIFOs.
>
> Changes in V2:
>   - Modified TXFIFO resizing logic to ensure that each EP is reserved a
>     FIFO.
>   - Removed dev_dbg() prints and fixed typos from patches
>   - Added some more description on the dt-bindings commit message
>
> Currently, there is no functionality to allow for resizing the TXFIFOs, and
> relying on the HW default setting for the TXFIFO depth.  In most cases, the
> HW default is probably sufficient, but for USB compositions that contain
> multiple functions that require EP bursting, the default settings
> might not be enough.  Also to note, the current SW will assign an EP to a
> function driver w/o checking to see if the TXFIFO size for that particular
> EP is large enough. (this is a problem if there are multiple HW defined
> values for the TXFIFO size)
>
> It is mentioned in the SNPS databook that a minimum of TX FIFO depth = 3
> is required for an EP that supports bursting.  Otherwise, there may be
> frequent occurences of bursts ending.  For high bandwidth functions,
> such as data tethering (protocols that support data aggregation), mass
> storage, and media transfer protocol (over FFS), the bMaxBurst value can be
> large, and a bigger TXFIFO depth may prove to be beneficial in terms of USB
> throughput. (which can be associated to system access latency, etc...)  It
> allows for a more consistent burst of traffic, w/o any interruptions, as
> data is readily available in the FIFO.
>
> With testing done using the mass storage function driver, the results show
> that with a larger TXFIFO depth, the bandwidth increased significantly.
>
> Test Parameters:
>   - Platform: Qualcomm SM8150
>   - bMaxBurst = 6
>   - USB req size = 256kB
>   - Num of USB reqs = 16
>   - USB Speed = Super-Speed
>   - Function Driver: Mass Storage (w/ ramdisk)
>   - Test Application: CrystalDiskMark
>
> Results:
>
> TXFIFO Depth = 3 max packets
>
> Test Case | Data Size | AVG tput (in MB/s)
> -------------------------------------------
> Sequential|1 GB x     |
> Read      |9 loops    | 193.60
> 	  |           | 195.86
>            |           | 184.77
>            |           | 193.60
> -------------------------------------------
>
> TXFIFO Depth = 6 max packets
>
> Test Case | Data Size | AVG tput (in MB/s)
> -------------------------------------------
> Sequential|1 GB x     |
> Read      |9 loops    | 287.35
> 	  |           | 304.94
>            |           | 289.64
>            |           | 293.61
> -------------------------------------------
>
> Wesley Cheng (6):
>    usb: gadget: udc: core: Introduce check_config to verify USB
>      configuration
>    usb: gadget: configfs: Check USB configuration before adding
>    usb: dwc3: Resize TX FIFOs to meet EP bursting requirements
>    of: Add stub for of_add_property()
>    usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by default
>    dt-bindings: usb: dwc3: Update dwc3 TX fifo properties
>
>   .../devicetree/bindings/usb/snps,dwc3.yaml         |  15 +-
>   drivers/usb/dwc3/core.c                            |   9 +
>   drivers/usb/dwc3/core.h                            |  15 ++
>   drivers/usb/dwc3/dwc3-qcom.c                       |   9 +
>   drivers/usb/dwc3/ep0.c                             |   2 +
>   drivers/usb/dwc3/gadget.c                          | 212 +++++++++++++++++++++
>   drivers/usb/gadget/configfs.c                      |  22 +++
>   drivers/usb/gadget/udc/core.c                      |  25 +++
>   include/linux/of.h                                 |   5 +
>   include/linux/usb/gadget.h                         |   5 +
>   10 files changed, 317 insertions(+), 2 deletions(-)
>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-06-17 21:01 ` [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting Ferry Toth
@ 2021-06-17 21:48   ` Wesley Cheng
       [not found]     ` <f5ed0ee7-e333-681f-0f1a-d0227562204b@gmail.com>
  0 siblings, 1 reply; 73+ messages in thread
From: Wesley Cheng @ 2021-06-17 21:48 UTC (permalink / raw)
  To: Ferry Toth, balbi, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list
  Cc: linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	heikki.krogerus, andy.shevchenko

Hi,

On 6/17/2021 2:01 PM, Ferry Toth wrote:
> Hi
> 
> Op 17-06-2021 om 11:58 schreef Wesley Cheng:
>> Changes in V10:
>>   - Fixed compilation errors in config where OF is not used (error due to
>>     unknown symbol for of_add_property()).  Add of_add_property() stub.
>>   - Fixed compilation warning for incorrect argument being passed to
>> dwc3_mdwidth
> This fixes the OOPS I had in V9. I do not see any change in performance
> on Merrifield though.

I see...thanks Ferry! With your testing, are you writing to the device's
internal storage (ie UFS, eMMC, etc...), or did you use a ramdisk as well?

If not with a ramdisk, we might want to give that a try to avoid the
storage path being the bottleneck.  You can use "dd" to create an empty
file, and then just use that as the LUN's backing file.

echo ramdisk.img >
/sys/kernel/config/usb_gadget/g1/functions/mass_storage.0/lun.0/file

Thanks
Wesley Cheng

>> Changes in V9:
>>   - Fixed incorrect patch in series.  Removed changes in DTSI, as
>> dwc3-qcom will
>>     add the property by default from the kernel.
>>
>> Changes in V8:
>>   - Rebased to usb-testing
>>   - Using devm_kzalloc for adding txfifo property in dwc3-qcom
>>   - Removed DWC3 QCOM ACPI property for enabling the txfifo resize
>>
>> Changes in V7:
>>   - Added a new property tx-fifo-max-num for limiting how much fifo
>> space the
>>     resizing logic can allocate for endpoints with large burst
>> values.  This
>>     can differ across platforms, and tie in closely with overall
>> system latency.
>>   - Added recommended checks for DWC32.
>>   - Added changes to set the tx-fifo-resize property from dwc3-qcom by
>> default
>>     instead of modifying the current DTSI files.
>>   - Added comments on all APIs/variables introduced.
>>   - Updated the DWC3 YAML to include a better description of the
>> tx-fifo-resize
>>     property and added an entry for tx-fifo-max-num.
>>
>> Changes in V6:
>>   - Rebased patches to usb-testing.
>>   - Renamed to PATCH series instead of RFC.
>>   - Checking for fs_descriptors instead of ss_descriptors for
>> determining the
>>     endpoint count for a particular configuration.
>>   - Re-ordered patch series to fix patch dependencies.
>>
>> Changes in V5:
>>   - Added check_config() logic, which is used to communicate the
>> number of EPs
>>     used in a particular configuration.  Based on this, the DWC3
>> gadget driver
>>     has the ability to know the maximum number of eps utilized in all
>> configs.
>>     This helps reduce unnecessary allocation to unused eps, and will
>> catch fifo
>>     allocation issues at bind() time.
>>   - Fixed variable declaration to single line per variable, and
>> reverse xmas.
>>   - Created a helper for fifo clearing, which is used by ep0.c
>>
>> Changes in V4:
>>   - Removed struct dwc3* as an argument for dwc3_gadget_resize_tx_fifos()
>>   - Removed WARN_ON(1) in case we run out of fifo space
>>   Changes in V3:
>>   - Removed "Reviewed-by" tags
>>   - Renamed series back to RFC
>>   - Modified logic to ensure that fifo_size is reset if we pass the
>> minimum
>>     threshold.  Tested with binding multiple FDs requesting 6 FIFOs.
>>
>> Changes in V2:
>>   - Modified TXFIFO resizing logic to ensure that each EP is reserved a
>>     FIFO.
>>   - Removed dev_dbg() prints and fixed typos from patches
>>   - Added some more description on the dt-bindings commit message
>>
>> Currently, there is no functionality to allow for resizing the
>> TXFIFOs, and
>> relying on the HW default setting for the TXFIFO depth.  In most
>> cases, the
>> HW default is probably sufficient, but for USB compositions that contain
>> multiple functions that require EP bursting, the default settings
>> might not be enough.  Also to note, the current SW will assign an EP to a
>> function driver w/o checking to see if the TXFIFO size for that
>> particular
>> EP is large enough. (this is a problem if there are multiple HW defined
>> values for the TXFIFO size)
>>
>> It is mentioned in the SNPS databook that a minimum of TX FIFO depth = 3
>> is required for an EP that supports bursting.  Otherwise, there may be
>> frequent occurences of bursts ending.  For high bandwidth functions,
>> such as data tethering (protocols that support data aggregation), mass
>> storage, and media transfer protocol (over FFS), the bMaxBurst value
>> can be
>> large, and a bigger TXFIFO depth may prove to be beneficial in terms
>> of USB
>> throughput. (which can be associated to system access latency,
>> etc...)  It
>> allows for a more consistent burst of traffic, w/o any interruptions, as
>> data is readily available in the FIFO.
>>
>> With testing done using the mass storage function driver, the results
>> show
>> that with a larger TXFIFO depth, the bandwidth increased significantly.
>>
>> Test Parameters:
>>   - Platform: Qualcomm SM8150
>>   - bMaxBurst = 6
>>   - USB req size = 256kB
>>   - Num of USB reqs = 16
>>   - USB Speed = Super-Speed
>>   - Function Driver: Mass Storage (w/ ramdisk)
>>   - Test Application: CrystalDiskMark
>>
>> Results:
>>
>> TXFIFO Depth = 3 max packets
>>
>> Test Case | Data Size | AVG tput (in MB/s)
>> -------------------------------------------
>> Sequential|1 GB x     |
>> Read      |9 loops    | 193.60
>>       |           | 195.86
>>            |           | 184.77
>>            |           | 193.60
>> -------------------------------------------
>>
>> TXFIFO Depth = 6 max packets
>>
>> Test Case | Data Size | AVG tput (in MB/s)
>> -------------------------------------------
>> Sequential|1 GB x     |
>> Read      |9 loops    | 287.35
>>       |           | 304.94
>>            |           | 289.64
>>            |           | 293.61
>> -------------------------------------------
>>
>> Wesley Cheng (6):
>>    usb: gadget: udc: core: Introduce check_config to verify USB
>>      configuration
>>    usb: gadget: configfs: Check USB configuration before adding
>>    usb: dwc3: Resize TX FIFOs to meet EP bursting requirements
>>    of: Add stub for of_add_property()
>>    usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by default
>>    dt-bindings: usb: dwc3: Update dwc3 TX fifo properties
>>
>>   .../devicetree/bindings/usb/snps,dwc3.yaml         |  15 +-
>>   drivers/usb/dwc3/core.c                            |   9 +
>>   drivers/usb/dwc3/core.h                            |  15 ++
>>   drivers/usb/dwc3/dwc3-qcom.c                       |   9 +
>>   drivers/usb/dwc3/ep0.c                             |   2 +
>>   drivers/usb/dwc3/gadget.c                          | 212
>> +++++++++++++++++++++
>>   drivers/usb/gadget/configfs.c                      |  22 +++
>>   drivers/usb/gadget/udc/core.c                      |  25 +++
>>   include/linux/of.h                                 |   5 +
>>   include/linux/usb/gadget.h                         |   5 +
>>   10 files changed, 317 insertions(+), 2 deletions(-)
>>

-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
       [not found]     ` <f5ed0ee7-e333-681f-0f1a-d0227562204b@gmail.com>
@ 2021-06-17 22:25       ` Wesley Cheng
  2021-06-19 12:40         ` Ferry Toth
  0 siblings, 1 reply; 73+ messages in thread
From: Wesley Cheng @ 2021-06-17 22:25 UTC (permalink / raw)
  To: Ferry Toth, balbi, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list
  Cc: linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	heikki.krogerus, andy.shevchenko

Hi,

On 6/17/2021 2:55 PM, Ferry Toth wrote:
> Hi
> 
> Op 17-06-2021 om 23:48 schreef Wesley Cheng:
>> Hi,
>>
>> On 6/17/2021 2:01 PM, Ferry Toth wrote:
>>> Hi
>>>
>>> Op 17-06-2021 om 11:58 schreef Wesley Cheng:
>>>> Changes in V10:
>>>>   - Fixed compilation errors in config where OF is not used (error due to
>>>>     unknown symbol for of_add_property()).  Add of_add_property() stub.
>>>>   - Fixed compilation warning for incorrect argument being passed to
>>>> dwc3_mdwidth
>>> This fixes the OOPS I had in V9. I do not see any change in performance
>>> on Merrifield though.
>> I see...thanks Ferry! With your testing, are you writing to the device's
>> internal storage (ie UFS, eMMC, etc...), or did you use a ramdisk as well?
> 
> In this case I just tested the EEM path using iperf3.
> 

Got it.  I don't believe f_eem will use a high enough (if at all)
bMaxBurst value to change the TXFIFO size.

>> If not with a ramdisk, we might want to give that a try to avoid the
>> storage path being the bottleneck.  You can use "dd" to create an empty
>> file, and then just use that as the LUN's backing file.
>>
>> echo ramdisk.img >
>> /sys/kernel/config/usb_gadget/g1/functions/mass_storage.0/lun.0/file
> 
> Ah, why didn't I think of that. I have currently mass storage setup with
> eMMC but it seems that is indeed the bottleneck.
> 

:), not a problem...I've been working on getting the ideal set up for
the performance profiling for awhile, so anything I can do to make sure
we get some good results.

> I'll try with a ramdisk and let you know.
> 

Thanks again for the testing, Ferry.

Thanks
Wesley Cheng

>> Thanks
>> Wesley Cheng
>>
>>>> Changes in V9:
>>>>   - Fixed incorrect patch in series.  Removed changes in DTSI, as
>>>> dwc3-qcom will
>>>>     add the property by default from the kernel.
>>>>
>>>> Changes in V8:
>>>>   - Rebased to usb-testing
>>>>   - Using devm_kzalloc for adding txfifo property in dwc3-qcom
>>>>   - Removed DWC3 QCOM ACPI property for enabling the txfifo resize
>>>>
>>>> Changes in V7:
>>>>   - Added a new property tx-fifo-max-num for limiting how much fifo
>>>> space the
>>>>     resizing logic can allocate for endpoints with large burst
>>>> values.  This
>>>>     can differ across platforms, and tie in closely with overall
>>>> system latency.
>>>>   - Added recommended checks for DWC32.
>>>>   - Added changes to set the tx-fifo-resize property from dwc3-qcom by
>>>> default
>>>>     instead of modifying the current DTSI files.
>>>>   - Added comments on all APIs/variables introduced.
>>>>   - Updated the DWC3 YAML to include a better description of the
>>>> tx-fifo-resize
>>>>     property and added an entry for tx-fifo-max-num.
>>>>
>>>> Changes in V6:
>>>>   - Rebased patches to usb-testing.
>>>>   - Renamed to PATCH series instead of RFC.
>>>>   - Checking for fs_descriptors instead of ss_descriptors for
>>>> determining the
>>>>     endpoint count for a particular configuration.
>>>>   - Re-ordered patch series to fix patch dependencies.
>>>>
>>>> Changes in V5:
>>>>   - Added check_config() logic, which is used to communicate the
>>>> number of EPs
>>>>     used in a particular configuration.  Based on this, the DWC3
>>>> gadget driver
>>>>     has the ability to know the maximum number of eps utilized in all
>>>> configs.
>>>>     This helps reduce unnecessary allocation to unused eps, and will
>>>> catch fifo
>>>>     allocation issues at bind() time.
>>>>   - Fixed variable declaration to single line per variable, and
>>>> reverse xmas.
>>>>   - Created a helper for fifo clearing, which is used by ep0.c
>>>>
>>>> Changes in V4:
>>>>   - Removed struct dwc3* as an argument for dwc3_gadget_resize_tx_fifos()
>>>>   - Removed WARN_ON(1) in case we run out of fifo space
>>>>   Changes in V3:
>>>>   - Removed "Reviewed-by" tags
>>>>   - Renamed series back to RFC
>>>>   - Modified logic to ensure that fifo_size is reset if we pass the
>>>> minimum
>>>>     threshold.  Tested with binding multiple FDs requesting 6 FIFOs.
>>>>
>>>> Changes in V2:
>>>>   - Modified TXFIFO resizing logic to ensure that each EP is reserved a
>>>>     FIFO.
>>>>   - Removed dev_dbg() prints and fixed typos from patches
>>>>   - Added some more description on the dt-bindings commit message
>>>>
>>>> Currently, there is no functionality to allow for resizing the
>>>> TXFIFOs, and
>>>> relying on the HW default setting for the TXFIFO depth.  In most
>>>> cases, the
>>>> HW default is probably sufficient, but for USB compositions that contain
>>>> multiple functions that require EP bursting, the default settings
>>>> might not be enough.  Also to note, the current SW will assign an EP to a
>>>> function driver w/o checking to see if the TXFIFO size for that
>>>> particular
>>>> EP is large enough. (this is a problem if there are multiple HW defined
>>>> values for the TXFIFO size)
>>>>
>>>> It is mentioned in the SNPS databook that a minimum of TX FIFO depth = 3
>>>> is required for an EP that supports bursting.  Otherwise, there may be
>>>> frequent occurences of bursts ending.  For high bandwidth functions,
>>>> such as data tethering (protocols that support data aggregation), mass
>>>> storage, and media transfer protocol (over FFS), the bMaxBurst value
>>>> can be
>>>> large, and a bigger TXFIFO depth may prove to be beneficial in terms
>>>> of USB
>>>> throughput. (which can be associated to system access latency,
>>>> etc...)  It
>>>> allows for a more consistent burst of traffic, w/o any interruptions, as
>>>> data is readily available in the FIFO.
>>>>
>>>> With testing done using the mass storage function driver, the results
>>>> show
>>>> that with a larger TXFIFO depth, the bandwidth increased significantly.
>>>>
>>>> Test Parameters:
>>>>   - Platform: Qualcomm SM8150
>>>>   - bMaxBurst = 6
>>>>   - USB req size = 256kB
>>>>   - Num of USB reqs = 16
>>>>   - USB Speed = Super-Speed
>>>>   - Function Driver: Mass Storage (w/ ramdisk)
>>>>   - Test Application: CrystalDiskMark
>>>>
>>>> Results:
>>>>
>>>> TXFIFO Depth = 3 max packets
>>>>
>>>> Test Case | Data Size | AVG tput (in MB/s)
>>>> -------------------------------------------
>>>> Sequential|1 GB x     |
>>>> Read      |9 loops    | 193.60
>>>>       |           | 195.86
>>>>            |           | 184.77
>>>>            |           | 193.60
>>>> -------------------------------------------
>>>>
>>>> TXFIFO Depth = 6 max packets
>>>>
>>>> Test Case | Data Size | AVG tput (in MB/s)
>>>> -------------------------------------------
>>>> Sequential|1 GB x     |
>>>> Read      |9 loops    | 287.35
>>>>       |           | 304.94
>>>>            |           | 289.64
>>>>            |           | 293.61
>>>> -------------------------------------------
>>>>
>>>> Wesley Cheng (6):
>>>>    usb: gadget: udc: core: Introduce check_config to verify USB
>>>>      configuration
>>>>    usb: gadget: configfs: Check USB configuration before adding
>>>>    usb: dwc3: Resize TX FIFOs to meet EP bursting requirements
>>>>    of: Add stub for of_add_property()
>>>>    usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by default
>>>>    dt-bindings: usb: dwc3: Update dwc3 TX fifo properties
>>>>
>>>>   .../devicetree/bindings/usb/snps,dwc3.yaml         |  15 +-
>>>>   drivers/usb/dwc3/core.c                            |   9 +
>>>>   drivers/usb/dwc3/core.h                            |  15 ++
>>>>   drivers/usb/dwc3/dwc3-qcom.c                       |   9 +
>>>>   drivers/usb/dwc3/ep0.c                             |   2 +
>>>>   drivers/usb/dwc3/gadget.c                          | 212
>>>> +++++++++++++++++++++
>>>>   drivers/usb/gadget/configfs.c                      |  22 +++
>>>>   drivers/usb/gadget/udc/core.c                      |  25 +++
>>>>   include/linux/of.h                                 |   5 +
>>>>   include/linux/usb/gadget.h                         |   5 +
>>>>   10 files changed, 317 insertions(+), 2 deletions(-)
>>>>

-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 5/6] usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by default
  2021-06-17 18:04   ` kernel test robot
@ 2021-06-18  4:35     ` Jack Pham
  0 siblings, 0 replies; 73+ messages in thread
From: Jack Pham @ 2021-06-18  4:35 UTC (permalink / raw)
  To: Wesley Cheng
  Cc: kernel test robot, balbi, gregkh, robh+dt, agross,
	bjorn.andersson, frowand.list, kbuild-all, linux-usb,
	linux-kernel, devicetree, linux-arm-msm

Hi Wesley,

On Fri, Jun 18, 2021 at 02:04:40AM +0800, kernel test robot wrote:
> Hi Wesley,
> 
> Thank you for the patch! Yet something to improve:
> 
> [auto build test ERROR on usb/usb-testing]
> [also build test ERROR on robh/for-next v5.13-rc6 next-20210617]
> [cannot apply to balbi-usb/testing/next]
> [If your patch is applied to the wrong git tree, kindly drop us a note.
> And when submitting patch, we suggest to use '--base' as documented in
> https://git-scm.com/docs/git-format-patch]
> 
> url:    https://github.com/0day-ci/linux/commits/Wesley-Cheng/Re-introduce-TX-FIFO-resize-for-larger-EP-bursting/20210617-180037
> base:   https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing
> config: csky-randconfig-s031-20210617 (attached as .config)
> compiler: csky-linux-gcc (GCC) 9.3.0
> reproduce:
>         wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
>         chmod +x ~/bin/make.cross
>         # apt-get install sparse
>         # sparse version: v0.6.3-341-g8af24329-dirty
>         # https://github.com/0day-ci/linux/commit/54b6eb3d44464c519a4546e9a2c3041eeced1009
>         git remote add linux-review https://github.com/0day-ci/linux
>         git fetch --no-tags linux-review Wesley-Cheng/Re-introduce-TX-FIFO-resize-for-larger-EP-bursting/20210617-180037
>         git checkout 54b6eb3d44464c519a4546e9a2c3041eeced1009
>         # save the attached .config to linux build tree
>         COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' W=1 ARCH=csky 
> 
> If you fix the issue, kindly add following tag as appropriate
> Reported-by: kernel test robot <lkp@intel.com>
> 
> All errors (new ones prefixed by >>, old ones prefixed by <<):
> 
> >> ERROR: modpost: "of_add_property" [drivers/usb/dwc3/dwc3-qcom.ko] undefined!

*sigh* this symbol is not EXPORTed...so you can't call it from a module.

Actually couldn't we just use device_add_properties() instead? And drop
patch 4/6 as well.

Jack
-- 
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-06-17 22:25       ` Wesley Cheng
@ 2021-06-19 12:40         ` Ferry Toth
  2021-06-22 18:38           ` Wesley Cheng
  0 siblings, 1 reply; 73+ messages in thread
From: Ferry Toth @ 2021-06-19 12:40 UTC (permalink / raw)
  To: Wesley Cheng, balbi, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list
  Cc: linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	heikki.krogerus, andy.shevchenko

Hi

Op 18-06-2021 om 00:25 schreef Wesley Cheng:
> Hi,
>
> On 6/17/2021 2:55 PM, Ferry Toth wrote:
>> Hi
>>
>> Op 17-06-2021 om 23:48 schreef Wesley Cheng:
>>> Hi,
>>>
>>> On 6/17/2021 2:01 PM, Ferry Toth wrote:
>>>> Hi
>>>>
>>>> Op 17-06-2021 om 11:58 schreef Wesley Cheng:
>>>>> Changes in V10:
>>>>>    - Fixed compilation errors in config where OF is not used (error due to
>>>>>      unknown symbol for of_add_property()).  Add of_add_property() stub.
>>>>>    - Fixed compilation warning for incorrect argument being passed to
>>>>> dwc3_mdwidth
>>>> This fixes the OOPS I had in V9. I do not see any change in performance
>>>> on Merrifield though.
>>> I see...thanks Ferry! With your testing, are you writing to the device's
>>> internal storage (ie UFS, eMMC, etc...), or did you use a ramdisk as well?
>> In this case I just tested the EEM path using iperf3.
>>
> Got it.  I don't believe f_eem will use a high enough (if at all)
> bMaxBurst value to change the TXFIFO size.
>
>>> If not with a ramdisk, we might want to give that a try to avoid the
>>> storage path being the bottleneck.  You can use "dd" to create an empty
>>> file, and then just use that as the LUN's backing file.
>>>
>>> echo ramdisk.img >
>>> /sys/kernel/config/usb_gadget/g1/functions/mass_storage.0/lun.0/file
>> Ah, why didn't I think of that. I have currently mass storage setup with
>> eMMC but it seems that is indeed the bottleneck.
>>
I created a 64MB disk following the instructions here 
http://www.linux-usb.org/gadget/file_storage.html (that seems a little 
outdated, at least I can not start the first partition at sector 8, but 
minimum 2048), and added a test file on it.

I then copy the file to /dev/shm prior to setting configfs (composite 
device gser/eem/mass_storage/uac2).

journal shows:

kernel: Mass Storage Function, version: 2009/09/11
kernel: LUN: removable file: (no medium)

I don't know what that means, because I see the test file on the ramdisk.

Then I again used gnome disks to benchmark (read/write 10MB):

With V10 on top v5.13.0-rc5:

R/W speed = 35.6/35.8MB/s, access time 0.35ms

With no patches on top v5.12.0:

R/W speed = 35.7/36.1MB/s, access time 0.35ms

I see no speed difference (and it's about the same as with the eMMC 
backed disk). But the patches are causing a new call trace

kernel: using random self ethernet address
kernel: using random host ethernet address
kernel: Mass Storage Function, version: 2009/09/11
kernel: LUN: removable file: (no medium)
kernel: usb0: HOST MAC aa:bb:cc:dd:ee:f2
kernel: usb0: MAC aa:bb:cc:dd:ee:f1
kernel: IPv6: ADDRCONF(NETDEV_CHANGE): usb0: link becomes ready
kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in 
depth:115540359
kernel: ------------[ cut here ]------------
kernel: WARNING: CPU: 0 PID: 594 at drivers/usb/gadget/udc/core.c:278 
usb_ep_queue+0x75/0x80
kernel: Modules linked in: usb_f_uac2 u_audio usb_f_mass_storage 
usb_f_eem u_ether usb_f_serial u_serial libcomposite rfcomm iptable_nat 
bnep snd_sof_nocodec spi_pxa2>
kernel: CPU: 0 PID: 594 Comm: irq/14-dwc3 Not tainted 
5.13.0-rc5-edison-acpi-standard #1
kernel: Hardware name: Intel Corporation Merrifield/BODEGA BAY, BIOS 542 
2015.01.21:18.19.48
kernel: RIP: 0010:usb_ep_queue+0x75/0x80
kernel: Code: 01 73 e4 48 8b 05 fb 63 06 01 48 85 c0 74 12 48 8b 78 08 
44 89 e9 4c 89 e2 48 89 ee e8 74 05 00 00 44 89 e8 5d 41 5c 41 5d c3 
<0f> 0b 41 bd 94 ff ff ff >
kernel: RSP: 0000:ffff91eec083fc98 EFLAGS: 00010082
kernel: RAX: ffff8af20357d960 RBX: 0000000000000000 RCX: ffff8af202f06400
kernel: RDX: 0000000000000a20 RSI: ffff8af208785780 RDI: ffff8af202e9ae00
kernel: RBP: ffff8af202e9ae00 R08: 00000000000000c0 R09: ffff8af208785780
kernel: R10: 00000000ffffe000 R11: 3fffffffffffffff R12: ffff8af208785780
kernel: R13: 0000000000000000 R14: ffff8af202e9ae00 R15: ffff8af203e26cc0
kernel: FS:  0000000000000000(0000) GS:ffff8af23e200000(0000) 
knlGS:0000000000000000
kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 000055e2c21f2100 CR3: 0000000003b38000 CR4: 00000000001006f0
kernel: Call Trace:
kernel:  u_audio_start_playback+0x107/0x1a0 [u_audio]
kernel:  composite_setup+0x224/0x1ba0 [libcomposite]
kernel:  ? dwc3_gadget_ep_queue+0xf6/0x1a0
kernel:  ? usb_ep_queue+0x2a/0x80
kernel:  ? configfs_composite_setup+0x6b/0x90 [libcomposite]
kernel:  configfs_composite_setup+0x6b/0x90 [libcomposite]
kernel:  dwc3_ep0_interrupt+0x469/0xa80
kernel:  dwc3_thread_interrupt+0x8ee/0xf40
kernel:  ? __wake_up_common_lock+0x85/0xb0
kernel:  ? disable_irq_nosync+0x10/0x10
kernel:  irq_thread_fn+0x1b/0x60
kernel:  irq_thread+0xd6/0x170
kernel:  ? irq_thread_check_affinity+0x70/0x70
kernel:  ? irq_forced_thread_fn+0x70/0x70
kernel:  kthread+0x116/0x130
kernel:  ? kthread_create_worker_on_cpu+0x60/0x60
kernel:  ret_from_fork+0x22/0x30
kernel: ---[ end trace e5b9e28058c53584 ]---
kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
kernel: dwc3 dwc3.0.auto: request 000000003c32dcc5 was not queued to ep5in
kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to ep5in
kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in 
depth:115540359
kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to ep5in
kernel: dwc3 dwc3.0.auto: request 00000000036ac129 was not queued to ep5in
kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in 
depth:115540359
kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
kernel: dwc3 dwc3.0.auto: request 00000000ad1b8c18 was not queued to ep5in
kernel: dwc3 dwc3.0.auto: request 00000000fbc71244 was not queued to ep5in
kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in 
depth:115540359
kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
kernel: dwc3 dwc3.0.auto: request 00000000fbc71244 was not queued to ep5in
kernel: dwc3 dwc3.0.auto: request 00000000ad1b8c18 was not queued to ep5in
kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in 
depth:115540359
kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
kernel: dwc3 dwc3.0.auto: request 000000003c32dcc5 was not queued to ep5in
kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to ep5in

Removing uac2 from the config makes the call trace go away, but the R/W 
speed does not change.

> :), not a problem...I've been working on getting the ideal set up for
> the performance profiling for awhile, so anything I can do to make sure
> we get some good results.
>
>> I'll try with a ramdisk and let you know.
>>
> Thanks again for the testing, Ferry.
>
> Thanks
> Wesley Cheng
>
>>> Thanks
>>> Wesley Cheng
>>>
>>>>> Changes in V9:
>>>>>    - Fixed incorrect patch in series.  Removed changes in DTSI, as
>>>>> dwc3-qcom will
>>>>>      add the property by default from the kernel.
>>>>>
>>>>> Changes in V8:
>>>>>    - Rebased to usb-testing
>>>>>    - Using devm_kzalloc for adding txfifo property in dwc3-qcom
>>>>>    - Removed DWC3 QCOM ACPI property for enabling the txfifo resize
>>>>>
>>>>> Changes in V7:
>>>>>    - Added a new property tx-fifo-max-num for limiting how much fifo
>>>>> space the
>>>>>      resizing logic can allocate for endpoints with large burst
>>>>> values.  This
>>>>>      can differ across platforms, and tie in closely with overall
>>>>> system latency.
>>>>>    - Added recommended checks for DWC32.
>>>>>    - Added changes to set the tx-fifo-resize property from dwc3-qcom by
>>>>> default
>>>>>      instead of modifying the current DTSI files.
>>>>>    - Added comments on all APIs/variables introduced.
>>>>>    - Updated the DWC3 YAML to include a better description of the
>>>>> tx-fifo-resize
>>>>>      property and added an entry for tx-fifo-max-num.
>>>>>
>>>>> Changes in V6:
>>>>>    - Rebased patches to usb-testing.
>>>>>    - Renamed to PATCH series instead of RFC.
>>>>>    - Checking for fs_descriptors instead of ss_descriptors for
>>>>> determining the
>>>>>      endpoint count for a particular configuration.
>>>>>    - Re-ordered patch series to fix patch dependencies.
>>>>>
>>>>> Changes in V5:
>>>>>    - Added check_config() logic, which is used to communicate the
>>>>> number of EPs
>>>>>      used in a particular configuration.  Based on this, the DWC3
>>>>> gadget driver
>>>>>      has the ability to know the maximum number of eps utilized in all
>>>>> configs.
>>>>>      This helps reduce unnecessary allocation to unused eps, and will
>>>>> catch fifo
>>>>>      allocation issues at bind() time.
>>>>>    - Fixed variable declaration to single line per variable, and
>>>>> reverse xmas.
>>>>>    - Created a helper for fifo clearing, which is used by ep0.c
>>>>>
>>>>> Changes in V4:
>>>>>    - Removed struct dwc3* as an argument for dwc3_gadget_resize_tx_fifos()
>>>>>    - Removed WARN_ON(1) in case we run out of fifo space
>>>>>    Changes in V3:
>>>>>    - Removed "Reviewed-by" tags
>>>>>    - Renamed series back to RFC
>>>>>    - Modified logic to ensure that fifo_size is reset if we pass the
>>>>> minimum
>>>>>      threshold.  Tested with binding multiple FDs requesting 6 FIFOs.
>>>>>
>>>>> Changes in V2:
>>>>>    - Modified TXFIFO resizing logic to ensure that each EP is reserved a
>>>>>      FIFO.
>>>>>    - Removed dev_dbg() prints and fixed typos from patches
>>>>>    - Added some more description on the dt-bindings commit message
>>>>>
>>>>> Currently, there is no functionality to allow for resizing the
>>>>> TXFIFOs, and
>>>>> relying on the HW default setting for the TXFIFO depth.  In most
>>>>> cases, the
>>>>> HW default is probably sufficient, but for USB compositions that contain
>>>>> multiple functions that require EP bursting, the default settings
>>>>> might not be enough.  Also to note, the current SW will assign an EP to a
>>>>> function driver w/o checking to see if the TXFIFO size for that
>>>>> particular
>>>>> EP is large enough. (this is a problem if there are multiple HW defined
>>>>> values for the TXFIFO size)
>>>>>
>>>>> It is mentioned in the SNPS databook that a minimum of TX FIFO depth = 3
>>>>> is required for an EP that supports bursting.  Otherwise, there may be
>>>>> frequent occurences of bursts ending.  For high bandwidth functions,
>>>>> such as data tethering (protocols that support data aggregation), mass
>>>>> storage, and media transfer protocol (over FFS), the bMaxBurst value
>>>>> can be
>>>>> large, and a bigger TXFIFO depth may prove to be beneficial in terms
>>>>> of USB
>>>>> throughput. (which can be associated to system access latency,
>>>>> etc...)  It
>>>>> allows for a more consistent burst of traffic, w/o any interruptions, as
>>>>> data is readily available in the FIFO.
>>>>>
>>>>> With testing done using the mass storage function driver, the results
>>>>> show
>>>>> that with a larger TXFIFO depth, the bandwidth increased significantly.
>>>>>
>>>>> Test Parameters:
>>>>>    - Platform: Qualcomm SM8150
>>>>>    - bMaxBurst = 6
>>>>>    - USB req size = 256kB
>>>>>    - Num of USB reqs = 16
>>>>>    - USB Speed = Super-Speed
>>>>>    - Function Driver: Mass Storage (w/ ramdisk)
>>>>>    - Test Application: CrystalDiskMark
>>>>>
>>>>> Results:
>>>>>
>>>>> TXFIFO Depth = 3 max packets
>>>>>
>>>>> Test Case | Data Size | AVG tput (in MB/s)
>>>>> -------------------------------------------
>>>>> Sequential|1 GB x     |
>>>>> Read      |9 loops    | 193.60
>>>>>        |           | 195.86
>>>>>             |           | 184.77
>>>>>             |           | 193.60
>>>>> -------------------------------------------
>>>>>
>>>>> TXFIFO Depth = 6 max packets
>>>>>
>>>>> Test Case | Data Size | AVG tput (in MB/s)
>>>>> -------------------------------------------
>>>>> Sequential|1 GB x     |
>>>>> Read      |9 loops    | 287.35
>>>>>        |           | 304.94
>>>>>             |           | 289.64
>>>>>             |           | 293.61
>>>>> -------------------------------------------
>>>>>
>>>>> Wesley Cheng (6):
>>>>>     usb: gadget: udc: core: Introduce check_config to verify USB
>>>>>       configuration
>>>>>     usb: gadget: configfs: Check USB configuration before adding
>>>>>     usb: dwc3: Resize TX FIFOs to meet EP bursting requirements
>>>>>     of: Add stub for of_add_property()
>>>>>     usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by default
>>>>>     dt-bindings: usb: dwc3: Update dwc3 TX fifo properties
>>>>>
>>>>>    .../devicetree/bindings/usb/snps,dwc3.yaml         |  15 +-
>>>>>    drivers/usb/dwc3/core.c                            |   9 +
>>>>>    drivers/usb/dwc3/core.h                            |  15 ++
>>>>>    drivers/usb/dwc3/dwc3-qcom.c                       |   9 +
>>>>>    drivers/usb/dwc3/ep0.c                             |   2 +
>>>>>    drivers/usb/dwc3/gadget.c                          | 212
>>>>> +++++++++++++++++++++
>>>>>    drivers/usb/gadget/configfs.c                      |  22 +++
>>>>>    drivers/usb/gadget/udc/core.c                      |  25 +++
>>>>>    include/linux/of.h                                 |   5 +
>>>>>    include/linux/usb/gadget.h                         |   5 +
>>>>>    10 files changed, 317 insertions(+), 2 deletions(-)
>>>>>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 1/6] usb: gadget: udc: core: Introduce check_config to verify USB configuration
  2021-06-17 11:09   ` Greg KH
@ 2021-06-22  5:27     ` Wesley Cheng
  0 siblings, 0 replies; 73+ messages in thread
From: Wesley Cheng @ 2021-06-22  5:27 UTC (permalink / raw)
  To: Greg KH
  Cc: balbi, robh+dt, agross, bjorn.andersson, frowand.list, linux-usb,
	linux-kernel, devicetree, linux-arm-msm, jackp, fntoth,
	heikki.krogerus, andy.shevchenko



On 6/17/2021 4:09 AM, Greg KH wrote:
> On Thu, Jun 17, 2021 at 02:58:14AM -0700, Wesley Cheng wrote:
>> Some UDCs may have constraints on how many high bandwidth endpoints it can
>> support in a certain configuration.  This API allows for the composite
>> driver to pass down the total number of endpoints to the UDC so it can verify
>> it has the required resources to support the configuration.
>>
>> Signed-off-by: Wesley Cheng <wcheng@codeaurora.org>
>> ---
>>  drivers/usb/gadget/udc/core.c | 25 +++++++++++++++++++++++++
>>  include/linux/usb/gadget.h    |  5 +++++
>>  2 files changed, 30 insertions(+)
>>
>> diff --git a/drivers/usb/gadget/udc/core.c b/drivers/usb/gadget/udc/core.c
>> index b7f0b1e..e33ae2d 100644
>> --- a/drivers/usb/gadget/udc/core.c
>> +++ b/drivers/usb/gadget/udc/core.c
>> @@ -1003,6 +1003,31 @@ int usb_gadget_ep_match_desc(struct usb_gadget *gadget,
>>  }
>>  EXPORT_SYMBOL_GPL(usb_gadget_ep_match_desc);
>>  
>> +/**
>> + * usb_gadget_check_config - checks if the UDC can support the number of eps
>> + * @gadget: controller to check the USB configuration
>> + * @ep_map: bitmap of endpoints being requested by a USB configuration

Hi Greg,

> 
> Will a u64 really hold all of the possible endpoints?
> 

Ah, should be a u32 bitmap, and that is enough as USB spec only allows
32 EPs (IN+OUT eps) total.  That hasn't changed since USB2, so I assume
that will stay the same moving forward.  I can fix that.

> Why make it odd like this, why not just provide a list like we do in the
> USB core with the structure that USB drivers use?  What can a driver do
> with a bitmap only?
> 

I didn't want the ep bookeeping here to be too complicated.  For
example, in the TXFIFO resize situation, just knowing the number of
endpoints used can help determine if we have enough internal controller
memory to allocate per endpoint. (at minimum 1 FIFO per EP)  If the USB
configuration is going to be requesting more endpoints than FIFO memory
it has (unlikely), then it will fail the config bind.  Otherwise, we'd
end up enumerating w/ the host w/ the interfaces that were starved of
FIFO memory to be broken/non-functional.  This was one of the concerns
that Felipe had in our initial discussions.

In addition, at the time of function driver binding, the amount of
information we have is minimal per endpoint, as most of it is populated
once the host places the device into the CONFIGURED state. (ie when the
device knows which configuration is being enabled, hence which EPs are
being used)

Thanks
Wesley Cheng

> 
> thanks,
> 
> greg k-h
> 

-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 2/6] usb: gadget: configfs: Check USB configuration before adding
  2021-06-17 11:07   ` Greg KH
@ 2021-06-22  5:27     ` Wesley Cheng
  2021-06-22  6:05       ` Greg KH
  0 siblings, 1 reply; 73+ messages in thread
From: Wesley Cheng @ 2021-06-22  5:27 UTC (permalink / raw)
  To: Greg KH
  Cc: balbi, robh+dt, agross, bjorn.andersson, frowand.list, linux-usb,
	linux-kernel, devicetree, linux-arm-msm, jackp, fntoth,
	heikki.krogerus, andy.shevchenko



On 6/17/2021 4:07 AM, Greg KH wrote:
> On Thu, Jun 17, 2021 at 02:58:15AM -0700, Wesley Cheng wrote:
>> Ensure that the USB gadget is able to support the configuration being
>> added based on the number of endpoints required from all interfaces.  This
>> is for accounting for any bandwidth or space limitations.
>>
>> Signed-off-by: Wesley Cheng <wcheng@codeaurora.org>
>> ---
>>  drivers/usb/gadget/configfs.c | 22 ++++++++++++++++++++++
>>  1 file changed, 22 insertions(+)
>>
>> diff --git a/drivers/usb/gadget/configfs.c b/drivers/usb/gadget/configfs.c
>> index 15a607c..76b9983 100644
>> --- a/drivers/usb/gadget/configfs.c
>> +++ b/drivers/usb/gadget/configfs.c
>> @@ -1374,6 +1374,7 @@ static int configfs_composite_bind(struct usb_gadget *gadget,
>>  		struct usb_function *f;
>>  		struct usb_function *tmp;
>>  		struct gadget_config_name *cn;
>> +		unsigned long ep_map = 0;
>>  
>>  		if (gadget_is_otg(gadget))
>>  			c->descriptors = otg_desc;
>> @@ -1403,7 +1404,28 @@ static int configfs_composite_bind(struct usb_gadget *gadget,
>>  				list_add(&f->list, &cfg->func_list);
>>  				goto err_purge_funcs;
>>  			}
>> +			if (f->fs_descriptors) {
>> +				struct usb_descriptor_header **d;
>> +
>> +				d = f->fs_descriptors;
>> +				for (; *d; ++d) {

Hi Greg,

Thanks for the review and feedback.

> 
> With this check, there really is not a need to check for
> f->fs_descriptors above in the if statement, right?
> 

f->fs_descriptor will carry the table of descriptors that a particular
function driver has assigned to it.  The for loop here, will dereference
the individual descriptors within that descriptor array, so we need to
first ensure the descriptor array is present before traversing through
the individual entries/elements.

>> +					struct usb_endpoint_descriptor *ep;
>> +					int addr;
>> +
>> +					if ((*d)->bDescriptorType != USB_DT_ENDPOINT)
>> +						continue;
>> +
>> +					ep = (struct usb_endpoint_descriptor *)*d;
>> +					addr = ((ep->bEndpointAddress & 0x80) >> 3) |
>> +						(ep->bEndpointAddress & 0x0f);
> 
> Don't we have direction macros for this type of check?
> 

I don't believe we have a macro which would be able to convert the
bEndpointAddress field into the bit which needs to be set, assuming that
the 32bit ep_map has the lower 16bits carrying OUT EPs, and the upper
16bits carrying the IN EPs.

>> +					set_bit(addr, &ep_map);
>> +				}
> 
> What is this loop trying to do?  Please document it as I can not figure
> it out at all.
> 

Sure I can add some comments to this loop.  All its trying to do is to
set corresponding EPs being used (both IN and OUT eps) by a particular
USB configuration, then passing this bit mapped value to the DWC3 gadget
to evaluate.  As mentioned in the previous comment, the upper 16 bits
carries all IN eps used, and the lower 16 bits carries all the OUT eps used.

Thanks
Wesley Cheng

> thanks,
> 
> greg k-h
> 

-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 3/6] usb: dwc3: Resize TX FIFOs to meet EP bursting requirements
  2021-06-17 11:10   ` Greg KH
@ 2021-06-22  5:27     ` Wesley Cheng
  2021-06-22  6:06       ` Greg KH
  0 siblings, 1 reply; 73+ messages in thread
From: Wesley Cheng @ 2021-06-22  5:27 UTC (permalink / raw)
  To: Greg KH
  Cc: balbi, robh+dt, agross, bjorn.andersson, frowand.list, linux-usb,
	linux-kernel, devicetree, linux-arm-msm, jackp, fntoth,
	heikki.krogerus, andy.shevchenko



On 6/17/2021 4:10 AM, Greg KH wrote:
> On Thu, Jun 17, 2021 at 02:58:16AM -0700, Wesley Cheng wrote:
>> +static int dwc3_gadget_check_config(struct usb_gadget *g, unsigned long ep_map)
>> +{
>> +	struct dwc3 *dwc = gadget_to_dwc(g);
>> +	unsigned long in_ep_map;
>> +	int fifo_size = 0;
>> +	int ram1_depth;
>> +	int ep_num;
>> +
>> +	if (!dwc->do_fifo_resize)
>> +		return 0;
>> +
>> +	/* Only interested in the IN endpoints */
>> +	in_ep_map = ep_map >> 16;

Hi Greg,

> 
> Wait, this "map" is split up into 16/16 somehow?  So it's only 32bits
> big?
>

Yes, correct. Upper 16 carries IN eps, lower 16 carries OUT eps.  Will
fix that based off your other comment.

> Where did you document this map structure?  Why is it needed at all, you
> have the gadget, don't you have access to the full list of endpoints
> here as well?
> 
> confused,
> 

Unfortunately, we do not have the entire list of endpoints for each
function until the composite driver receives the SET_CONFIG packet from
the host.  By this time, if we incorrectly allowed the configuration to
start enumeration w/ the host, and there were some EPs which have no
FIFO memory allocated, that would lead to those interfaces/functions
being broken.

The USB UDC driver does have the in_epnum and out_epnum, but those are
currently only being incremented in case we don't match EPs by string.
(in usb_ep_autoconfig_ss())

Thanks
Wesley Cheng

> greg k-h
> 

-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 2/6] usb: gadget: configfs: Check USB configuration before adding
  2021-06-22  5:27     ` Wesley Cheng
@ 2021-06-22  6:05       ` Greg KH
  2021-06-23  9:38         ` Wesley Cheng
  0 siblings, 1 reply; 73+ messages in thread
From: Greg KH @ 2021-06-22  6:05 UTC (permalink / raw)
  To: Wesley Cheng
  Cc: balbi, robh+dt, agross, bjorn.andersson, frowand.list, linux-usb,
	linux-kernel, devicetree, linux-arm-msm, jackp, fntoth,
	heikki.krogerus, andy.shevchenko

On Mon, Jun 21, 2021 at 10:27:09PM -0700, Wesley Cheng wrote:
> 
> 
> On 6/17/2021 4:07 AM, Greg KH wrote:
> > On Thu, Jun 17, 2021 at 02:58:15AM -0700, Wesley Cheng wrote:
> >> Ensure that the USB gadget is able to support the configuration being
> >> added based on the number of endpoints required from all interfaces.  This
> >> is for accounting for any bandwidth or space limitations.
> >>
> >> Signed-off-by: Wesley Cheng <wcheng@codeaurora.org>
> >> ---
> >>  drivers/usb/gadget/configfs.c | 22 ++++++++++++++++++++++
> >>  1 file changed, 22 insertions(+)
> >>
> >> diff --git a/drivers/usb/gadget/configfs.c b/drivers/usb/gadget/configfs.c
> >> index 15a607c..76b9983 100644
> >> --- a/drivers/usb/gadget/configfs.c
> >> +++ b/drivers/usb/gadget/configfs.c
> >> @@ -1374,6 +1374,7 @@ static int configfs_composite_bind(struct usb_gadget *gadget,
> >>  		struct usb_function *f;
> >>  		struct usb_function *tmp;
> >>  		struct gadget_config_name *cn;
> >> +		unsigned long ep_map = 0;
> >>  
> >>  		if (gadget_is_otg(gadget))
> >>  			c->descriptors = otg_desc;
> >> @@ -1403,7 +1404,28 @@ static int configfs_composite_bind(struct usb_gadget *gadget,
> >>  				list_add(&f->list, &cfg->func_list);
> >>  				goto err_purge_funcs;
> >>  			}
> >> +			if (f->fs_descriptors) {
> >> +				struct usb_descriptor_header **d;
> >> +
> >> +				d = f->fs_descriptors;
> >> +				for (; *d; ++d) {
> 
> Hi Greg,
> 
> Thanks for the review and feedback.
> 
> > 
> > With this check, there really is not a need to check for
> > f->fs_descriptors above in the if statement, right?
> > 
> 
> f->fs_descriptor will carry the table of descriptors that a particular
> function driver has assigned to it.  The for loop here, will dereference
> the individual descriptors within that descriptor array, so we need to
> first ensure the descriptor array is present before traversing through
> the individual entries/elements.

Ah, it's a dereference of an array element.  Subtle.  Tricky.  Messy :(

> >> +					struct usb_endpoint_descriptor *ep;
> >> +					int addr;
> >> +
> >> +					if ((*d)->bDescriptorType != USB_DT_ENDPOINT)
> >> +						continue;
> >> +
> >> +					ep = (struct usb_endpoint_descriptor *)*d;
> >> +					addr = ((ep->bEndpointAddress & 0x80) >> 3) |
> >> +						(ep->bEndpointAddress & 0x0f);
> > 
> > Don't we have direction macros for this type of check?
> > 
> 
> I don't believe we have a macro which would be able to convert the
> bEndpointAddress field into the bit which needs to be set, assuming that
> the 32bit ep_map has the lower 16bits carrying OUT EPs, and the upper
> 16bits carrying the IN EPs.

We have macros to tell if an endpoint is IN or OUT, please use those.

And this "cram the whole thing into 64 bits" is not obvious at all.
Just pass around the original pointer to the descriptors if someone
wants to use it or not, don't make up yet-another-data-structure here
for no good reason.  We aren't so memory constrained we need to pack
stuff into bits here.

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 3/6] usb: dwc3: Resize TX FIFOs to meet EP bursting requirements
  2021-06-22  5:27     ` Wesley Cheng
@ 2021-06-22  6:06       ` Greg KH
  0 siblings, 0 replies; 73+ messages in thread
From: Greg KH @ 2021-06-22  6:06 UTC (permalink / raw)
  To: Wesley Cheng
  Cc: balbi, robh+dt, agross, bjorn.andersson, frowand.list, linux-usb,
	linux-kernel, devicetree, linux-arm-msm, jackp, fntoth,
	heikki.krogerus, andy.shevchenko

On Mon, Jun 21, 2021 at 10:27:13PM -0700, Wesley Cheng wrote:
> 
> 
> On 6/17/2021 4:10 AM, Greg KH wrote:
> > On Thu, Jun 17, 2021 at 02:58:16AM -0700, Wesley Cheng wrote:
> >> +static int dwc3_gadget_check_config(struct usb_gadget *g, unsigned long ep_map)
> >> +{
> >> +	struct dwc3 *dwc = gadget_to_dwc(g);
> >> +	unsigned long in_ep_map;
> >> +	int fifo_size = 0;
> >> +	int ram1_depth;
> >> +	int ep_num;
> >> +
> >> +	if (!dwc->do_fifo_resize)
> >> +		return 0;
> >> +
> >> +	/* Only interested in the IN endpoints */
> >> +	in_ep_map = ep_map >> 16;
> 
> Hi Greg,
> 
> > 
> > Wait, this "map" is split up into 16/16 somehow?  So it's only 32bits
> > big?
> >
> 
> Yes, correct. Upper 16 carries IN eps, lower 16 carries OUT eps.  Will
> fix that based off your other comment.
> 
> > Where did you document this map structure?  Why is it needed at all, you
> > have the gadget, don't you have access to the full list of endpoints
> > here as well?
> > 
> > confused,
> > 
> 
> Unfortunately, we do not have the entire list of endpoints for each
> function until the composite driver receives the SET_CONFIG packet from
> the host.  By this time, if we incorrectly allowed the configuration to
> start enumeration w/ the host, and there were some EPs which have no
> FIFO memory allocated, that would lead to those interfaces/functions
> being broken.

So how would this magic bitmap be any different here?

I'm not understanding why you need to create something new here to
represent data we already have.

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-06-19 12:40         ` Ferry Toth
@ 2021-06-22 18:38           ` Wesley Cheng
  2021-06-22 20:09             ` Ferry Toth
  2021-07-22 18:43             ` Ferry Toth
  0 siblings, 2 replies; 73+ messages in thread
From: Wesley Cheng @ 2021-06-22 18:38 UTC (permalink / raw)
  To: Ferry Toth, balbi, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list
  Cc: linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	heikki.krogerus, andy.shevchenko



On 6/19/2021 5:40 AM, Ferry Toth wrote:
> Hi
> 
> Op 18-06-2021 om 00:25 schreef Wesley Cheng:
>> Hi,
>>
>> On 6/17/2021 2:55 PM, Ferry Toth wrote:
>>> Hi
>>>
>>> Op 17-06-2021 om 23:48 schreef Wesley Cheng:
>>>> Hi,
>>>>
>>>> On 6/17/2021 2:01 PM, Ferry Toth wrote:
>>>>> Hi
>>>>>
>>>>> Op 17-06-2021 om 11:58 schreef Wesley Cheng:
>>>>>> Changes in V10:
>>>>>>    - Fixed compilation errors in config where OF is not used
>>>>>> (error due to
>>>>>>      unknown symbol for of_add_property()).  Add of_add_property()
>>>>>> stub.
>>>>>>    - Fixed compilation warning for incorrect argument being passed to
>>>>>> dwc3_mdwidth
>>>>> This fixes the OOPS I had in V9. I do not see any change in
>>>>> performance
>>>>> on Merrifield though.
>>>> I see...thanks Ferry! With your testing, are you writing to the
>>>> device's
>>>> internal storage (ie UFS, eMMC, etc...), or did you use a ramdisk as
>>>> well?
>>> In this case I just tested the EEM path using iperf3.
>>>
>> Got it.  I don't believe f_eem will use a high enough (if at all)
>> bMaxBurst value to change the TXFIFO size.
>>
>>>> If not with a ramdisk, we might want to give that a try to avoid the
>>>> storage path being the bottleneck.  You can use "dd" to create an empty
>>>> file, and then just use that as the LUN's backing file.
>>>>
>>>> echo ramdisk.img >
>>>> /sys/kernel/config/usb_gadget/g1/functions/mass_storage.0/lun.0/file
>>> Ah, why didn't I think of that. I have currently mass storage setup with
>>> eMMC but it seems that is indeed the bottleneck.
>>>
> I created a 64MB disk following the instructions here
> http://www.linux-usb.org/gadget/file_storage.html (that seems a little
> outdated, at least I can not start the first partition at sector 8, but
> minimum 2048), and added a test file on it.
> 
> I then copy the file to /dev/shm prior to setting configfs (composite
> device gser/eem/mass_storage/uac2).
> 
> journal shows:
> 
> kernel: Mass Storage Function, version: 2009/09/11
> kernel: LUN: removable file: (no medium)
> 
> I don't know what that means, because I see the test file on the ramdisk.
> 
> Then I again used gnome disks to benchmark (read/write 10MB):
> 
> With V10 on top v5.13.0-rc5:
> 
> R/W speed = 35.6/35.8MB/s, access time 0.35ms
> 
> With no patches on top v5.12.0:
> 
> R/W speed = 35.7/36.1MB/s, access time 0.35ms

Hi Ferry,

> 
> I see no speed difference (and it's about the same as with the eMMC
> backed disk). But the patches are causing a new call trace
> 

Would you happen to know what DWC3 controller revision the device is
using?  The callstack print occurs, because it looks like it ran out of
internal memory, although there should be logic present for making sure
that at least there is enough room for 1 FIFO per endpoint.  (possibly
the logic/math depends on the controller revision)

Also, is there a way to use just a mass storage only composition?  Based
on the above observation, that probably means that the mass storage
interface wasn't resized at all, because the configuration took up a lot
of the internal FIFO space.

Thanks
Wesley Cheng

> kernel: using random self ethernet address
> kernel: using random host ethernet address
> kernel: Mass Storage Function, version: 2009/09/11
> kernel: LUN: removable file: (no medium)
> kernel: usb0: HOST MAC aa:bb:cc:dd:ee:f2
> kernel: usb0: MAC aa:bb:cc:dd:ee:f1
> kernel: IPv6: ADDRCONF(NETDEV_CHANGE): usb0: link becomes ready
> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
> depth:115540359
> kernel: ------------[ cut here ]------------
> kernel: WARNING: CPU: 0 PID: 594 at drivers/usb/gadget/udc/core.c:278
> usb_ep_queue+0x75/0x80
> kernel: Modules linked in: usb_f_uac2 u_audio usb_f_mass_storage
> usb_f_eem u_ether usb_f_serial u_serial libcomposite rfcomm iptable_nat
> bnep snd_sof_nocodec spi_pxa2>
> kernel: CPU: 0 PID: 594 Comm: irq/14-dwc3 Not tainted
> 5.13.0-rc5-edison-acpi-standard #1
> kernel: Hardware name: Intel Corporation Merrifield/BODEGA BAY, BIOS 542
> 2015.01.21:18.19.48
> kernel: RIP: 0010:usb_ep_queue+0x75/0x80
> kernel: Code: 01 73 e4 48 8b 05 fb 63 06 01 48 85 c0 74 12 48 8b 78 08
> 44 89 e9 4c 89 e2 48 89 ee e8 74 05 00 00 44 89 e8 5d 41 5c 41 5d c3
> <0f> 0b 41 bd 94 ff ff ff >
> kernel: RSP: 0000:ffff91eec083fc98 EFLAGS: 00010082
> kernel: RAX: ffff8af20357d960 RBX: 0000000000000000 RCX: ffff8af202f06400
> kernel: RDX: 0000000000000a20 RSI: ffff8af208785780 RDI: ffff8af202e9ae00
> kernel: RBP: ffff8af202e9ae00 R08: 00000000000000c0 R09: ffff8af208785780
> kernel: R10: 00000000ffffe000 R11: 3fffffffffffffff R12: ffff8af208785780
> kernel: R13: 0000000000000000 R14: ffff8af202e9ae00 R15: ffff8af203e26cc0
> kernel: FS:  0000000000000000(0000) GS:ffff8af23e200000(0000)
> knlGS:0000000000000000
> kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> kernel: CR2: 000055e2c21f2100 CR3: 0000000003b38000 CR4: 00000000001006f0
> kernel: Call Trace:
> kernel:  u_audio_start_playback+0x107/0x1a0 [u_audio]
> kernel:  composite_setup+0x224/0x1ba0 [libcomposite]
> kernel:  ? dwc3_gadget_ep_queue+0xf6/0x1a0
> kernel:  ? usb_ep_queue+0x2a/0x80
> kernel:  ? configfs_composite_setup+0x6b/0x90 [libcomposite]
> kernel:  configfs_composite_setup+0x6b/0x90 [libcomposite]
> kernel:  dwc3_ep0_interrupt+0x469/0xa80
> kernel:  dwc3_thread_interrupt+0x8ee/0xf40
> kernel:  ? __wake_up_common_lock+0x85/0xb0
> kernel:  ? disable_irq_nosync+0x10/0x10
> kernel:  irq_thread_fn+0x1b/0x60
> kernel:  irq_thread+0xd6/0x170
> kernel:  ? irq_thread_check_affinity+0x70/0x70
> kernel:  ? irq_forced_thread_fn+0x70/0x70
> kernel:  kthread+0x116/0x130
> kernel:  ? kthread_create_worker_on_cpu+0x60/0x60
> kernel:  ret_from_fork+0x22/0x30
> kernel: ---[ end trace e5b9e28058c53584 ]---
> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
> kernel: dwc3 dwc3.0.auto: request 000000003c32dcc5 was not queued to ep5in
> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to ep5in
> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
> depth:115540359
> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to ep5in
> kernel: dwc3 dwc3.0.auto: request 00000000036ac129 was not queued to ep5in
> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
> depth:115540359
> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
> kernel: dwc3 dwc3.0.auto: request 00000000ad1b8c18 was not queued to ep5in
> kernel: dwc3 dwc3.0.auto: request 00000000fbc71244 was not queued to ep5in
> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
> depth:115540359
> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
> kernel: dwc3 dwc3.0.auto: request 00000000fbc71244 was not queued to ep5in
> kernel: dwc3 dwc3.0.auto: request 00000000ad1b8c18 was not queued to ep5in
> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
> depth:115540359
> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
> kernel: dwc3 dwc3.0.auto: request 000000003c32dcc5 was not queued to ep5in
> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to ep5in
> 
> Removing uac2 from the config makes the call trace go away, but the R/W
> speed does not change.
> 
>> :), not a problem...I've been working on getting the ideal set up for
>> the performance profiling for awhile, so anything I can do to make sure
>> we get some good results.
>>
>>> I'll try with a ramdisk and let you know.
>>>
>> Thanks again for the testing, Ferry.
>>
>> Thanks
>> Wesley Cheng
>>
>>>> Thanks
>>>> Wesley Cheng
>>>>
>>>>>> Changes in V9:
>>>>>>    - Fixed incorrect patch in series.  Removed changes in DTSI, as
>>>>>> dwc3-qcom will
>>>>>>      add the property by default from the kernel.
>>>>>>
>>>>>> Changes in V8:
>>>>>>    - Rebased to usb-testing
>>>>>>    - Using devm_kzalloc for adding txfifo property in dwc3-qcom
>>>>>>    - Removed DWC3 QCOM ACPI property for enabling the txfifo resize
>>>>>>
>>>>>> Changes in V7:
>>>>>>    - Added a new property tx-fifo-max-num for limiting how much fifo
>>>>>> space the
>>>>>>      resizing logic can allocate for endpoints with large burst
>>>>>> values.  This
>>>>>>      can differ across platforms, and tie in closely with overall
>>>>>> system latency.
>>>>>>    - Added recommended checks for DWC32.
>>>>>>    - Added changes to set the tx-fifo-resize property from
>>>>>> dwc3-qcom by
>>>>>> default
>>>>>>      instead of modifying the current DTSI files.
>>>>>>    - Added comments on all APIs/variables introduced.
>>>>>>    - Updated the DWC3 YAML to include a better description of the
>>>>>> tx-fifo-resize
>>>>>>      property and added an entry for tx-fifo-max-num.
>>>>>>
>>>>>> Changes in V6:
>>>>>>    - Rebased patches to usb-testing.
>>>>>>    - Renamed to PATCH series instead of RFC.
>>>>>>    - Checking for fs_descriptors instead of ss_descriptors for
>>>>>> determining the
>>>>>>      endpoint count for a particular configuration.
>>>>>>    - Re-ordered patch series to fix patch dependencies.
>>>>>>
>>>>>> Changes in V5:
>>>>>>    - Added check_config() logic, which is used to communicate the
>>>>>> number of EPs
>>>>>>      used in a particular configuration.  Based on this, the DWC3
>>>>>> gadget driver
>>>>>>      has the ability to know the maximum number of eps utilized in
>>>>>> all
>>>>>> configs.
>>>>>>      This helps reduce unnecessary allocation to unused eps, and will
>>>>>> catch fifo
>>>>>>      allocation issues at bind() time.
>>>>>>    - Fixed variable declaration to single line per variable, and
>>>>>> reverse xmas.
>>>>>>    - Created a helper for fifo clearing, which is used by ep0.c
>>>>>>
>>>>>> Changes in V4:
>>>>>>    - Removed struct dwc3* as an argument for
>>>>>> dwc3_gadget_resize_tx_fifos()
>>>>>>    - Removed WARN_ON(1) in case we run out of fifo space
>>>>>>    Changes in V3:
>>>>>>    - Removed "Reviewed-by" tags
>>>>>>    - Renamed series back to RFC
>>>>>>    - Modified logic to ensure that fifo_size is reset if we pass the
>>>>>> minimum
>>>>>>      threshold.  Tested with binding multiple FDs requesting 6 FIFOs.
>>>>>>
>>>>>> Changes in V2:
>>>>>>    - Modified TXFIFO resizing logic to ensure that each EP is
>>>>>> reserved a
>>>>>>      FIFO.
>>>>>>    - Removed dev_dbg() prints and fixed typos from patches
>>>>>>    - Added some more description on the dt-bindings commit message
>>>>>>
>>>>>> Currently, there is no functionality to allow for resizing the
>>>>>> TXFIFOs, and
>>>>>> relying on the HW default setting for the TXFIFO depth.  In most
>>>>>> cases, the
>>>>>> HW default is probably sufficient, but for USB compositions that
>>>>>> contain
>>>>>> multiple functions that require EP bursting, the default settings
>>>>>> might not be enough.  Also to note, the current SW will assign an
>>>>>> EP to a
>>>>>> function driver w/o checking to see if the TXFIFO size for that
>>>>>> particular
>>>>>> EP is large enough. (this is a problem if there are multiple HW
>>>>>> defined
>>>>>> values for the TXFIFO size)
>>>>>>
>>>>>> It is mentioned in the SNPS databook that a minimum of TX FIFO
>>>>>> depth = 3
>>>>>> is required for an EP that supports bursting.  Otherwise, there
>>>>>> may be
>>>>>> frequent occurences of bursts ending.  For high bandwidth functions,
>>>>>> such as data tethering (protocols that support data aggregation),
>>>>>> mass
>>>>>> storage, and media transfer protocol (over FFS), the bMaxBurst value
>>>>>> can be
>>>>>> large, and a bigger TXFIFO depth may prove to be beneficial in terms
>>>>>> of USB
>>>>>> throughput. (which can be associated to system access latency,
>>>>>> etc...)  It
>>>>>> allows for a more consistent burst of traffic, w/o any
>>>>>> interruptions, as
>>>>>> data is readily available in the FIFO.
>>>>>>
>>>>>> With testing done using the mass storage function driver, the results
>>>>>> show
>>>>>> that with a larger TXFIFO depth, the bandwidth increased
>>>>>> significantly.
>>>>>>
>>>>>> Test Parameters:
>>>>>>    - Platform: Qualcomm SM8150
>>>>>>    - bMaxBurst = 6
>>>>>>    - USB req size = 256kB
>>>>>>    - Num of USB reqs = 16
>>>>>>    - USB Speed = Super-Speed
>>>>>>    - Function Driver: Mass Storage (w/ ramdisk)
>>>>>>    - Test Application: CrystalDiskMark
>>>>>>
>>>>>> Results:
>>>>>>
>>>>>> TXFIFO Depth = 3 max packets
>>>>>>
>>>>>> Test Case | Data Size | AVG tput (in MB/s)
>>>>>> -------------------------------------------
>>>>>> Sequential|1 GB x     |
>>>>>> Read      |9 loops    | 193.60
>>>>>>        |           | 195.86
>>>>>>             |           | 184.77
>>>>>>             |           | 193.60
>>>>>> -------------------------------------------
>>>>>>
>>>>>> TXFIFO Depth = 6 max packets
>>>>>>
>>>>>> Test Case | Data Size | AVG tput (in MB/s)
>>>>>> -------------------------------------------
>>>>>> Sequential|1 GB x     |
>>>>>> Read      |9 loops    | 287.35
>>>>>>        |           | 304.94
>>>>>>             |           | 289.64
>>>>>>             |           | 293.61
>>>>>> -------------------------------------------
>>>>>>
>>>>>> Wesley Cheng (6):
>>>>>>     usb: gadget: udc: core: Introduce check_config to verify USB
>>>>>>       configuration
>>>>>>     usb: gadget: configfs: Check USB configuration before adding
>>>>>>     usb: dwc3: Resize TX FIFOs to meet EP bursting requirements
>>>>>>     of: Add stub for of_add_property()
>>>>>>     usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by default
>>>>>>     dt-bindings: usb: dwc3: Update dwc3 TX fifo properties
>>>>>>
>>>>>>    .../devicetree/bindings/usb/snps,dwc3.yaml         |  15 +-
>>>>>>    drivers/usb/dwc3/core.c                            |   9 +
>>>>>>    drivers/usb/dwc3/core.h                            |  15 ++
>>>>>>    drivers/usb/dwc3/dwc3-qcom.c                       |   9 +
>>>>>>    drivers/usb/dwc3/ep0.c                             |   2 +
>>>>>>    drivers/usb/dwc3/gadget.c                          | 212
>>>>>> +++++++++++++++++++++
>>>>>>    drivers/usb/gadget/configfs.c                      |  22 +++
>>>>>>    drivers/usb/gadget/udc/core.c                      |  25 +++
>>>>>>    include/linux/of.h                                 |   5 +
>>>>>>    include/linux/usb/gadget.h                         |   5 +
>>>>>>    10 files changed, 317 insertions(+), 2 deletions(-)
>>>>>>

-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-06-22 18:38           ` Wesley Cheng
@ 2021-06-22 20:09             ` Ferry Toth
  2021-06-22 22:00               ` Wesley Cheng
  2021-07-22 18:43             ` Ferry Toth
  1 sibling, 1 reply; 73+ messages in thread
From: Ferry Toth @ 2021-06-22 20:09 UTC (permalink / raw)
  To: Wesley Cheng, balbi, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list
  Cc: linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	heikki.krogerus, andy.shevchenko

Hi

Op 22-06-2021 om 20:38 schreef Wesley Cheng:
>
> On 6/19/2021 5:40 AM, Ferry Toth wrote:
>> Hi
>>
>> Op 18-06-2021 om 00:25 schreef Wesley Cheng:
>>> Hi,
>>>
>>> On 6/17/2021 2:55 PM, Ferry Toth wrote:
>>>> Hi
>>>>
>>>> Op 17-06-2021 om 23:48 schreef Wesley Cheng:
>>>>> Hi,
>>>>>
>>>>> On 6/17/2021 2:01 PM, Ferry Toth wrote:
>>>>>> Hi
>>>>>>
>>>>>> Op 17-06-2021 om 11:58 schreef Wesley Cheng:
>>>>>>> Changes in V10:
>>>>>>>     - Fixed compilation errors in config where OF is not used
>>>>>>> (error due to
>>>>>>>       unknown symbol for of_add_property()).  Add of_add_property()
>>>>>>> stub.
>>>>>>>     - Fixed compilation warning for incorrect argument being passed to
>>>>>>> dwc3_mdwidth
>>>>>> This fixes the OOPS I had in V9. I do not see any change in
>>>>>> performance
>>>>>> on Merrifield though.
>>>>> I see...thanks Ferry! With your testing, are you writing to the
>>>>> device's
>>>>> internal storage (ie UFS, eMMC, etc...), or did you use a ramdisk as
>>>>> well?
>>>> In this case I just tested the EEM path using iperf3.
>>>>
>>> Got it.  I don't believe f_eem will use a high enough (if at all)
>>> bMaxBurst value to change the TXFIFO size.
>>>
>>>>> If not with a ramdisk, we might want to give that a try to avoid the
>>>>> storage path being the bottleneck.  You can use "dd" to create an empty
>>>>> file, and then just use that as the LUN's backing file.
>>>>>
>>>>> echo ramdisk.img >
>>>>> /sys/kernel/config/usb_gadget/g1/functions/mass_storage.0/lun.0/file
>>>> Ah, why didn't I think of that. I have currently mass storage setup with
>>>> eMMC but it seems that is indeed the bottleneck.
>>>>
>> I created a 64MB disk following the instructions here
>> http://www.linux-usb.org/gadget/file_storage.html (that seems a little
>> outdated, at least I can not start the first partition at sector 8, but
>> minimum 2048), and added a test file on it.
>>
>> I then copy the file to /dev/shm prior to setting configfs (composite
>> device gser/eem/mass_storage/uac2).
>>
>> journal shows:
>>
>> kernel: Mass Storage Function, version: 2009/09/11
>> kernel: LUN: removable file: (no medium)
>>
>> I don't know what that means, because I see the test file on the ramdisk.
>>
>> Then I again used gnome disks to benchmark (read/write 10MB):
>>
>> With V10 on top v5.13.0-rc5:
>>
>> R/W speed = 35.6/35.8MB/s, access time 0.35ms
>>
>> With no patches on top v5.12.0:
>>
>> R/W speed = 35.7/36.1MB/s, access time 0.35ms
> Hi Ferry,
>
>> I see no speed difference (and it's about the same as with the eMMC
>> backed disk). But the patches are causing a new call trace
>>
> Would you happen to know what DWC3 controller revision the device is
> using?  The callstack print occurs, because it looks like it ran out of
> internal memory, although there should be logic present for making sure
> that at least there is enough room for 1 FIFO per endpoint.  (possibly
> the logic/math depends on the controller revision)

Do you know where I could find that in a file on the device?

Otherwise, I'm hoping Andy will know?

>
> Also, is there a way to use just a mass storage only composition?  Based
> on the above observation, that probably means that the mass storage
> interface wasn't resized at all, because the configuration took up a lot
> of the internal FIFO space.

Sure, it's configured through configfs. With only mass_storage I have:

With V10 on top v5.13.0-rc5:

R/W speed = 41,6/39,3MB/s, access time 0.33ms

With no patches on top v5.12.0:

R/W speed = 41,1/38,7MB/s, access time 0.38ms

> Thanks
> Wesley Cheng
>
>> kernel: using random self ethernet address
>> kernel: using random host ethernet address
>> kernel: Mass Storage Function, version: 2009/09/11
>> kernel: LUN: removable file: (no medium)
>> kernel: usb0: HOST MAC aa:bb:cc:dd:ee:f2
>> kernel: usb0: MAC aa:bb:cc:dd:ee:f1
>> kernel: IPv6: ADDRCONF(NETDEV_CHANGE): usb0: link becomes ready
>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>> depth:115540359
>> kernel: ------------[ cut here ]------------
>> kernel: WARNING: CPU: 0 PID: 594 at drivers/usb/gadget/udc/core.c:278
>> usb_ep_queue+0x75/0x80
>> kernel: Modules linked in: usb_f_uac2 u_audio usb_f_mass_storage
>> usb_f_eem u_ether usb_f_serial u_serial libcomposite rfcomm iptable_nat
>> bnep snd_sof_nocodec spi_pxa2>
>> kernel: CPU: 0 PID: 594 Comm: irq/14-dwc3 Not tainted
>> 5.13.0-rc5-edison-acpi-standard #1
>> kernel: Hardware name: Intel Corporation Merrifield/BODEGA BAY, BIOS 542
>> 2015.01.21:18.19.48
>> kernel: RIP: 0010:usb_ep_queue+0x75/0x80
>> kernel: Code: 01 73 e4 48 8b 05 fb 63 06 01 48 85 c0 74 12 48 8b 78 08
>> 44 89 e9 4c 89 e2 48 89 ee e8 74 05 00 00 44 89 e8 5d 41 5c 41 5d c3
>> <0f> 0b 41 bd 94 ff ff ff >
>> kernel: RSP: 0000:ffff91eec083fc98 EFLAGS: 00010082
>> kernel: RAX: ffff8af20357d960 RBX: 0000000000000000 RCX: ffff8af202f06400
>> kernel: RDX: 0000000000000a20 RSI: ffff8af208785780 RDI: ffff8af202e9ae00
>> kernel: RBP: ffff8af202e9ae00 R08: 00000000000000c0 R09: ffff8af208785780
>> kernel: R10: 00000000ffffe000 R11: 3fffffffffffffff R12: ffff8af208785780
>> kernel: R13: 0000000000000000 R14: ffff8af202e9ae00 R15: ffff8af203e26cc0
>> kernel: FS:  0000000000000000(0000) GS:ffff8af23e200000(0000)
>> knlGS:0000000000000000
>> kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> kernel: CR2: 000055e2c21f2100 CR3: 0000000003b38000 CR4: 00000000001006f0
>> kernel: Call Trace:
>> kernel:  u_audio_start_playback+0x107/0x1a0 [u_audio]
>> kernel:  composite_setup+0x224/0x1ba0 [libcomposite]
>> kernel:  ? dwc3_gadget_ep_queue+0xf6/0x1a0
>> kernel:  ? usb_ep_queue+0x2a/0x80
>> kernel:  ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>> kernel:  configfs_composite_setup+0x6b/0x90 [libcomposite]
>> kernel:  dwc3_ep0_interrupt+0x469/0xa80
>> kernel:  dwc3_thread_interrupt+0x8ee/0xf40
>> kernel:  ? __wake_up_common_lock+0x85/0xb0
>> kernel:  ? disable_irq_nosync+0x10/0x10
>> kernel:  irq_thread_fn+0x1b/0x60
>> kernel:  irq_thread+0xd6/0x170
>> kernel:  ? irq_thread_check_affinity+0x70/0x70
>> kernel:  ? irq_forced_thread_fn+0x70/0x70
>> kernel:  kthread+0x116/0x130
>> kernel:  ? kthread_create_worker_on_cpu+0x60/0x60
>> kernel:  ret_from_fork+0x22/0x30
>> kernel: ---[ end trace e5b9e28058c53584 ]---
>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>> kernel: dwc3 dwc3.0.auto: request 000000003c32dcc5 was not queued to ep5in
>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to ep5in
>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>> depth:115540359
>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to ep5in
>> kernel: dwc3 dwc3.0.auto: request 00000000036ac129 was not queued to ep5in
>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>> depth:115540359
>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>> kernel: dwc3 dwc3.0.auto: request 00000000ad1b8c18 was not queued to ep5in
>> kernel: dwc3 dwc3.0.auto: request 00000000fbc71244 was not queued to ep5in
>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>> depth:115540359
>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>> kernel: dwc3 dwc3.0.auto: request 00000000fbc71244 was not queued to ep5in
>> kernel: dwc3 dwc3.0.auto: request 00000000ad1b8c18 was not queued to ep5in
>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>> depth:115540359
>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>> kernel: dwc3 dwc3.0.auto: request 000000003c32dcc5 was not queued to ep5in
>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to ep5in
>>
>> Removing uac2 from the config makes the call trace go away, but the R/W
>> speed does not change.
>>
>>> :), not a problem...I've been working on getting the ideal set up for
>>> the performance profiling for awhile, so anything I can do to make sure
>>> we get some good results.
>>>
>>>> I'll try with a ramdisk and let you know.
>>>>
>>> Thanks again for the testing, Ferry.
>>>
>>> Thanks
>>> Wesley Cheng
>>>
>>>>> Thanks
>>>>> Wesley Cheng
>>>>>
>>>>>>> Changes in V9:
>>>>>>>     - Fixed incorrect patch in series.  Removed changes in DTSI, as
>>>>>>> dwc3-qcom will
>>>>>>>       add the property by default from the kernel.
>>>>>>>
>>>>>>> Changes in V8:
>>>>>>>     - Rebased to usb-testing
>>>>>>>     - Using devm_kzalloc for adding txfifo property in dwc3-qcom
>>>>>>>     - Removed DWC3 QCOM ACPI property for enabling the txfifo resize
>>>>>>>
>>>>>>> Changes in V7:
>>>>>>>     - Added a new property tx-fifo-max-num for limiting how much fifo
>>>>>>> space the
>>>>>>>       resizing logic can allocate for endpoints with large burst
>>>>>>> values.  This
>>>>>>>       can differ across platforms, and tie in closely with overall
>>>>>>> system latency.
>>>>>>>     - Added recommended checks for DWC32.
>>>>>>>     - Added changes to set the tx-fifo-resize property from
>>>>>>> dwc3-qcom by
>>>>>>> default
>>>>>>>       instead of modifying the current DTSI files.
>>>>>>>     - Added comments on all APIs/variables introduced.
>>>>>>>     - Updated the DWC3 YAML to include a better description of the
>>>>>>> tx-fifo-resize
>>>>>>>       property and added an entry for tx-fifo-max-num.
>>>>>>>
>>>>>>> Changes in V6:
>>>>>>>     - Rebased patches to usb-testing.
>>>>>>>     - Renamed to PATCH series instead of RFC.
>>>>>>>     - Checking for fs_descriptors instead of ss_descriptors for
>>>>>>> determining the
>>>>>>>       endpoint count for a particular configuration.
>>>>>>>     - Re-ordered patch series to fix patch dependencies.
>>>>>>>
>>>>>>> Changes in V5:
>>>>>>>     - Added check_config() logic, which is used to communicate the
>>>>>>> number of EPs
>>>>>>>       used in a particular configuration.  Based on this, the DWC3
>>>>>>> gadget driver
>>>>>>>       has the ability to know the maximum number of eps utilized in
>>>>>>> all
>>>>>>> configs.
>>>>>>>       This helps reduce unnecessary allocation to unused eps, and will
>>>>>>> catch fifo
>>>>>>>       allocation issues at bind() time.
>>>>>>>     - Fixed variable declaration to single line per variable, and
>>>>>>> reverse xmas.
>>>>>>>     - Created a helper for fifo clearing, which is used by ep0.c
>>>>>>>
>>>>>>> Changes in V4:
>>>>>>>     - Removed struct dwc3* as an argument for
>>>>>>> dwc3_gadget_resize_tx_fifos()
>>>>>>>     - Removed WARN_ON(1) in case we run out of fifo space
>>>>>>>     Changes in V3:
>>>>>>>     - Removed "Reviewed-by" tags
>>>>>>>     - Renamed series back to RFC
>>>>>>>     - Modified logic to ensure that fifo_size is reset if we pass the
>>>>>>> minimum
>>>>>>>       threshold.  Tested with binding multiple FDs requesting 6 FIFOs.
>>>>>>>
>>>>>>> Changes in V2:
>>>>>>>     - Modified TXFIFO resizing logic to ensure that each EP is
>>>>>>> reserved a
>>>>>>>       FIFO.
>>>>>>>     - Removed dev_dbg() prints and fixed typos from patches
>>>>>>>     - Added some more description on the dt-bindings commit message
>>>>>>>
>>>>>>> Currently, there is no functionality to allow for resizing the
>>>>>>> TXFIFOs, and
>>>>>>> relying on the HW default setting for the TXFIFO depth.  In most
>>>>>>> cases, the
>>>>>>> HW default is probably sufficient, but for USB compositions that
>>>>>>> contain
>>>>>>> multiple functions that require EP bursting, the default settings
>>>>>>> might not be enough.  Also to note, the current SW will assign an
>>>>>>> EP to a
>>>>>>> function driver w/o checking to see if the TXFIFO size for that
>>>>>>> particular
>>>>>>> EP is large enough. (this is a problem if there are multiple HW
>>>>>>> defined
>>>>>>> values for the TXFIFO size)
>>>>>>>
>>>>>>> It is mentioned in the SNPS databook that a minimum of TX FIFO
>>>>>>> depth = 3
>>>>>>> is required for an EP that supports bursting.  Otherwise, there
>>>>>>> may be
>>>>>>> frequent occurences of bursts ending.  For high bandwidth functions,
>>>>>>> such as data tethering (protocols that support data aggregation),
>>>>>>> mass
>>>>>>> storage, and media transfer protocol (over FFS), the bMaxBurst value
>>>>>>> can be
>>>>>>> large, and a bigger TXFIFO depth may prove to be beneficial in terms
>>>>>>> of USB
>>>>>>> throughput. (which can be associated to system access latency,
>>>>>>> etc...)  It
>>>>>>> allows for a more consistent burst of traffic, w/o any
>>>>>>> interruptions, as
>>>>>>> data is readily available in the FIFO.
>>>>>>>
>>>>>>> With testing done using the mass storage function driver, the results
>>>>>>> show
>>>>>>> that with a larger TXFIFO depth, the bandwidth increased
>>>>>>> significantly.
>>>>>>>
>>>>>>> Test Parameters:
>>>>>>>     - Platform: Qualcomm SM8150
>>>>>>>     - bMaxBurst = 6
>>>>>>>     - USB req size = 256kB
>>>>>>>     - Num of USB reqs = 16
>>>>>>>     - USB Speed = Super-Speed
>>>>>>>     - Function Driver: Mass Storage (w/ ramdisk)
>>>>>>>     - Test Application: CrystalDiskMark
>>>>>>>
>>>>>>> Results:
>>>>>>>
>>>>>>> TXFIFO Depth = 3 max packets
>>>>>>>
>>>>>>> Test Case | Data Size | AVG tput (in MB/s)
>>>>>>> -------------------------------------------
>>>>>>> Sequential|1 GB x     |
>>>>>>> Read      |9 loops    | 193.60
>>>>>>>         |           | 195.86
>>>>>>>              |           | 184.77
>>>>>>>              |           | 193.60
>>>>>>> -------------------------------------------
>>>>>>>
>>>>>>> TXFIFO Depth = 6 max packets
>>>>>>>
>>>>>>> Test Case | Data Size | AVG tput (in MB/s)
>>>>>>> -------------------------------------------
>>>>>>> Sequential|1 GB x     |
>>>>>>> Read      |9 loops    | 287.35
>>>>>>>         |           | 304.94
>>>>>>>              |           | 289.64
>>>>>>>              |           | 293.61
>>>>>>> -------------------------------------------
>>>>>>>
>>>>>>> Wesley Cheng (6):
>>>>>>>      usb: gadget: udc: core: Introduce check_config to verify USB
>>>>>>>        configuration
>>>>>>>      usb: gadget: configfs: Check USB configuration before adding
>>>>>>>      usb: dwc3: Resize TX FIFOs to meet EP bursting requirements
>>>>>>>      of: Add stub for of_add_property()
>>>>>>>      usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by default
>>>>>>>      dt-bindings: usb: dwc3: Update dwc3 TX fifo properties
>>>>>>>
>>>>>>>     .../devicetree/bindings/usb/snps,dwc3.yaml         |  15 +-
>>>>>>>     drivers/usb/dwc3/core.c                            |   9 +
>>>>>>>     drivers/usb/dwc3/core.h                            |  15 ++
>>>>>>>     drivers/usb/dwc3/dwc3-qcom.c                       |   9 +
>>>>>>>     drivers/usb/dwc3/ep0.c                             |   2 +
>>>>>>>     drivers/usb/dwc3/gadget.c                          | 212
>>>>>>> +++++++++++++++++++++
>>>>>>>     drivers/usb/gadget/configfs.c                      |  22 +++
>>>>>>>     drivers/usb/gadget/udc/core.c                      |  25 +++
>>>>>>>     include/linux/of.h                                 |   5 +
>>>>>>>     include/linux/usb/gadget.h                         |   5 +
>>>>>>>     10 files changed, 317 insertions(+), 2 deletions(-)
>>>>>>>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-06-22 20:09             ` Ferry Toth
@ 2021-06-22 22:00               ` Wesley Cheng
  2021-06-22 22:12                 ` Ferry Toth
  0 siblings, 1 reply; 73+ messages in thread
From: Wesley Cheng @ 2021-06-22 22:00 UTC (permalink / raw)
  To: Ferry Toth, balbi, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list
  Cc: linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	heikki.krogerus, andy.shevchenko



On 6/22/2021 1:09 PM, Ferry Toth wrote:
> Hi
> 
> Op 22-06-2021 om 20:38 schreef Wesley Cheng:
>>
>> On 6/19/2021 5:40 AM, Ferry Toth wrote:
>>> Hi
>>>
>>> Op 18-06-2021 om 00:25 schreef Wesley Cheng:
>>>> Hi,
>>>>
>>>> On 6/17/2021 2:55 PM, Ferry Toth wrote:
>>>>> Hi
>>>>>
>>>>> Op 17-06-2021 om 23:48 schreef Wesley Cheng:
>>>>>> Hi,
>>>>>>
>>>>>> On 6/17/2021 2:01 PM, Ferry Toth wrote:
>>>>>>> Hi
>>>>>>>
>>>>>>> Op 17-06-2021 om 11:58 schreef Wesley Cheng:
>>>>>>>> Changes in V10:
>>>>>>>>     - Fixed compilation errors in config where OF is not used
>>>>>>>> (error due to
>>>>>>>>       unknown symbol for of_add_property()).  Add of_add_property()
>>>>>>>> stub.
>>>>>>>>     - Fixed compilation warning for incorrect argument being
>>>>>>>> passed to
>>>>>>>> dwc3_mdwidth
>>>>>>> This fixes the OOPS I had in V9. I do not see any change in
>>>>>>> performance
>>>>>>> on Merrifield though.
>>>>>> I see...thanks Ferry! With your testing, are you writing to the
>>>>>> device's
>>>>>> internal storage (ie UFS, eMMC, etc...), or did you use a ramdisk as
>>>>>> well?
>>>>> In this case I just tested the EEM path using iperf3.
>>>>>
>>>> Got it.  I don't believe f_eem will use a high enough (if at all)
>>>> bMaxBurst value to change the TXFIFO size.
>>>>
>>>>>> If not with a ramdisk, we might want to give that a try to avoid the
>>>>>> storage path being the bottleneck.  You can use "dd" to create an
>>>>>> empty
>>>>>> file, and then just use that as the LUN's backing file.
>>>>>>
>>>>>> echo ramdisk.img >
>>>>>> /sys/kernel/config/usb_gadget/g1/functions/mass_storage.0/lun.0/file
>>>>> Ah, why didn't I think of that. I have currently mass storage setup
>>>>> with
>>>>> eMMC but it seems that is indeed the bottleneck.
>>>>>
>>> I created a 64MB disk following the instructions here
>>> http://www.linux-usb.org/gadget/file_storage.html (that seems a little
>>> outdated, at least I can not start the first partition at sector 8, but
>>> minimum 2048), and added a test file on it.
>>>
>>> I then copy the file to /dev/shm prior to setting configfs (composite
>>> device gser/eem/mass_storage/uac2).
>>>
>>> journal shows:
>>>
>>> kernel: Mass Storage Function, version: 2009/09/11
>>> kernel: LUN: removable file: (no medium)
>>>
>>> I don't know what that means, because I see the test file on the
>>> ramdisk.
>>>
>>> Then I again used gnome disks to benchmark (read/write 10MB):
>>>
>>> With V10 on top v5.13.0-rc5:
>>>
>>> R/W speed = 35.6/35.8MB/s, access time 0.35ms
>>>
>>> With no patches on top v5.12.0:
>>>
>>> R/W speed = 35.7/36.1MB/s, access time 0.35ms
>> Hi Ferry,
>>
>>> I see no speed difference (and it's about the same as with the eMMC
>>> backed disk). But the patches are causing a new call trace
>>>
>> Would you happen to know what DWC3 controller revision the device is
>> using?  The callstack print occurs, because it looks like it ran out of
>> internal memory, although there should be logic present for making sure
>> that at least there is enough room for 1 FIFO per endpoint.  (possibly
>> the logic/math depends on the controller revision)

Hi Ferry,

> 
> Do you know where I could find that in a file on the device?
> 

Maybe you can just dump the DWC3 registers?
cat /sys/kernel/debug/usb/<controller name>/regdump

Was going to ask for the same to confirm the TXFIFO sizes from your
results below :).

> Otherwise, I'm hoping Andy will know?
> 
>>
>> Also, is there a way to use just a mass storage only composition?  Based
>> on the above observation, that probably means that the mass storage
>> interface wasn't resized at all, because the configuration took up a lot
>> of the internal FIFO space.
> 
> Sure, it's configured through configfs. With only mass_storage I have:
> 
> With V10 on top v5.13.0-rc5:
> 
> R/W speed = 41,6/39,3MB/s, access time 0.33ms
> 
> With no patches on top v5.12.0:
> 
> R/W speed = 41,1/38,7MB/s, access time 0.38ms
> 

Thanks Ferry!  Could you collect the regdump, so I can confirm the two
things mentioned?

Thanks
Wesley Cheng

>> Thanks
>> Wesley Cheng
>>
>>> kernel: using random self ethernet address
>>> kernel: using random host ethernet address
>>> kernel: Mass Storage Function, version: 2009/09/11
>>> kernel: LUN: removable file: (no medium)
>>> kernel: usb0: HOST MAC aa:bb:cc:dd:ee:f2
>>> kernel: usb0: MAC aa:bb:cc:dd:ee:f1
>>> kernel: IPv6: ADDRCONF(NETDEV_CHANGE): usb0: link becomes ready
>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>> depth:115540359
>>> kernel: ------------[ cut here ]------------
>>> kernel: WARNING: CPU: 0 PID: 594 at drivers/usb/gadget/udc/core.c:278
>>> usb_ep_queue+0x75/0x80
>>> kernel: Modules linked in: usb_f_uac2 u_audio usb_f_mass_storage
>>> usb_f_eem u_ether usb_f_serial u_serial libcomposite rfcomm iptable_nat
>>> bnep snd_sof_nocodec spi_pxa2>
>>> kernel: CPU: 0 PID: 594 Comm: irq/14-dwc3 Not tainted
>>> 5.13.0-rc5-edison-acpi-standard #1
>>> kernel: Hardware name: Intel Corporation Merrifield/BODEGA BAY, BIOS 542
>>> 2015.01.21:18.19.48
>>> kernel: RIP: 0010:usb_ep_queue+0x75/0x80
>>> kernel: Code: 01 73 e4 48 8b 05 fb 63 06 01 48 85 c0 74 12 48 8b 78 08
>>> 44 89 e9 4c 89 e2 48 89 ee e8 74 05 00 00 44 89 e8 5d 41 5c 41 5d c3
>>> <0f> 0b 41 bd 94 ff ff ff >
>>> kernel: RSP: 0000:ffff91eec083fc98 EFLAGS: 00010082
>>> kernel: RAX: ffff8af20357d960 RBX: 0000000000000000 RCX:
>>> ffff8af202f06400
>>> kernel: RDX: 0000000000000a20 RSI: ffff8af208785780 RDI:
>>> ffff8af202e9ae00
>>> kernel: RBP: ffff8af202e9ae00 R08: 00000000000000c0 R09:
>>> ffff8af208785780
>>> kernel: R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
>>> ffff8af208785780
>>> kernel: R13: 0000000000000000 R14: ffff8af202e9ae00 R15:
>>> ffff8af203e26cc0
>>> kernel: FS:  0000000000000000(0000) GS:ffff8af23e200000(0000)
>>> knlGS:0000000000000000
>>> kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> kernel: CR2: 000055e2c21f2100 CR3: 0000000003b38000 CR4:
>>> 00000000001006f0
>>> kernel: Call Trace:
>>> kernel:  u_audio_start_playback+0x107/0x1a0 [u_audio]
>>> kernel:  composite_setup+0x224/0x1ba0 [libcomposite]
>>> kernel:  ? dwc3_gadget_ep_queue+0xf6/0x1a0
>>> kernel:  ? usb_ep_queue+0x2a/0x80
>>> kernel:  ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>> kernel:  configfs_composite_setup+0x6b/0x90 [libcomposite]
>>> kernel:  dwc3_ep0_interrupt+0x469/0xa80
>>> kernel:  dwc3_thread_interrupt+0x8ee/0xf40
>>> kernel:  ? __wake_up_common_lock+0x85/0xb0
>>> kernel:  ? disable_irq_nosync+0x10/0x10
>>> kernel:  irq_thread_fn+0x1b/0x60
>>> kernel:  irq_thread+0xd6/0x170
>>> kernel:  ? irq_thread_check_affinity+0x70/0x70
>>> kernel:  ? irq_forced_thread_fn+0x70/0x70
>>> kernel:  kthread+0x116/0x130
>>> kernel:  ? kthread_create_worker_on_cpu+0x60/0x60
>>> kernel:  ret_from_fork+0x22/0x30
>>> kernel: ---[ end trace e5b9e28058c53584 ]---
>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>> kernel: dwc3 dwc3.0.auto: request 000000003c32dcc5 was not queued to
>>> ep5in
>>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to
>>> ep5in
>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>> depth:115540359
>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to
>>> ep5in
>>> kernel: dwc3 dwc3.0.auto: request 00000000036ac129 was not queued to
>>> ep5in
>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>> depth:115540359
>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>> kernel: dwc3 dwc3.0.auto: request 00000000ad1b8c18 was not queued to
>>> ep5in
>>> kernel: dwc3 dwc3.0.auto: request 00000000fbc71244 was not queued to
>>> ep5in
>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>> depth:115540359
>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>> kernel: dwc3 dwc3.0.auto: request 00000000fbc71244 was not queued to
>>> ep5in
>>> kernel: dwc3 dwc3.0.auto: request 00000000ad1b8c18 was not queued to
>>> ep5in
>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>> depth:115540359
>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>> kernel: dwc3 dwc3.0.auto: request 000000003c32dcc5 was not queued to
>>> ep5in
>>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to
>>> ep5in
>>>
>>> Removing uac2 from the config makes the call trace go away, but the R/W
>>> speed does not change.
>>>
>>>> :), not a problem...I've been working on getting the ideal set up for
>>>> the performance profiling for awhile, so anything I can do to make sure
>>>> we get some good results.
>>>>
>>>>> I'll try with a ramdisk and let you know.
>>>>>
>>>> Thanks again for the testing, Ferry.
>>>>
>>>> Thanks
>>>> Wesley Cheng
>>>>
>>>>>> Thanks
>>>>>> Wesley Cheng
>>>>>>
>>>>>>>> Changes in V9:
>>>>>>>>     - Fixed incorrect patch in series.  Removed changes in DTSI, as
>>>>>>>> dwc3-qcom will
>>>>>>>>       add the property by default from the kernel.
>>>>>>>>
>>>>>>>> Changes in V8:
>>>>>>>>     - Rebased to usb-testing
>>>>>>>>     - Using devm_kzalloc for adding txfifo property in dwc3-qcom
>>>>>>>>     - Removed DWC3 QCOM ACPI property for enabling the txfifo
>>>>>>>> resize
>>>>>>>>
>>>>>>>> Changes in V7:
>>>>>>>>     - Added a new property tx-fifo-max-num for limiting how much
>>>>>>>> fifo
>>>>>>>> space the
>>>>>>>>       resizing logic can allocate for endpoints with large burst
>>>>>>>> values.  This
>>>>>>>>       can differ across platforms, and tie in closely with overall
>>>>>>>> system latency.
>>>>>>>>     - Added recommended checks for DWC32.
>>>>>>>>     - Added changes to set the tx-fifo-resize property from
>>>>>>>> dwc3-qcom by
>>>>>>>> default
>>>>>>>>       instead of modifying the current DTSI files.
>>>>>>>>     - Added comments on all APIs/variables introduced.
>>>>>>>>     - Updated the DWC3 YAML to include a better description of the
>>>>>>>> tx-fifo-resize
>>>>>>>>       property and added an entry for tx-fifo-max-num.
>>>>>>>>
>>>>>>>> Changes in V6:
>>>>>>>>     - Rebased patches to usb-testing.
>>>>>>>>     - Renamed to PATCH series instead of RFC.
>>>>>>>>     - Checking for fs_descriptors instead of ss_descriptors for
>>>>>>>> determining the
>>>>>>>>       endpoint count for a particular configuration.
>>>>>>>>     - Re-ordered patch series to fix patch dependencies.
>>>>>>>>
>>>>>>>> Changes in V5:
>>>>>>>>     - Added check_config() logic, which is used to communicate the
>>>>>>>> number of EPs
>>>>>>>>       used in a particular configuration.  Based on this, the DWC3
>>>>>>>> gadget driver
>>>>>>>>       has the ability to know the maximum number of eps utilized in
>>>>>>>> all
>>>>>>>> configs.
>>>>>>>>       This helps reduce unnecessary allocation to unused eps,
>>>>>>>> and will
>>>>>>>> catch fifo
>>>>>>>>       allocation issues at bind() time.
>>>>>>>>     - Fixed variable declaration to single line per variable, and
>>>>>>>> reverse xmas.
>>>>>>>>     - Created a helper for fifo clearing, which is used by ep0.c
>>>>>>>>
>>>>>>>> Changes in V4:
>>>>>>>>     - Removed struct dwc3* as an argument for
>>>>>>>> dwc3_gadget_resize_tx_fifos()
>>>>>>>>     - Removed WARN_ON(1) in case we run out of fifo space
>>>>>>>>     Changes in V3:
>>>>>>>>     - Removed "Reviewed-by" tags
>>>>>>>>     - Renamed series back to RFC
>>>>>>>>     - Modified logic to ensure that fifo_size is reset if we
>>>>>>>> pass the
>>>>>>>> minimum
>>>>>>>>       threshold.  Tested with binding multiple FDs requesting 6
>>>>>>>> FIFOs.
>>>>>>>>
>>>>>>>> Changes in V2:
>>>>>>>>     - Modified TXFIFO resizing logic to ensure that each EP is
>>>>>>>> reserved a
>>>>>>>>       FIFO.
>>>>>>>>     - Removed dev_dbg() prints and fixed typos from patches
>>>>>>>>     - Added some more description on the dt-bindings commit message
>>>>>>>>
>>>>>>>> Currently, there is no functionality to allow for resizing the
>>>>>>>> TXFIFOs, and
>>>>>>>> relying on the HW default setting for the TXFIFO depth.  In most
>>>>>>>> cases, the
>>>>>>>> HW default is probably sufficient, but for USB compositions that
>>>>>>>> contain
>>>>>>>> multiple functions that require EP bursting, the default settings
>>>>>>>> might not be enough.  Also to note, the current SW will assign an
>>>>>>>> EP to a
>>>>>>>> function driver w/o checking to see if the TXFIFO size for that
>>>>>>>> particular
>>>>>>>> EP is large enough. (this is a problem if there are multiple HW
>>>>>>>> defined
>>>>>>>> values for the TXFIFO size)
>>>>>>>>
>>>>>>>> It is mentioned in the SNPS databook that a minimum of TX FIFO
>>>>>>>> depth = 3
>>>>>>>> is required for an EP that supports bursting.  Otherwise, there
>>>>>>>> may be
>>>>>>>> frequent occurences of bursts ending.  For high bandwidth
>>>>>>>> functions,
>>>>>>>> such as data tethering (protocols that support data aggregation),
>>>>>>>> mass
>>>>>>>> storage, and media transfer protocol (over FFS), the bMaxBurst
>>>>>>>> value
>>>>>>>> can be
>>>>>>>> large, and a bigger TXFIFO depth may prove to be beneficial in
>>>>>>>> terms
>>>>>>>> of USB
>>>>>>>> throughput. (which can be associated to system access latency,
>>>>>>>> etc...)  It
>>>>>>>> allows for a more consistent burst of traffic, w/o any
>>>>>>>> interruptions, as
>>>>>>>> data is readily available in the FIFO.
>>>>>>>>
>>>>>>>> With testing done using the mass storage function driver, the
>>>>>>>> results
>>>>>>>> show
>>>>>>>> that with a larger TXFIFO depth, the bandwidth increased
>>>>>>>> significantly.
>>>>>>>>
>>>>>>>> Test Parameters:
>>>>>>>>     - Platform: Qualcomm SM8150
>>>>>>>>     - bMaxBurst = 6
>>>>>>>>     - USB req size = 256kB
>>>>>>>>     - Num of USB reqs = 16
>>>>>>>>     - USB Speed = Super-Speed
>>>>>>>>     - Function Driver: Mass Storage (w/ ramdisk)
>>>>>>>>     - Test Application: CrystalDiskMark
>>>>>>>>
>>>>>>>> Results:
>>>>>>>>
>>>>>>>> TXFIFO Depth = 3 max packets
>>>>>>>>
>>>>>>>> Test Case | Data Size | AVG tput (in MB/s)
>>>>>>>> -------------------------------------------
>>>>>>>> Sequential|1 GB x     |
>>>>>>>> Read      |9 loops    | 193.60
>>>>>>>>         |           | 195.86
>>>>>>>>              |           | 184.77
>>>>>>>>              |           | 193.60
>>>>>>>> -------------------------------------------
>>>>>>>>
>>>>>>>> TXFIFO Depth = 6 max packets
>>>>>>>>
>>>>>>>> Test Case | Data Size | AVG tput (in MB/s)
>>>>>>>> -------------------------------------------
>>>>>>>> Sequential|1 GB x     |
>>>>>>>> Read      |9 loops    | 287.35
>>>>>>>>         |           | 304.94
>>>>>>>>              |           | 289.64
>>>>>>>>              |           | 293.61
>>>>>>>> -------------------------------------------
>>>>>>>>
>>>>>>>> Wesley Cheng (6):
>>>>>>>>      usb: gadget: udc: core: Introduce check_config to verify USB
>>>>>>>>        configuration
>>>>>>>>      usb: gadget: configfs: Check USB configuration before adding
>>>>>>>>      usb: dwc3: Resize TX FIFOs to meet EP bursting requirements
>>>>>>>>      of: Add stub for of_add_property()
>>>>>>>>      usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by
>>>>>>>> default
>>>>>>>>      dt-bindings: usb: dwc3: Update dwc3 TX fifo properties
>>>>>>>>
>>>>>>>>     .../devicetree/bindings/usb/snps,dwc3.yaml         |  15 +-
>>>>>>>>     drivers/usb/dwc3/core.c                            |   9 +
>>>>>>>>     drivers/usb/dwc3/core.h                            |  15 ++
>>>>>>>>     drivers/usb/dwc3/dwc3-qcom.c                       |   9 +
>>>>>>>>     drivers/usb/dwc3/ep0.c                             |   2 +
>>>>>>>>     drivers/usb/dwc3/gadget.c                          | 212
>>>>>>>> +++++++++++++++++++++
>>>>>>>>     drivers/usb/gadget/configfs.c                      |  22 +++
>>>>>>>>     drivers/usb/gadget/udc/core.c                      |  25 +++
>>>>>>>>     include/linux/of.h                                 |   5 +
>>>>>>>>     include/linux/usb/gadget.h                         |   5 +
>>>>>>>>     10 files changed, 317 insertions(+), 2 deletions(-)
>>>>>>>>

-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-06-22 22:00               ` Wesley Cheng
@ 2021-06-22 22:12                 ` Ferry Toth
  2021-06-23  7:50                   ` Wesley Cheng
  0 siblings, 1 reply; 73+ messages in thread
From: Ferry Toth @ 2021-06-22 22:12 UTC (permalink / raw)
  To: Wesley Cheng, balbi, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list
  Cc: linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	heikki.krogerus, andy.shevchenko

Hi

Op 23-06-2021 om 00:00 schreef Wesley Cheng:
>
> On 6/22/2021 1:09 PM, Ferry Toth wrote:
>> Hi
>>
>> Op 22-06-2021 om 20:38 schreef Wesley Cheng:
>>> On 6/19/2021 5:40 AM, Ferry Toth wrote:
>>>> Hi
>>>>
>>>> Op 18-06-2021 om 00:25 schreef Wesley Cheng:
>>>>> Hi,
>>>>>
>>>>> On 6/17/2021 2:55 PM, Ferry Toth wrote:
>>>>>> Hi
>>>>>>
>>>>>> Op 17-06-2021 om 23:48 schreef Wesley Cheng:
>>>>>>> Hi,
>>>>>>>
>>>>>>> On 6/17/2021 2:01 PM, Ferry Toth wrote:
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> Op 17-06-2021 om 11:58 schreef Wesley Cheng:
>>>>>>>>> Changes in V10:
>>>>>>>>>      - Fixed compilation errors in config where OF is not used
>>>>>>>>> (error due to
>>>>>>>>>        unknown symbol for of_add_property()).  Add of_add_property()
>>>>>>>>> stub.
>>>>>>>>>      - Fixed compilation warning for incorrect argument being
>>>>>>>>> passed to
>>>>>>>>> dwc3_mdwidth
>>>>>>>> This fixes the OOPS I had in V9. I do not see any change in
>>>>>>>> performance
>>>>>>>> on Merrifield though.
>>>>>>> I see...thanks Ferry! With your testing, are you writing to the
>>>>>>> device's
>>>>>>> internal storage (ie UFS, eMMC, etc...), or did you use a ramdisk as
>>>>>>> well?
>>>>>> In this case I just tested the EEM path using iperf3.
>>>>>>
>>>>> Got it.  I don't believe f_eem will use a high enough (if at all)
>>>>> bMaxBurst value to change the TXFIFO size.
>>>>>
>>>>>>> If not with a ramdisk, we might want to give that a try to avoid the
>>>>>>> storage path being the bottleneck.  You can use "dd" to create an
>>>>>>> empty
>>>>>>> file, and then just use that as the LUN's backing file.
>>>>>>>
>>>>>>> echo ramdisk.img >
>>>>>>> /sys/kernel/config/usb_gadget/g1/functions/mass_storage.0/lun.0/file
>>>>>> Ah, why didn't I think of that. I have currently mass storage setup
>>>>>> with
>>>>>> eMMC but it seems that is indeed the bottleneck.
>>>>>>
>>>> I created a 64MB disk following the instructions here
>>>> http://www.linux-usb.org/gadget/file_storage.html (that seems a little
>>>> outdated, at least I can not start the first partition at sector 8, but
>>>> minimum 2048), and added a test file on it.
>>>>
>>>> I then copy the file to /dev/shm prior to setting configfs (composite
>>>> device gser/eem/mass_storage/uac2).
>>>>
>>>> journal shows:
>>>>
>>>> kernel: Mass Storage Function, version: 2009/09/11
>>>> kernel: LUN: removable file: (no medium)
>>>>
>>>> I don't know what that means, because I see the test file on the
>>>> ramdisk.
>>>>
>>>> Then I again used gnome disks to benchmark (read/write 10MB):
>>>>
>>>> With V10 on top v5.13.0-rc5:
>>>>
>>>> R/W speed = 35.6/35.8MB/s, access time 0.35ms
>>>>
>>>> With no patches on top v5.12.0:
>>>>
>>>> R/W speed = 35.7/36.1MB/s, access time 0.35ms
>>> Hi Ferry,
>>>
>>>> I see no speed difference (and it's about the same as with the eMMC
>>>> backed disk). But the patches are causing a new call trace
>>>>
>>> Would you happen to know what DWC3 controller revision the device is
>>> using?  The callstack print occurs, because it looks like it ran out of
>>> internal memory, although there should be logic present for making sure
>>> that at least there is enough room for 1 FIFO per endpoint.  (possibly
>>> the logic/math depends on the controller revision)
> Hi Ferry,
>
>> Do you know where I could find that in a file on the device?
>>
> Maybe you can just dump the DWC3 registers?
> cat /sys/kernel/debug/usb/<controller name>/regdump

This isin host mode:

~# cat /sys/kernel/debug/usb/dwc3.0.auto/regdump
GSBUSCFG0 = 0x00000006
GSBUSCFG1 = 0x00000f00
GTXTHRCFG = 0x230a0000
GRXTHRCFG = 0x22800000
GCTL = 0x45801002
GEVTEN = 0x00000000
GSTS = 0x3e800001
GUCTL1 = 0x00000000
GSNPSID = 0x5533250a
GGPIO = 0x00000000
GUID = 0x00050d00
GUCTL = 0x0200ce00
GBUSERRADDR0 = 0x00000000
GBUSERRADDR1 = 0x00000000
GPRTBIMAP0 = 0x00000000
GPRTBIMAP1 = 0x00000000
GHWPARAMS0 = 0x2020400a
GHWPARAMS1 = 0x0260c93b
GHWPARAMS2 = 0x008086a0
GHWPARAMS3 = 0x10420089
GHWPARAMS4 = 0x47a22004
GHWPARAMS5 = 0x04202088
GHWPARAMS6 = 0x0c00ac20
GHWPARAMS7 = 0x038807e6
GDBGFIFOSPACE = 0x00820000
GDBGLTSSM = 0x09550442
GDBGBMU = 0x20300000
GPRTBIMAP_HS0 = 0x00000000
GPRTBIMAP_HS1 = 0x00000000
GPRTBIMAP_FS0 = 0x00000000
GPRTBIMAP_FS1 = 0x00000000
GUSB2PHYCFG(0) = 0x0000a710
GUSB2PHYCFG(1) = 0x00000000
GUSB2PHYCFG(2) = 0x00000000
GUSB2PHYCFG(3) = 0x00000000
GUSB2PHYCFG(4) = 0x00000000
GUSB2PHYCFG(5) = 0x00000000
GUSB2PHYCFG(6) = 0x00000000
GUSB2PHYCFG(7) = 0x00000000
GUSB2PHYCFG(8) = 0x00000000
GUSB2PHYCFG(9) = 0x00000000
GUSB2PHYCFG(10) = 0x00000000
GUSB2PHYCFG(11) = 0x00000000
GUSB2PHYCFG(12) = 0x00000000
GUSB2PHYCFG(13) = 0x00000000
GUSB2PHYCFG(14) = 0x00000000
GUSB2PHYCFG(15) = 0x00000000
GUSB2I2CCTL(0) = 0x00000000
GUSB2I2CCTL(1) = 0x00000000
GUSB2I2CCTL(2) = 0x00000000
GUSB2I2CCTL(3) = 0x00000000
GUSB2I2CCTL(4) = 0x00000000
GUSB2I2CCTL(5) = 0x00000000
GUSB2I2CCTL(6) = 0x00000000
GUSB2I2CCTL(7) = 0x00000000
GUSB2I2CCTL(8) = 0x00000000
GUSB2I2CCTL(9) = 0x00000000
GUSB2I2CCTL(10) = 0x00000000
GUSB2I2CCTL(11) = 0x00000000
GUSB2I2CCTL(12) = 0x00000000
GUSB2I2CCTL(13) = 0x00000000
GUSB2I2CCTL(14) = 0x00000000
GUSB2I2CCTL(15) = 0x00000000
GUSB2PHYACC(0) = 0x014a0027
GUSB2PHYACC(1) = 0x00000000
GUSB2PHYACC(2) = 0x00000000
GUSB2PHYACC(3) = 0x00000000
GUSB2PHYACC(4) = 0x00000000
GUSB2PHYACC(5) = 0x00000000
GUSB2PHYACC(6) = 0x00000000
GUSB2PHYACC(7) = 0x00000000
GUSB2PHYACC(8) = 0x00000000
GUSB2PHYACC(9) = 0x00000000
GUSB2PHYACC(10) = 0x00000000
GUSB2PHYACC(11) = 0x00000000
GUSB2PHYACC(12) = 0x00000000
GUSB2PHYACC(13) = 0x00000000
GUSB2PHYACC(14) = 0x00000000
GUSB2PHYACC(15) = 0x00000000
GUSB3PIPECTL(0) = 0x02000002
GUSB3PIPECTL(1) = 0x00000000
GUSB3PIPECTL(2) = 0x00000000
GUSB3PIPECTL(3) = 0x00000000
GUSB3PIPECTL(4) = 0x00000000
GUSB3PIPECTL(5) = 0x00000000
GUSB3PIPECTL(6) = 0x00000000
GUSB3PIPECTL(7) = 0x00000000
GUSB3PIPECTL(8) = 0x00000000
GUSB3PIPECTL(9) = 0x00000000
GUSB3PIPECTL(10) = 0x00000000
GUSB3PIPECTL(11) = 0x00000000
GUSB3PIPECTL(12) = 0x00000000
GUSB3PIPECTL(13) = 0x00000000
GUSB3PIPECTL(14) = 0x00000000
GUSB3PIPECTL(15) = 0x00000000
GTXFIFOSIZ(0) = 0x00000082
GTXFIFOSIZ(1) = 0x00820184
GTXFIFOSIZ(2) = 0x02060286
GTXFIFOSIZ(3) = 0x048c0000
GTXFIFOSIZ(4) = 0x048c0000
GTXFIFOSIZ(5) = 0x048c0000
GTXFIFOSIZ(6) = 0x048c0000
GTXFIFOSIZ(7) = 0x048c0000
GTXFIFOSIZ(8) = 0x048c0000
GTXFIFOSIZ(9) = 0x048c0000
GTXFIFOSIZ(10) = 0x048c0000
GTXFIFOSIZ(11) = 0x048c0000
GTXFIFOSIZ(12) = 0x048c0000
GTXFIFOSIZ(13) = 0x048c0000
GTXFIFOSIZ(14) = 0x048c0000
GTXFIFOSIZ(15) = 0x048c0000
GTXFIFOSIZ(16) = 0x00000000
GTXFIFOSIZ(17) = 0x00000000
GTXFIFOSIZ(18) = 0x00000000
GTXFIFOSIZ(19) = 0x00000000
GTXFIFOSIZ(20) = 0x00000000
GTXFIFOSIZ(21) = 0x00000000
GTXFIFOSIZ(22) = 0x00000000
GTXFIFOSIZ(23) = 0x00000000
GTXFIFOSIZ(24) = 0x00000000
GTXFIFOSIZ(25) = 0x00000000
GTXFIFOSIZ(26) = 0x00000000
GTXFIFOSIZ(27) = 0x00000000
GTXFIFOSIZ(28) = 0x00000000
GTXFIFOSIZ(29) = 0x00000000
GTXFIFOSIZ(30) = 0x00000000
GTXFIFOSIZ(31) = 0x00000000
GRXFIFOSIZ(0) = 0x00000084
GRXFIFOSIZ(1) = 0x00840104
GRXFIFOSIZ(2) = 0x01880200
GRXFIFOSIZ(3) = 0x00000000
GRXFIFOSIZ(4) = 0x00000000
GRXFIFOSIZ(5) = 0x00000000
GRXFIFOSIZ(6) = 0x00000000
GRXFIFOSIZ(7) = 0x00000000
GRXFIFOSIZ(8) = 0x00000000
GRXFIFOSIZ(9) = 0x00000000
GRXFIFOSIZ(10) = 0x00000000
GRXFIFOSIZ(11) = 0x00000000
GRXFIFOSIZ(12) = 0x00000000
GRXFIFOSIZ(13) = 0x00000000
GRXFIFOSIZ(14) = 0x00000000
GRXFIFOSIZ(15) = 0x00000000
GRXFIFOSIZ(16) = 0x00000000
GRXFIFOSIZ(17) = 0x00000000
GRXFIFOSIZ(18) = 0x00000000
GRXFIFOSIZ(19) = 0x00000000
GRXFIFOSIZ(20) = 0x00000000
GRXFIFOSIZ(21) = 0x00000000
GRXFIFOSIZ(22) = 0x00000000
GRXFIFOSIZ(23) = 0x00000000
GRXFIFOSIZ(24) = 0x00000000
GRXFIFOSIZ(25) = 0x00000000
GRXFIFOSIZ(26) = 0x00000000
GRXFIFOSIZ(27) = 0x00000000
GRXFIFOSIZ(28) = 0x00000000
GRXFIFOSIZ(29) = 0x00000000
GRXFIFOSIZ(30) = 0x00000000
GRXFIFOSIZ(31) = 0x00000000
GEVNTADRLO(0) = 0x00000000
GEVNTADRHI(0) = 0x00000000
GEVNTSIZ(0) = 0x00001000
GEVNTCOUNT(0) = 0x00000000
GHWPARAMS8 = 0x00000c00
DCFG = 0x00080800
DCTL = 0x00f00000
DEVTEN = 0x00000000
DSTS = 0x00539e19
DGCMDPAR = 0x00000000
DGCMD = 0x00000000
DALEPENA = 0x00000000
DEPCMDPAR2(0) = 0x00000000
DEPCMDPAR1(0) = 0x00000002
DEPCMDPAR0(0) = 0x06a39001
DEPCMD(0) = 0x00000000
DEPCMDPAR2(1) = 0x00000000
DEPCMDPAR1(1) = 0x00000000
DEPCMDPAR0(1) = 0x00000000
DEPCMD(1) = 0x00000000
DEPCMDPAR2(2) = 0x07359000
DEPCMDPAR1(2) = 0x00000000
DEPCMDPAR0(2) = 0x0000007f
DEPCMD(2) = 0x00000000
DEPCMDPAR2(3) = 0x00000000
DEPCMDPAR1(3) = 0x00000000
DEPCMDPAR0(3) = 0x00000000
DEPCMD(3) = 0x00000000
DEPCMDPAR2(4) = 0x06a40000
DEPCMDPAR1(4) = 0x00000000
DEPCMDPAR0(4) = 0x06a3feb8
DEPCMD(4) = 0x00000000
DEPCMDPAR2(5) = 0x00000000
DEPCMDPAR1(5) = 0x00000000
DEPCMDPAR0(5) = 0x00000008
DEPCMD(5) = 0x00000000
DEPCMDPAR2(6) = 0x00000000
DEPCMDPAR1(6) = 0x00000000
DEPCMDPAR0(6) = 0x00000000
DEPCMD(6) = 0x00000000
DEPCMDPAR2(7) = 0x00000000
DEPCMDPAR1(7) = 0x00000000
DEPCMDPAR0(7) = 0x00000000
DEPCMD(7) = 0x00000000
DEPCMDPAR2(8) = 0x00000000
DEPCMDPAR1(8) = 0x00000000
DEPCMDPAR0(8) = 0x00000000
DEPCMD(8) = 0x00000000
DEPCMDPAR2(9) = 0x00000000
DEPCMDPAR1(9) = 0x00000000
DEPCMDPAR0(9) = 0x00000000
DEPCMD(9) = 0x00000000
DEPCMDPAR2(10) = 0x00000000
DEPCMDPAR1(10) = 0x00000000
DEPCMDPAR0(10) = 0x00000000
DEPCMD(10) = 0x00000000
DEPCMDPAR2(11) = 0x00000000
DEPCMDPAR1(11) = 0x00000000
DEPCMDPAR0(11) = 0x00000000
DEPCMD(11) = 0x00000000
DEPCMDPAR2(12) = 0x00000000
DEPCMDPAR1(12) = 0x00000000
DEPCMDPAR0(12) = 0x00000000
DEPCMD(12) = 0x00000000
DEPCMDPAR2(13) = 0x00000000
DEPCMDPAR1(13) = 0x00000000
DEPCMDPAR0(13) = 0x00000000
DEPCMD(13) = 0x00000000
DEPCMDPAR2(14) = 0x00000000
DEPCMDPAR1(14) = 0x00000000
DEPCMDPAR0(14) = 0x00000000
DEPCMD(14) = 0x00000000
DEPCMDPAR2(15) = 0x00000000
DEPCMDPAR1(15) = 0x00000000
DEPCMDPAR0(15) = 0x00000000
DEPCMD(15) = 0x00000000
DEPCMDPAR2(16) = 0x00000000
DEPCMDPAR1(16) = 0x00000000
DEPCMDPAR0(16) = 0x00000000
DEPCMD(16) = 0x00000000
DEPCMDPAR2(17) = 0x00000000
DEPCMDPAR1(17) = 0x00000000
DEPCMDPAR0(17) = 0x00000000
DEPCMD(17) = 0x00000000
DEPCMDPAR2(18) = 0x00000000
DEPCMDPAR1(18) = 0x00000000
DEPCMDPAR0(18) = 0x00000000
DEPCMD(18) = 0x00000000
DEPCMDPAR2(19) = 0x00000000
DEPCMDPAR1(19) = 0x00000000
DEPCMDPAR0(19) = 0x00000000
DEPCMD(19) = 0x00000000
DEPCMDPAR2(20) = 0x00000000
DEPCMDPAR1(20) = 0x00000000
DEPCMDPAR0(20) = 0x00000000
DEPCMD(20) = 0x00000000
DEPCMDPAR2(21) = 0x00000000
DEPCMDPAR1(21) = 0x00000000
DEPCMDPAR0(21) = 0x00000000
DEPCMD(21) = 0x00000000
DEPCMDPAR2(22) = 0x00000000
DEPCMDPAR1(22) = 0x00000000
DEPCMDPAR0(22) = 0x00000000
DEPCMD(22) = 0x00000000
DEPCMDPAR2(23) = 0x00000000
DEPCMDPAR1(23) = 0x00000000
DEPCMDPAR0(23) = 0x00000000
DEPCMD(23) = 0x00000000
DEPCMDPAR2(24) = 0x00000000
DEPCMDPAR1(24) = 0x00000000
DEPCMDPAR0(24) = 0x00000000
DEPCMD(24) = 0x00000000
DEPCMDPAR2(25) = 0x00000000
DEPCMDPAR1(25) = 0x00000000
DEPCMDPAR0(25) = 0x00000000
DEPCMD(25) = 0x00000000
DEPCMDPAR2(26) = 0x00000000
DEPCMDPAR1(26) = 0x00000000
DEPCMDPAR0(26) = 0x00000000
DEPCMD(26) = 0x00000000
DEPCMDPAR2(27) = 0x00000000
DEPCMDPAR1(27) = 0x00000000
DEPCMDPAR0(27) = 0x00000000
DEPCMD(27) = 0x00000000
DEPCMDPAR2(28) = 0x00000000
DEPCMDPAR1(28) = 0x00000000
DEPCMDPAR0(28) = 0x00000000
DEPCMD(28) = 0x00000000
DEPCMDPAR2(29) = 0x00000000
DEPCMDPAR1(29) = 0x00000000
DEPCMDPAR0(29) = 0x00000000
DEPCMD(29) = 0x00000000
DEPCMDPAR2(30) = 0x00000000
DEPCMDPAR1(30) = 0x00000000
DEPCMDPAR0(30) = 0x00000000
DEPCMD(30) = 0x00000000
DEPCMDPAR2(31) = 0x00000000
DEPCMDPAR1(31) = 0x00000000
DEPCMDPAR0(31) = 0x00000000
DEPCMD(31) = 0x00000000
OCFG = 0x00000000
OCTL = 0x00000040
OEVT = 0x00000000
OEVTEN = 0x00000000
OSTS = 0x00000008

> Was going to ask for the same to confirm the TXFIFO sizes from your
> results below :).
>
>> Otherwise, I'm hoping Andy will know?
>>
>>> Also, is there a way to use just a mass storage only composition?  Based
>>> on the above observation, that probably means that the mass storage
>>> interface wasn't resized at all, because the configuration took up a lot
>>> of the internal FIFO space.
>> Sure, it's configured through configfs. With only mass_storage I have:
>>
>> With V10 on top v5.13.0-rc5:
>>
>> R/W speed = 41,6/39,3MB/s, access time 0.33ms
>>
>> With no patches on top v5.12.0:
>>
>> R/W speed = 41,1/38,7MB/s, access time 0.38ms
>>
> Thanks Ferry!  Could you collect the regdump, so I can confirm the two
> things mentioned?
>
> Thanks
> Wesley Cheng
>
>>> Thanks
>>> Wesley Cheng
>>>
>>>> kernel: using random self ethernet address
>>>> kernel: using random host ethernet address
>>>> kernel: Mass Storage Function, version: 2009/09/11
>>>> kernel: LUN: removable file: (no medium)
>>>> kernel: usb0: HOST MAC aa:bb:cc:dd:ee:f2
>>>> kernel: usb0: MAC aa:bb:cc:dd:ee:f1
>>>> kernel: IPv6: ADDRCONF(NETDEV_CHANGE): usb0: link becomes ready
>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>> depth:115540359
>>>> kernel: ------------[ cut here ]------------
>>>> kernel: WARNING: CPU: 0 PID: 594 at drivers/usb/gadget/udc/core.c:278
>>>> usb_ep_queue+0x75/0x80
>>>> kernel: Modules linked in: usb_f_uac2 u_audio usb_f_mass_storage
>>>> usb_f_eem u_ether usb_f_serial u_serial libcomposite rfcomm iptable_nat
>>>> bnep snd_sof_nocodec spi_pxa2>
>>>> kernel: CPU: 0 PID: 594 Comm: irq/14-dwc3 Not tainted
>>>> 5.13.0-rc5-edison-acpi-standard #1
>>>> kernel: Hardware name: Intel Corporation Merrifield/BODEGA BAY, BIOS 542
>>>> 2015.01.21:18.19.48
>>>> kernel: RIP: 0010:usb_ep_queue+0x75/0x80
>>>> kernel: Code: 01 73 e4 48 8b 05 fb 63 06 01 48 85 c0 74 12 48 8b 78 08
>>>> 44 89 e9 4c 89 e2 48 89 ee e8 74 05 00 00 44 89 e8 5d 41 5c 41 5d c3
>>>> <0f> 0b 41 bd 94 ff ff ff >
>>>> kernel: RSP: 0000:ffff91eec083fc98 EFLAGS: 00010082
>>>> kernel: RAX: ffff8af20357d960 RBX: 0000000000000000 RCX:
>>>> ffff8af202f06400
>>>> kernel: RDX: 0000000000000a20 RSI: ffff8af208785780 RDI:
>>>> ffff8af202e9ae00
>>>> kernel: RBP: ffff8af202e9ae00 R08: 00000000000000c0 R09:
>>>> ffff8af208785780
>>>> kernel: R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
>>>> ffff8af208785780
>>>> kernel: R13: 0000000000000000 R14: ffff8af202e9ae00 R15:
>>>> ffff8af203e26cc0
>>>> kernel: FS:  0000000000000000(0000) GS:ffff8af23e200000(0000)
>>>> knlGS:0000000000000000
>>>> kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>> kernel: CR2: 000055e2c21f2100 CR3: 0000000003b38000 CR4:
>>>> 00000000001006f0
>>>> kernel: Call Trace:
>>>> kernel:  u_audio_start_playback+0x107/0x1a0 [u_audio]
>>>> kernel:  composite_setup+0x224/0x1ba0 [libcomposite]
>>>> kernel:  ? dwc3_gadget_ep_queue+0xf6/0x1a0
>>>> kernel:  ? usb_ep_queue+0x2a/0x80
>>>> kernel:  ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>> kernel:  configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>> kernel:  dwc3_ep0_interrupt+0x469/0xa80
>>>> kernel:  dwc3_thread_interrupt+0x8ee/0xf40
>>>> kernel:  ? __wake_up_common_lock+0x85/0xb0
>>>> kernel:  ? disable_irq_nosync+0x10/0x10
>>>> kernel:  irq_thread_fn+0x1b/0x60
>>>> kernel:  irq_thread+0xd6/0x170
>>>> kernel:  ? irq_thread_check_affinity+0x70/0x70
>>>> kernel:  ? irq_forced_thread_fn+0x70/0x70
>>>> kernel:  kthread+0x116/0x130
>>>> kernel:  ? kthread_create_worker_on_cpu+0x60/0x60
>>>> kernel:  ret_from_fork+0x22/0x30
>>>> kernel: ---[ end trace e5b9e28058c53584 ]---
>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>> kernel: dwc3 dwc3.0.auto: request 000000003c32dcc5 was not queued to
>>>> ep5in
>>>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to
>>>> ep5in
>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>> depth:115540359
>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to
>>>> ep5in
>>>> kernel: dwc3 dwc3.0.auto: request 00000000036ac129 was not queued to
>>>> ep5in
>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>> depth:115540359
>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>> kernel: dwc3 dwc3.0.auto: request 00000000ad1b8c18 was not queued to
>>>> ep5in
>>>> kernel: dwc3 dwc3.0.auto: request 00000000fbc71244 was not queued to
>>>> ep5in
>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>> depth:115540359
>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>> kernel: dwc3 dwc3.0.auto: request 00000000fbc71244 was not queued to
>>>> ep5in
>>>> kernel: dwc3 dwc3.0.auto: request 00000000ad1b8c18 was not queued to
>>>> ep5in
>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>> depth:115540359
>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>> kernel: dwc3 dwc3.0.auto: request 000000003c32dcc5 was not queued to
>>>> ep5in
>>>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to
>>>> ep5in
>>>>
>>>> Removing uac2 from the config makes the call trace go away, but the R/W
>>>> speed does not change.
>>>>
>>>>> :), not a problem...I've been working on getting the ideal set up for
>>>>> the performance profiling for awhile, so anything I can do to make sure
>>>>> we get some good results.
>>>>>
>>>>>> I'll try with a ramdisk and let you know.
>>>>>>
>>>>> Thanks again for the testing, Ferry.
>>>>>
>>>>> Thanks
>>>>> Wesley Cheng
>>>>>
>>>>>>> Thanks
>>>>>>> Wesley Cheng
>>>>>>>
>>>>>>>>> Changes in V9:
>>>>>>>>>      - Fixed incorrect patch in series.  Removed changes in DTSI, as
>>>>>>>>> dwc3-qcom will
>>>>>>>>>        add the property by default from the kernel.
>>>>>>>>>
>>>>>>>>> Changes in V8:
>>>>>>>>>      - Rebased to usb-testing
>>>>>>>>>      - Using devm_kzalloc for adding txfifo property in dwc3-qcom
>>>>>>>>>      - Removed DWC3 QCOM ACPI property for enabling the txfifo
>>>>>>>>> resize
>>>>>>>>>
>>>>>>>>> Changes in V7:
>>>>>>>>>      - Added a new property tx-fifo-max-num for limiting how much
>>>>>>>>> fifo
>>>>>>>>> space the
>>>>>>>>>        resizing logic can allocate for endpoints with large burst
>>>>>>>>> values.  This
>>>>>>>>>        can differ across platforms, and tie in closely with overall
>>>>>>>>> system latency.
>>>>>>>>>      - Added recommended checks for DWC32.
>>>>>>>>>      - Added changes to set the tx-fifo-resize property from
>>>>>>>>> dwc3-qcom by
>>>>>>>>> default
>>>>>>>>>        instead of modifying the current DTSI files.
>>>>>>>>>      - Added comments on all APIs/variables introduced.
>>>>>>>>>      - Updated the DWC3 YAML to include a better description of the
>>>>>>>>> tx-fifo-resize
>>>>>>>>>        property and added an entry for tx-fifo-max-num.
>>>>>>>>>
>>>>>>>>> Changes in V6:
>>>>>>>>>      - Rebased patches to usb-testing.
>>>>>>>>>      - Renamed to PATCH series instead of RFC.
>>>>>>>>>      - Checking for fs_descriptors instead of ss_descriptors for
>>>>>>>>> determining the
>>>>>>>>>        endpoint count for a particular configuration.
>>>>>>>>>      - Re-ordered patch series to fix patch dependencies.
>>>>>>>>>
>>>>>>>>> Changes in V5:
>>>>>>>>>      - Added check_config() logic, which is used to communicate the
>>>>>>>>> number of EPs
>>>>>>>>>        used in a particular configuration.  Based on this, the DWC3
>>>>>>>>> gadget driver
>>>>>>>>>        has the ability to know the maximum number of eps utilized in
>>>>>>>>> all
>>>>>>>>> configs.
>>>>>>>>>        This helps reduce unnecessary allocation to unused eps,
>>>>>>>>> and will
>>>>>>>>> catch fifo
>>>>>>>>>        allocation issues at bind() time.
>>>>>>>>>      - Fixed variable declaration to single line per variable, and
>>>>>>>>> reverse xmas.
>>>>>>>>>      - Created a helper for fifo clearing, which is used by ep0.c
>>>>>>>>>
>>>>>>>>> Changes in V4:
>>>>>>>>>      - Removed struct dwc3* as an argument for
>>>>>>>>> dwc3_gadget_resize_tx_fifos()
>>>>>>>>>      - Removed WARN_ON(1) in case we run out of fifo space
>>>>>>>>>      Changes in V3:
>>>>>>>>>      - Removed "Reviewed-by" tags
>>>>>>>>>      - Renamed series back to RFC
>>>>>>>>>      - Modified logic to ensure that fifo_size is reset if we
>>>>>>>>> pass the
>>>>>>>>> minimum
>>>>>>>>>        threshold.  Tested with binding multiple FDs requesting 6
>>>>>>>>> FIFOs.
>>>>>>>>>
>>>>>>>>> Changes in V2:
>>>>>>>>>      - Modified TXFIFO resizing logic to ensure that each EP is
>>>>>>>>> reserved a
>>>>>>>>>        FIFO.
>>>>>>>>>      - Removed dev_dbg() prints and fixed typos from patches
>>>>>>>>>      - Added some more description on the dt-bindings commit message
>>>>>>>>>
>>>>>>>>> Currently, there is no functionality to allow for resizing the
>>>>>>>>> TXFIFOs, and
>>>>>>>>> relying on the HW default setting for the TXFIFO depth.  In most
>>>>>>>>> cases, the
>>>>>>>>> HW default is probably sufficient, but for USB compositions that
>>>>>>>>> contain
>>>>>>>>> multiple functions that require EP bursting, the default settings
>>>>>>>>> might not be enough.  Also to note, the current SW will assign an
>>>>>>>>> EP to a
>>>>>>>>> function driver w/o checking to see if the TXFIFO size for that
>>>>>>>>> particular
>>>>>>>>> EP is large enough. (this is a problem if there are multiple HW
>>>>>>>>> defined
>>>>>>>>> values for the TXFIFO size)
>>>>>>>>>
>>>>>>>>> It is mentioned in the SNPS databook that a minimum of TX FIFO
>>>>>>>>> depth = 3
>>>>>>>>> is required for an EP that supports bursting.  Otherwise, there
>>>>>>>>> may be
>>>>>>>>> frequent occurences of bursts ending.  For high bandwidth
>>>>>>>>> functions,
>>>>>>>>> such as data tethering (protocols that support data aggregation),
>>>>>>>>> mass
>>>>>>>>> storage, and media transfer protocol (over FFS), the bMaxBurst
>>>>>>>>> value
>>>>>>>>> can be
>>>>>>>>> large, and a bigger TXFIFO depth may prove to be beneficial in
>>>>>>>>> terms
>>>>>>>>> of USB
>>>>>>>>> throughput. (which can be associated to system access latency,
>>>>>>>>> etc...)  It
>>>>>>>>> allows for a more consistent burst of traffic, w/o any
>>>>>>>>> interruptions, as
>>>>>>>>> data is readily available in the FIFO.
>>>>>>>>>
>>>>>>>>> With testing done using the mass storage function driver, the
>>>>>>>>> results
>>>>>>>>> show
>>>>>>>>> that with a larger TXFIFO depth, the bandwidth increased
>>>>>>>>> significantly.
>>>>>>>>>
>>>>>>>>> Test Parameters:
>>>>>>>>>      - Platform: Qualcomm SM8150
>>>>>>>>>      - bMaxBurst = 6
>>>>>>>>>      - USB req size = 256kB
>>>>>>>>>      - Num of USB reqs = 16
>>>>>>>>>      - USB Speed = Super-Speed
>>>>>>>>>      - Function Driver: Mass Storage (w/ ramdisk)
>>>>>>>>>      - Test Application: CrystalDiskMark
>>>>>>>>>
>>>>>>>>> Results:
>>>>>>>>>
>>>>>>>>> TXFIFO Depth = 3 max packets
>>>>>>>>>
>>>>>>>>> Test Case | Data Size | AVG tput (in MB/s)
>>>>>>>>> -------------------------------------------
>>>>>>>>> Sequential|1 GB x     |
>>>>>>>>> Read      |9 loops    | 193.60
>>>>>>>>>          |           | 195.86
>>>>>>>>>               |           | 184.77
>>>>>>>>>               |           | 193.60
>>>>>>>>> -------------------------------------------
>>>>>>>>>
>>>>>>>>> TXFIFO Depth = 6 max packets
>>>>>>>>>
>>>>>>>>> Test Case | Data Size | AVG tput (in MB/s)
>>>>>>>>> -------------------------------------------
>>>>>>>>> Sequential|1 GB x     |
>>>>>>>>> Read      |9 loops    | 287.35
>>>>>>>>>          |           | 304.94
>>>>>>>>>               |           | 289.64
>>>>>>>>>               |           | 293.61
>>>>>>>>> -------------------------------------------
>>>>>>>>>
>>>>>>>>> Wesley Cheng (6):
>>>>>>>>>       usb: gadget: udc: core: Introduce check_config to verify USB
>>>>>>>>>         configuration
>>>>>>>>>       usb: gadget: configfs: Check USB configuration before adding
>>>>>>>>>       usb: dwc3: Resize TX FIFOs to meet EP bursting requirements
>>>>>>>>>       of: Add stub for of_add_property()
>>>>>>>>>       usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by
>>>>>>>>> default
>>>>>>>>>       dt-bindings: usb: dwc3: Update dwc3 TX fifo properties
>>>>>>>>>
>>>>>>>>>      .../devicetree/bindings/usb/snps,dwc3.yaml         |  15 +-
>>>>>>>>>      drivers/usb/dwc3/core.c                            |   9 +
>>>>>>>>>      drivers/usb/dwc3/core.h                            |  15 ++
>>>>>>>>>      drivers/usb/dwc3/dwc3-qcom.c                       |   9 +
>>>>>>>>>      drivers/usb/dwc3/ep0.c                             |   2 +
>>>>>>>>>      drivers/usb/dwc3/gadget.c                          | 212
>>>>>>>>> +++++++++++++++++++++
>>>>>>>>>      drivers/usb/gadget/configfs.c                      |  22 +++
>>>>>>>>>      drivers/usb/gadget/udc/core.c                      |  25 +++
>>>>>>>>>      include/linux/of.h                                 |   5 +
>>>>>>>>>      include/linux/usb/gadget.h                         |   5 +
>>>>>>>>>      10 files changed, 317 insertions(+), 2 deletions(-)
>>>>>>>>>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-06-22 22:12                 ` Ferry Toth
@ 2021-06-23  7:50                   ` Wesley Cheng
  2021-06-25 21:10                     ` Ferry Toth
  0 siblings, 1 reply; 73+ messages in thread
From: Wesley Cheng @ 2021-06-23  7:50 UTC (permalink / raw)
  To: Ferry Toth, balbi, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list
  Cc: linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	heikki.krogerus, andy.shevchenko



On 6/22/2021 3:12 PM, Ferry Toth wrote:
> Hi
> 
> Op 23-06-2021 om 00:00 schreef Wesley Cheng:
>>
>> On 6/22/2021 1:09 PM, Ferry Toth wrote:
>>> Hi
>>>
>>> Op 22-06-2021 om 20:38 schreef Wesley Cheng:
>>>> On 6/19/2021 5:40 AM, Ferry Toth wrote:
>>>>> Hi
>>>>>
>>>>> Op 18-06-2021 om 00:25 schreef Wesley Cheng:
>>>>>> Hi,
>>>>>>
>>>>>> On 6/17/2021 2:55 PM, Ferry Toth wrote:
>>>>>>> Hi
>>>>>>>
>>>>>>> Op 17-06-2021 om 23:48 schreef Wesley Cheng:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On 6/17/2021 2:01 PM, Ferry Toth wrote:
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> Op 17-06-2021 om 11:58 schreef Wesley Cheng:
>>>>>>>>>> Changes in V10:
>>>>>>>>>>      - Fixed compilation errors in config where OF is not used
>>>>>>>>>> (error due to
>>>>>>>>>>        unknown symbol for of_add_property()).  Add
>>>>>>>>>> of_add_property()
>>>>>>>>>> stub.
>>>>>>>>>>      - Fixed compilation warning for incorrect argument being
>>>>>>>>>> passed to
>>>>>>>>>> dwc3_mdwidth
>>>>>>>>> This fixes the OOPS I had in V9. I do not see any change in
>>>>>>>>> performance
>>>>>>>>> on Merrifield though.
>>>>>>>> I see...thanks Ferry! With your testing, are you writing to the
>>>>>>>> device's
>>>>>>>> internal storage (ie UFS, eMMC, etc...), or did you use a
>>>>>>>> ramdisk as
>>>>>>>> well?
>>>>>>> In this case I just tested the EEM path using iperf3.
>>>>>>>
>>>>>> Got it.  I don't believe f_eem will use a high enough (if at all)
>>>>>> bMaxBurst value to change the TXFIFO size.
>>>>>>
>>>>>>>> If not with a ramdisk, we might want to give that a try to avoid
>>>>>>>> the
>>>>>>>> storage path being the bottleneck.  You can use "dd" to create an
>>>>>>>> empty
>>>>>>>> file, and then just use that as the LUN's backing file.
>>>>>>>>
>>>>>>>> echo ramdisk.img >
>>>>>>>> /sys/kernel/config/usb_gadget/g1/functions/mass_storage.0/lun.0/file
>>>>>>>>
>>>>>>> Ah, why didn't I think of that. I have currently mass storage setup
>>>>>>> with
>>>>>>> eMMC but it seems that is indeed the bottleneck.
>>>>>>>
>>>>> I created a 64MB disk following the instructions here
>>>>> http://www.linux-usb.org/gadget/file_storage.html (that seems a little
>>>>> outdated, at least I can not start the first partition at sector 8,
>>>>> but
>>>>> minimum 2048), and added a test file on it.
>>>>>
>>>>> I then copy the file to /dev/shm prior to setting configfs (composite
>>>>> device gser/eem/mass_storage/uac2).
>>>>>
>>>>> journal shows:
>>>>>
>>>>> kernel: Mass Storage Function, version: 2009/09/11
>>>>> kernel: LUN: removable file: (no medium)
>>>>>
>>>>> I don't know what that means, because I see the test file on the
>>>>> ramdisk.
>>>>>
>>>>> Then I again used gnome disks to benchmark (read/write 10MB):
>>>>>
>>>>> With V10 on top v5.13.0-rc5:
>>>>>
>>>>> R/W speed = 35.6/35.8MB/s, access time 0.35ms
>>>>>
>>>>> With no patches on top v5.12.0:
>>>>>
>>>>> R/W speed = 35.7/36.1MB/s, access time 0.35ms
>>>> Hi Ferry,
>>>>
>>>>> I see no speed difference (and it's about the same as with the eMMC
>>>>> backed disk). But the patches are causing a new call trace
>>>>>
>>>> Would you happen to know what DWC3 controller revision the device is
>>>> using?  The callstack print occurs, because it looks like it ran out of
>>>> internal memory, although there should be logic present for making sure
>>>> that at least there is enough room for 1 FIFO per endpoint.  (possibly
>>>> the logic/math depends on the controller revision)
>> Hi Ferry,
>>
>>> Do you know where I could find that in a file on the device?
>>>
>> Maybe you can just dump the DWC3 registers?
>> cat /sys/kernel/debug/usb/<controller name>/regdump
> 
> This isin host mode:
> 
> ~# cat /sys/kernel/debug/usb/dwc3.0.auto/regdump
> GSBUSCFG0 = 0x00000006
> GSBUSCFG1 = 0x00000f00
> GTXTHRCFG = 0x230a0000
> GRXTHRCFG = 0x22800000
> GCTL = 0x45801002
> GEVTEN = 0x00000000
> GSTS = 0x3e800001
> GUCTL1 = 0x00000000
> GSNPSID = 0x5533250a
> GGPIO = 0x00000000
> GUID = 0x00050d00
> GUCTL = 0x0200ce00
> GBUSERRADDR0 = 0x00000000
> GBUSERRADDR1 = 0x00000000
...
> 

Hi Ferry,

Great!  Thanks, that got me the information regarding the DWC3 revision
you're using. (its version 2.50a)  I checked the DWC3 databook and looks
like certain versions have some different math to calculate the TXFIFO
size.  I have taken this into account and factored that in in the next
revision.

Is there a way you could collect the regdump in device mode w/ mass
storage only composition after enumerating with the PC?  That would help
confirm how the TXFIFO resizing logic is working on your platform.

Thanks
Wesley Cheng

>> Was going to ask for the same to confirm the TXFIFO sizes from your
>> results below :).
>>
>>> Otherwise, I'm hoping Andy will know?
>>>
>>>> Also, is there a way to use just a mass storage only composition? 
>>>> Based
>>>> on the above observation, that probably means that the mass storage
>>>> interface wasn't resized at all, because the configuration took up a
>>>> lot
>>>> of the internal FIFO space.
>>> Sure, it's configured through configfs. With only mass_storage I have:
>>>
>>> With V10 on top v5.13.0-rc5:
>>>
>>> R/W speed = 41,6/39,3MB/s, access time 0.33ms
>>>
>>> With no patches on top v5.12.0:
>>>
>>> R/W speed = 41,1/38,7MB/s, access time 0.38ms
>>>
>> Thanks Ferry!  Could you collect the regdump, so I can confirm the two
>> things mentioned?
>>
>> Thanks
>> Wesley Cheng
>>
>>>> Thanks
>>>> Wesley Cheng
>>>>
>>>>> kernel: using random self ethernet address
>>>>> kernel: using random host ethernet address
>>>>> kernel: Mass Storage Function, version: 2009/09/11
>>>>> kernel: LUN: removable file: (no medium)
>>>>> kernel: usb0: HOST MAC aa:bb:cc:dd:ee:f2
>>>>> kernel: usb0: MAC aa:bb:cc:dd:ee:f1
>>>>> kernel: IPv6: ADDRCONF(NETDEV_CHANGE): usb0: link becomes ready
>>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>>> depth:115540359
>>>>> kernel: ------------[ cut here ]------------
>>>>> kernel: WARNING: CPU: 0 PID: 594 at drivers/usb/gadget/udc/core.c:278
>>>>> usb_ep_queue+0x75/0x80
>>>>> kernel: Modules linked in: usb_f_uac2 u_audio usb_f_mass_storage
>>>>> usb_f_eem u_ether usb_f_serial u_serial libcomposite rfcomm
>>>>> iptable_nat
>>>>> bnep snd_sof_nocodec spi_pxa2>
>>>>> kernel: CPU: 0 PID: 594 Comm: irq/14-dwc3 Not tainted
>>>>> 5.13.0-rc5-edison-acpi-standard #1
>>>>> kernel: Hardware name: Intel Corporation Merrifield/BODEGA BAY,
>>>>> BIOS 542
>>>>> 2015.01.21:18.19.48
>>>>> kernel: RIP: 0010:usb_ep_queue+0x75/0x80
>>>>> kernel: Code: 01 73 e4 48 8b 05 fb 63 06 01 48 85 c0 74 12 48 8b 78 08
>>>>> 44 89 e9 4c 89 e2 48 89 ee e8 74 05 00 00 44 89 e8 5d 41 5c 41 5d c3
>>>>> <0f> 0b 41 bd 94 ff ff ff >
>>>>> kernel: RSP: 0000:ffff91eec083fc98 EFLAGS: 00010082
>>>>> kernel: RAX: ffff8af20357d960 RBX: 0000000000000000 RCX:
>>>>> ffff8af202f06400
>>>>> kernel: RDX: 0000000000000a20 RSI: ffff8af208785780 RDI:
>>>>> ffff8af202e9ae00
>>>>> kernel: RBP: ffff8af202e9ae00 R08: 00000000000000c0 R09:
>>>>> ffff8af208785780
>>>>> kernel: R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
>>>>> ffff8af208785780
>>>>> kernel: R13: 0000000000000000 R14: ffff8af202e9ae00 R15:
>>>>> ffff8af203e26cc0
>>>>> kernel: FS:  0000000000000000(0000) GS:ffff8af23e200000(0000)
>>>>> knlGS:0000000000000000
>>>>> kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>> kernel: CR2: 000055e2c21f2100 CR3: 0000000003b38000 CR4:
>>>>> 00000000001006f0
>>>>> kernel: Call Trace:
>>>>> kernel:  u_audio_start_playback+0x107/0x1a0 [u_audio]
>>>>> kernel:  composite_setup+0x224/0x1ba0 [libcomposite]
>>>>> kernel:  ? dwc3_gadget_ep_queue+0xf6/0x1a0
>>>>> kernel:  ? usb_ep_queue+0x2a/0x80
>>>>> kernel:  ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>> kernel:  configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>> kernel:  dwc3_ep0_interrupt+0x469/0xa80
>>>>> kernel:  dwc3_thread_interrupt+0x8ee/0xf40
>>>>> kernel:  ? __wake_up_common_lock+0x85/0xb0
>>>>> kernel:  ? disable_irq_nosync+0x10/0x10
>>>>> kernel:  irq_thread_fn+0x1b/0x60
>>>>> kernel:  irq_thread+0xd6/0x170
>>>>> kernel:  ? irq_thread_check_affinity+0x70/0x70
>>>>> kernel:  ? irq_forced_thread_fn+0x70/0x70
>>>>> kernel:  kthread+0x116/0x130
>>>>> kernel:  ? kthread_create_worker_on_cpu+0x60/0x60
>>>>> kernel:  ret_from_fork+0x22/0x30
>>>>> kernel: ---[ end trace e5b9e28058c53584 ]---
>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>> kernel: dwc3 dwc3.0.auto: request 000000003c32dcc5 was not queued to
>>>>> ep5in
>>>>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to
>>>>> ep5in
>>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>>> depth:115540359
>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to
>>>>> ep5in
>>>>> kernel: dwc3 dwc3.0.auto: request 00000000036ac129 was not queued to
>>>>> ep5in
>>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>>> depth:115540359
>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>> kernel: dwc3 dwc3.0.auto: request 00000000ad1b8c18 was not queued to
>>>>> ep5in
>>>>> kernel: dwc3 dwc3.0.auto: request 00000000fbc71244 was not queued to
>>>>> ep5in
>>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>>> depth:115540359
>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>> kernel: dwc3 dwc3.0.auto: request 00000000fbc71244 was not queued to
>>>>> ep5in
>>>>> kernel: dwc3 dwc3.0.auto: request 00000000ad1b8c18 was not queued to
>>>>> ep5in
>>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>>> depth:115540359
>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>> kernel: dwc3 dwc3.0.auto: request 000000003c32dcc5 was not queued to
>>>>> ep5in
>>>>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to
>>>>> ep5in
>>>>>
>>>>> Removing uac2 from the config makes the call trace go away, but the
>>>>> R/W
>>>>> speed does not change.
>>>>>
>>>>>> :), not a problem...I've been working on getting the ideal set up for
>>>>>> the performance profiling for awhile, so anything I can do to make
>>>>>> sure
>>>>>> we get some good results.
>>>>>>
>>>>>>> I'll try with a ramdisk and let you know.
>>>>>>>
>>>>>> Thanks again for the testing, Ferry.
>>>>>>
>>>>>> Thanks
>>>>>> Wesley Cheng
>>>>>>
>>>>>>>> Thanks
>>>>>>>> Wesley Cheng
>>>>>>>>
>>>>>>>>>> Changes in V9:
>>>>>>>>>>      - Fixed incorrect patch in series.  Removed changes in
>>>>>>>>>> DTSI, as
>>>>>>>>>> dwc3-qcom will
>>>>>>>>>>        add the property by default from the kernel.
>>>>>>>>>>
>>>>>>>>>> Changes in V8:
>>>>>>>>>>      - Rebased to usb-testing
>>>>>>>>>>      - Using devm_kzalloc for adding txfifo property in dwc3-qcom
>>>>>>>>>>      - Removed DWC3 QCOM ACPI property for enabling the txfifo
>>>>>>>>>> resize
>>>>>>>>>>
>>>>>>>>>> Changes in V7:
>>>>>>>>>>      - Added a new property tx-fifo-max-num for limiting how much
>>>>>>>>>> fifo
>>>>>>>>>> space the
>>>>>>>>>>        resizing logic can allocate for endpoints with large burst
>>>>>>>>>> values.  This
>>>>>>>>>>        can differ across platforms, and tie in closely with
>>>>>>>>>> overall
>>>>>>>>>> system latency.
>>>>>>>>>>      - Added recommended checks for DWC32.
>>>>>>>>>>      - Added changes to set the tx-fifo-resize property from
>>>>>>>>>> dwc3-qcom by
>>>>>>>>>> default
>>>>>>>>>>        instead of modifying the current DTSI files.
>>>>>>>>>>      - Added comments on all APIs/variables introduced.
>>>>>>>>>>      - Updated the DWC3 YAML to include a better description
>>>>>>>>>> of the
>>>>>>>>>> tx-fifo-resize
>>>>>>>>>>        property and added an entry for tx-fifo-max-num.
>>>>>>>>>>
>>>>>>>>>> Changes in V6:
>>>>>>>>>>      - Rebased patches to usb-testing.
>>>>>>>>>>      - Renamed to PATCH series instead of RFC.
>>>>>>>>>>      - Checking for fs_descriptors instead of ss_descriptors for
>>>>>>>>>> determining the
>>>>>>>>>>        endpoint count for a particular configuration.
>>>>>>>>>>      - Re-ordered patch series to fix patch dependencies.
>>>>>>>>>>
>>>>>>>>>> Changes in V5:
>>>>>>>>>>      - Added check_config() logic, which is used to
>>>>>>>>>> communicate the
>>>>>>>>>> number of EPs
>>>>>>>>>>        used in a particular configuration.  Based on this, the
>>>>>>>>>> DWC3
>>>>>>>>>> gadget driver
>>>>>>>>>>        has the ability to know the maximum number of eps
>>>>>>>>>> utilized in
>>>>>>>>>> all
>>>>>>>>>> configs.
>>>>>>>>>>        This helps reduce unnecessary allocation to unused eps,
>>>>>>>>>> and will
>>>>>>>>>> catch fifo
>>>>>>>>>>        allocation issues at bind() time.
>>>>>>>>>>      - Fixed variable declaration to single line per variable,
>>>>>>>>>> and
>>>>>>>>>> reverse xmas.
>>>>>>>>>>      - Created a helper for fifo clearing, which is used by ep0.c
>>>>>>>>>>
>>>>>>>>>> Changes in V4:
>>>>>>>>>>      - Removed struct dwc3* as an argument for
>>>>>>>>>> dwc3_gadget_resize_tx_fifos()
>>>>>>>>>>      - Removed WARN_ON(1) in case we run out of fifo space
>>>>>>>>>>      Changes in V3:
>>>>>>>>>>      - Removed "Reviewed-by" tags
>>>>>>>>>>      - Renamed series back to RFC
>>>>>>>>>>      - Modified logic to ensure that fifo_size is reset if we
>>>>>>>>>> pass the
>>>>>>>>>> minimum
>>>>>>>>>>        threshold.  Tested with binding multiple FDs requesting 6
>>>>>>>>>> FIFOs.
>>>>>>>>>>
>>>>>>>>>> Changes in V2:
>>>>>>>>>>      - Modified TXFIFO resizing logic to ensure that each EP is
>>>>>>>>>> reserved a
>>>>>>>>>>        FIFO.
>>>>>>>>>>      - Removed dev_dbg() prints and fixed typos from patches
>>>>>>>>>>      - Added some more description on the dt-bindings commit
>>>>>>>>>> message
>>>>>>>>>>
>>>>>>>>>> Currently, there is no functionality to allow for resizing the
>>>>>>>>>> TXFIFOs, and
>>>>>>>>>> relying on the HW default setting for the TXFIFO depth.  In most
>>>>>>>>>> cases, the
>>>>>>>>>> HW default is probably sufficient, but for USB compositions that
>>>>>>>>>> contain
>>>>>>>>>> multiple functions that require EP bursting, the default settings
>>>>>>>>>> might not be enough.  Also to note, the current SW will assign an
>>>>>>>>>> EP to a
>>>>>>>>>> function driver w/o checking to see if the TXFIFO size for that
>>>>>>>>>> particular
>>>>>>>>>> EP is large enough. (this is a problem if there are multiple HW
>>>>>>>>>> defined
>>>>>>>>>> values for the TXFIFO size)
>>>>>>>>>>
>>>>>>>>>> It is mentioned in the SNPS databook that a minimum of TX FIFO
>>>>>>>>>> depth = 3
>>>>>>>>>> is required for an EP that supports bursting.  Otherwise, there
>>>>>>>>>> may be
>>>>>>>>>> frequent occurences of bursts ending.  For high bandwidth
>>>>>>>>>> functions,
>>>>>>>>>> such as data tethering (protocols that support data aggregation),
>>>>>>>>>> mass
>>>>>>>>>> storage, and media transfer protocol (over FFS), the bMaxBurst
>>>>>>>>>> value
>>>>>>>>>> can be
>>>>>>>>>> large, and a bigger TXFIFO depth may prove to be beneficial in
>>>>>>>>>> terms
>>>>>>>>>> of USB
>>>>>>>>>> throughput. (which can be associated to system access latency,
>>>>>>>>>> etc...)  It
>>>>>>>>>> allows for a more consistent burst of traffic, w/o any
>>>>>>>>>> interruptions, as
>>>>>>>>>> data is readily available in the FIFO.
>>>>>>>>>>
>>>>>>>>>> With testing done using the mass storage function driver, the
>>>>>>>>>> results
>>>>>>>>>> show
>>>>>>>>>> that with a larger TXFIFO depth, the bandwidth increased
>>>>>>>>>> significantly.
>>>>>>>>>>
>>>>>>>>>> Test Parameters:
>>>>>>>>>>      - Platform: Qualcomm SM8150
>>>>>>>>>>      - bMaxBurst = 6
>>>>>>>>>>      - USB req size = 256kB
>>>>>>>>>>      - Num of USB reqs = 16
>>>>>>>>>>      - USB Speed = Super-Speed
>>>>>>>>>>      - Function Driver: Mass Storage (w/ ramdisk)
>>>>>>>>>>      - Test Application: CrystalDiskMark
>>>>>>>>>>
>>>>>>>>>> Results:
>>>>>>>>>>
>>>>>>>>>> TXFIFO Depth = 3 max packets
>>>>>>>>>>
>>>>>>>>>> Test Case | Data Size | AVG tput (in MB/s)
>>>>>>>>>> -------------------------------------------
>>>>>>>>>> Sequential|1 GB x     |
>>>>>>>>>> Read      |9 loops    | 193.60
>>>>>>>>>>          |           | 195.86
>>>>>>>>>>               |           | 184.77
>>>>>>>>>>               |           | 193.60
>>>>>>>>>> -------------------------------------------
>>>>>>>>>>
>>>>>>>>>> TXFIFO Depth = 6 max packets
>>>>>>>>>>
>>>>>>>>>> Test Case | Data Size | AVG tput (in MB/s)
>>>>>>>>>> -------------------------------------------
>>>>>>>>>> Sequential|1 GB x     |
>>>>>>>>>> Read      |9 loops    | 287.35
>>>>>>>>>>          |           | 304.94
>>>>>>>>>>               |           | 289.64
>>>>>>>>>>               |           | 293.61
>>>>>>>>>> -------------------------------------------
>>>>>>>>>>
>>>>>>>>>> Wesley Cheng (6):
>>>>>>>>>>       usb: gadget: udc: core: Introduce check_config to verify
>>>>>>>>>> USB
>>>>>>>>>>         configuration
>>>>>>>>>>       usb: gadget: configfs: Check USB configuration before
>>>>>>>>>> adding
>>>>>>>>>>       usb: dwc3: Resize TX FIFOs to meet EP bursting requirements
>>>>>>>>>>       of: Add stub for of_add_property()
>>>>>>>>>>       usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by
>>>>>>>>>> default
>>>>>>>>>>       dt-bindings: usb: dwc3: Update dwc3 TX fifo properties
>>>>>>>>>>
>>>>>>>>>>      .../devicetree/bindings/usb/snps,dwc3.yaml         |  15 +-
>>>>>>>>>>      drivers/usb/dwc3/core.c                            |   9 +
>>>>>>>>>>      drivers/usb/dwc3/core.h                            |  15 ++
>>>>>>>>>>      drivers/usb/dwc3/dwc3-qcom.c                       |   9 +
>>>>>>>>>>      drivers/usb/dwc3/ep0.c                             |   2 +
>>>>>>>>>>      drivers/usb/dwc3/gadget.c                          | 212
>>>>>>>>>> +++++++++++++++++++++
>>>>>>>>>>      drivers/usb/gadget/configfs.c                      |  22 +++
>>>>>>>>>>      drivers/usb/gadget/udc/core.c                      |  25 +++
>>>>>>>>>>      include/linux/of.h                                 |   5 +
>>>>>>>>>>      include/linux/usb/gadget.h                         |   5 +
>>>>>>>>>>      10 files changed, 317 insertions(+), 2 deletions(-)
>>>>>>>>>>

-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 2/6] usb: gadget: configfs: Check USB configuration before adding
  2021-06-22  6:05       ` Greg KH
@ 2021-06-23  9:38         ` Wesley Cheng
  2021-06-23 11:35           ` Greg KH
  0 siblings, 1 reply; 73+ messages in thread
From: Wesley Cheng @ 2021-06-23  9:38 UTC (permalink / raw)
  To: Greg KH
  Cc: balbi, robh+dt, agross, bjorn.andersson, frowand.list, linux-usb,
	linux-kernel, devicetree, linux-arm-msm, jackp, fntoth,
	heikki.krogerus, andy.shevchenko



On 6/21/2021 11:05 PM, Greg KH wrote:
> On Mon, Jun 21, 2021 at 10:27:09PM -0700, Wesley Cheng wrote:
>>
>>
>> On 6/17/2021 4:07 AM, Greg KH wrote:
>>> On Thu, Jun 17, 2021 at 02:58:15AM -0700, Wesley Cheng wrote:
>>>> Ensure that the USB gadget is able to support the configuration being
>>>> added based on the number of endpoints required from all interfaces.  This
>>>> is for accounting for any bandwidth or space limitations.
>>>>
>>>> Signed-off-by: Wesley Cheng <wcheng@codeaurora.org>
>>>> ---
>>>>  drivers/usb/gadget/configfs.c | 22 ++++++++++++++++++++++
>>>>  1 file changed, 22 insertions(+)
>>>>
>>>> diff --git a/drivers/usb/gadget/configfs.c b/drivers/usb/gadget/configfs.c
>>>> index 15a607c..76b9983 100644
>>>> --- a/drivers/usb/gadget/configfs.c
>>>> +++ b/drivers/usb/gadget/configfs.c
>>>> @@ -1374,6 +1374,7 @@ static int configfs_composite_bind(struct usb_gadget *gadget,
>>>>  		struct usb_function *f;
>>>>  		struct usb_function *tmp;
>>>>  		struct gadget_config_name *cn;
>>>> +		unsigned long ep_map = 0;
>>>>  
>>>>  		if (gadget_is_otg(gadget))
>>>>  			c->descriptors = otg_desc;
>>>> @@ -1403,7 +1404,28 @@ static int configfs_composite_bind(struct usb_gadget *gadget,
>>>>  				list_add(&f->list, &cfg->func_list);
>>>>  				goto err_purge_funcs;
>>>>  			}
>>>> +			if (f->fs_descriptors) {
>>>> +				struct usb_descriptor_header **d;
>>>> +
>>>> +				d = f->fs_descriptors;
>>>> +				for (; *d; ++d) {
>>
>> Hi Greg,
>>
>> Thanks for the review and feedback.
>>
>>>
>>> With this check, there really is not a need to check for
>>> f->fs_descriptors above in the if statement, right?
>>>
>>
>> f->fs_descriptor will carry the table of descriptors that a particular
>> function driver has assigned to it.  The for loop here, will dereference
>> the individual descriptors within that descriptor array, so we need to
>> first ensure the descriptor array is present before traversing through
>> the individual entries/elements.
> 
> Ah, it's a dereference of an array element.  Subtle.  Tricky.  Messy :(
> 
>>>> +					struct usb_endpoint_descriptor *ep;
>>>> +					int addr;
>>>> +
>>>> +					if ((*d)->bDescriptorType != USB_DT_ENDPOINT)
>>>> +						continue;
>>>> +
>>>> +					ep = (struct usb_endpoint_descriptor *)*d;
>>>> +					addr = ((ep->bEndpointAddress & 0x80) >> 3) |
>>>> +						(ep->bEndpointAddress & 0x0f);
>>>
>>> Don't we have direction macros for this type of check?
>>>
>>
>> I don't believe we have a macro which would be able to convert the
>> bEndpointAddress field into the bit which needs to be set, assuming that
>> the 32bit ep_map has the lower 16bits carrying OUT EPs, and the upper
>> 16bits carrying the IN EPs.

Hi Greg,

> 
> We have macros to tell if an endpoint is IN or OUT, please use those.
> 
> And this "cram the whole thing into 64 bits" is not obvious at all.
> Just pass around the original pointer to the descriptors if someone
> wants to use it or not, don't make up yet-another-data-structure here
> for no good reason.  We aren't so memory constrained we need to pack
> stuff into bits here.
> 

Hmm ok, what I can do is to move this logic into the check_config()
callback itself, which is implemented by the UDC driver.  So now, the
DWC3 will have to do something similar to what is done here, ie loop the
EP descriptors for each function to determine the number of IN endpoints
being used.

Thanks
Wesley Cheng

> thanks,
> 
> greg k-h
> 

-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 2/6] usb: gadget: configfs: Check USB configuration before adding
  2021-06-23  9:38         ` Wesley Cheng
@ 2021-06-23 11:35           ` Greg KH
  2021-06-23 21:44             ` Wesley Cheng
  0 siblings, 1 reply; 73+ messages in thread
From: Greg KH @ 2021-06-23 11:35 UTC (permalink / raw)
  To: Wesley Cheng
  Cc: balbi, robh+dt, agross, bjorn.andersson, frowand.list, linux-usb,
	linux-kernel, devicetree, linux-arm-msm, jackp, fntoth,
	heikki.krogerus, andy.shevchenko

On Wed, Jun 23, 2021 at 02:38:55AM -0700, Wesley Cheng wrote:
> 
> 
> On 6/21/2021 11:05 PM, Greg KH wrote:
> > On Mon, Jun 21, 2021 at 10:27:09PM -0700, Wesley Cheng wrote:
> >>
> >>
> >> On 6/17/2021 4:07 AM, Greg KH wrote:
> >>> On Thu, Jun 17, 2021 at 02:58:15AM -0700, Wesley Cheng wrote:
> >>>> Ensure that the USB gadget is able to support the configuration being
> >>>> added based on the number of endpoints required from all interfaces.  This
> >>>> is for accounting for any bandwidth or space limitations.
> >>>>
> >>>> Signed-off-by: Wesley Cheng <wcheng@codeaurora.org>
> >>>> ---
> >>>>  drivers/usb/gadget/configfs.c | 22 ++++++++++++++++++++++
> >>>>  1 file changed, 22 insertions(+)
> >>>>
> >>>> diff --git a/drivers/usb/gadget/configfs.c b/drivers/usb/gadget/configfs.c
> >>>> index 15a607c..76b9983 100644
> >>>> --- a/drivers/usb/gadget/configfs.c
> >>>> +++ b/drivers/usb/gadget/configfs.c
> >>>> @@ -1374,6 +1374,7 @@ static int configfs_composite_bind(struct usb_gadget *gadget,
> >>>>  		struct usb_function *f;
> >>>>  		struct usb_function *tmp;
> >>>>  		struct gadget_config_name *cn;
> >>>> +		unsigned long ep_map = 0;
> >>>>  
> >>>>  		if (gadget_is_otg(gadget))
> >>>>  			c->descriptors = otg_desc;
> >>>> @@ -1403,7 +1404,28 @@ static int configfs_composite_bind(struct usb_gadget *gadget,
> >>>>  				list_add(&f->list, &cfg->func_list);
> >>>>  				goto err_purge_funcs;
> >>>>  			}
> >>>> +			if (f->fs_descriptors) {
> >>>> +				struct usb_descriptor_header **d;
> >>>> +
> >>>> +				d = f->fs_descriptors;
> >>>> +				for (; *d; ++d) {
> >>
> >> Hi Greg,
> >>
> >> Thanks for the review and feedback.
> >>
> >>>
> >>> With this check, there really is not a need to check for
> >>> f->fs_descriptors above in the if statement, right?
> >>>
> >>
> >> f->fs_descriptor will carry the table of descriptors that a particular
> >> function driver has assigned to it.  The for loop here, will dereference
> >> the individual descriptors within that descriptor array, so we need to
> >> first ensure the descriptor array is present before traversing through
> >> the individual entries/elements.
> > 
> > Ah, it's a dereference of an array element.  Subtle.  Tricky.  Messy :(
> > 
> >>>> +					struct usb_endpoint_descriptor *ep;
> >>>> +					int addr;
> >>>> +
> >>>> +					if ((*d)->bDescriptorType != USB_DT_ENDPOINT)
> >>>> +						continue;
> >>>> +
> >>>> +					ep = (struct usb_endpoint_descriptor *)*d;
> >>>> +					addr = ((ep->bEndpointAddress & 0x80) >> 3) |
> >>>> +						(ep->bEndpointAddress & 0x0f);
> >>>
> >>> Don't we have direction macros for this type of check?
> >>>
> >>
> >> I don't believe we have a macro which would be able to convert the
> >> bEndpointAddress field into the bit which needs to be set, assuming that
> >> the 32bit ep_map has the lower 16bits carrying OUT EPs, and the upper
> >> 16bits carrying the IN EPs.
> 
> Hi Greg,
> 
> > 
> > We have macros to tell if an endpoint is IN or OUT, please use those.
> > 
> > And this "cram the whole thing into 64 bits" is not obvious at all.
> > Just pass around the original pointer to the descriptors if someone
> > wants to use it or not, don't make up yet-another-data-structure here
> > for no good reason.  We aren't so memory constrained we need to pack
> > stuff into bits here.
> > 
> 
> Hmm ok, what I can do is to move this logic into the check_config()
> callback itself, which is implemented by the UDC driver.  So now, the
> DWC3 will have to do something similar to what is done here, ie loop the
> EP descriptors for each function to determine the number of IN endpoints
> being used.

We have common USB core functions for this, why can't you just use them?

Please do not take data that we already have in one format, and convert
it to another one just for a single driver to consume.  That's
pointless.

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 2/6] usb: gadget: configfs: Check USB configuration before adding
  2021-06-23 11:35           ` Greg KH
@ 2021-06-23 21:44             ` Wesley Cheng
  2021-06-24 12:09               ` Greg KH
  0 siblings, 1 reply; 73+ messages in thread
From: Wesley Cheng @ 2021-06-23 21:44 UTC (permalink / raw)
  To: Greg KH
  Cc: balbi, robh+dt, agross, bjorn.andersson, frowand.list, linux-usb,
	linux-kernel, devicetree, linux-arm-msm, jackp, fntoth,
	heikki.krogerus, andy.shevchenko



On 6/23/2021 4:35 AM, Greg KH wrote:
> On Wed, Jun 23, 2021 at 02:38:55AM -0700, Wesley Cheng wrote:
>>
>>
>> On 6/21/2021 11:05 PM, Greg KH wrote:
>>> On Mon, Jun 21, 2021 at 10:27:09PM -0700, Wesley Cheng wrote:
>>>>
>>>>
>>>> On 6/17/2021 4:07 AM, Greg KH wrote:
>>>>> On Thu, Jun 17, 2021 at 02:58:15AM -0700, Wesley Cheng wrote:
>>>>>> Ensure that the USB gadget is able to support the configuration being
>>>>>> added based on the number of endpoints required from all interfaces.  This
>>>>>> is for accounting for any bandwidth or space limitations.
>>>>>>
>>>>>> Signed-off-by: Wesley Cheng <wcheng@codeaurora.org>
>>>>>> ---
>>>>>>  drivers/usb/gadget/configfs.c | 22 ++++++++++++++++++++++
>>>>>>  1 file changed, 22 insertions(+)
>>>>>>
>>>>>> diff --git a/drivers/usb/gadget/configfs.c b/drivers/usb/gadget/configfs.c
>>>>>> index 15a607c..76b9983 100644
>>>>>> --- a/drivers/usb/gadget/configfs.c
>>>>>> +++ b/drivers/usb/gadget/configfs.c
>>>>>> @@ -1374,6 +1374,7 @@ static int configfs_composite_bind(struct usb_gadget *gadget,
>>>>>>  		struct usb_function *f;
>>>>>>  		struct usb_function *tmp;
>>>>>>  		struct gadget_config_name *cn;
>>>>>> +		unsigned long ep_map = 0;
>>>>>>  
>>>>>>  		if (gadget_is_otg(gadget))
>>>>>>  			c->descriptors = otg_desc;
>>>>>> @@ -1403,7 +1404,28 @@ static int configfs_composite_bind(struct usb_gadget *gadget,
>>>>>>  				list_add(&f->list, &cfg->func_list);
>>>>>>  				goto err_purge_funcs;
>>>>>>  			}
>>>>>> +			if (f->fs_descriptors) {
>>>>>> +				struct usb_descriptor_header **d;
>>>>>> +
>>>>>> +				d = f->fs_descriptors;
>>>>>> +				for (; *d; ++d) {
>>>>
>>>> Hi Greg,
>>>>
>>>> Thanks for the review and feedback.
>>>>
>>>>>
>>>>> With this check, there really is not a need to check for
>>>>> f->fs_descriptors above in the if statement, right?
>>>>>
>>>>
>>>> f->fs_descriptor will carry the table of descriptors that a particular
>>>> function driver has assigned to it.  The for loop here, will dereference
>>>> the individual descriptors within that descriptor array, so we need to
>>>> first ensure the descriptor array is present before traversing through
>>>> the individual entries/elements.
>>>
>>> Ah, it's a dereference of an array element.  Subtle.  Tricky.  Messy :(
>>>
>>>>>> +					struct usb_endpoint_descriptor *ep;
>>>>>> +					int addr;
>>>>>> +
>>>>>> +					if ((*d)->bDescriptorType != USB_DT_ENDPOINT)
>>>>>> +						continue;
>>>>>> +
>>>>>> +					ep = (struct usb_endpoint_descriptor *)*d;
>>>>>> +					addr = ((ep->bEndpointAddress & 0x80) >> 3) |
>>>>>> +						(ep->bEndpointAddress & 0x0f);
>>>>>
>>>>> Don't we have direction macros for this type of check?
>>>>>
>>>>
>>>> I don't believe we have a macro which would be able to convert the
>>>> bEndpointAddress field into the bit which needs to be set, assuming that
>>>> the 32bit ep_map has the lower 16bits carrying OUT EPs, and the upper
>>>> 16bits carrying the IN EPs.
>>
>> Hi Greg,
>>
>>>
>>> We have macros to tell if an endpoint is IN or OUT, please use those.
>>>
>>> And this "cram the whole thing into 64 bits" is not obvious at all.
>>> Just pass around the original pointer to the descriptors if someone
>>> wants to use it or not, don't make up yet-another-data-structure here
>>> for no good reason.  We aren't so memory constrained we need to pack
>>> stuff into bits here.
>>>
>>
>> Hmm ok, what I can do is to move this logic into the check_config()
>> callback itself, which is implemented by the UDC driver.  So now, the
>> DWC3 will have to do something similar to what is done here, ie loop the
>> EP descriptors for each function to determine the number of IN endpoints
>> being used.

Hi Greg,

> 
> We have common USB core functions for this, why can't you just use them?
> 

So, I've tried to use pre-existing mechanisms there in the USB core, but
they are not populated correctly at the time of function binding.  I
will highlight some of the things I've tried, and why they do not work.
 If possible, if you could point which core functions can achieve what
we are trying to do here, that would help as well.

  - f->endpoints - This is a bitmap which carries the endpoints used by
a particular function driver.  This does not work, as this is set during
receiving the SET_CONFIG packet.  (we need this during the function
driver binding stage)

  - gadget->in_epnum/gadget->out_epnum - This carries the count of
endpoints used per configuration.  This would be perfect, but this count
is only incremented when we are not matching EPs using the EP name.  So
in designs where the EP name is used to match, it can not be used.

 - gadget->ep_list - I can use this now in the check_config() to iterate
through the list of eps to see which ones have been claimed for a
particular configuration.

So just to re-iterate, the TXFIFO resize logic kicks in when the host
sends the SET_CONFIG packet, which is the "end" of USB enumeration.  We
had discussed a concern previously where, what if we run the resize
logic, and there is not enough internal memory.  We'd end up with an
enumerated device w/ certain functions broken.

This is where the check_config() comes into the picture.  It uses the
number of endpoints collected during the bind() stage, and checks to
make sure the resize logic can at least allocate 1 TXFIFO per endpoint.
 If it can not, then it will fail the bind sequence.

Thanks
Wesley Cheng
> Please do not take data that we already have in one format, and convert
> it to another one just for a single driver to consume.  That's
> pointless.
> 
> thanks,
> 
> greg k-h
> 

-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 2/6] usb: gadget: configfs: Check USB configuration before adding
  2021-06-23 21:44             ` Wesley Cheng
@ 2021-06-24 12:09               ` Greg KH
  0 siblings, 0 replies; 73+ messages in thread
From: Greg KH @ 2021-06-24 12:09 UTC (permalink / raw)
  To: Wesley Cheng
  Cc: balbi, robh+dt, agross, bjorn.andersson, frowand.list, linux-usb,
	linux-kernel, devicetree, linux-arm-msm, jackp, fntoth,
	heikki.krogerus, andy.shevchenko

On Wed, Jun 23, 2021 at 02:44:31PM -0700, Wesley Cheng wrote:
> 
> 
> On 6/23/2021 4:35 AM, Greg KH wrote:
> > On Wed, Jun 23, 2021 at 02:38:55AM -0700, Wesley Cheng wrote:
> >>
> >>
> >> On 6/21/2021 11:05 PM, Greg KH wrote:
> >>> On Mon, Jun 21, 2021 at 10:27:09PM -0700, Wesley Cheng wrote:
> >>>>
> >>>>
> >>>> On 6/17/2021 4:07 AM, Greg KH wrote:
> >>>>> On Thu, Jun 17, 2021 at 02:58:15AM -0700, Wesley Cheng wrote:
> >>>>>> Ensure that the USB gadget is able to support the configuration being
> >>>>>> added based on the number of endpoints required from all interfaces.  This
> >>>>>> is for accounting for any bandwidth or space limitations.
> >>>>>>
> >>>>>> Signed-off-by: Wesley Cheng <wcheng@codeaurora.org>
> >>>>>> ---
> >>>>>>  drivers/usb/gadget/configfs.c | 22 ++++++++++++++++++++++
> >>>>>>  1 file changed, 22 insertions(+)
> >>>>>>
> >>>>>> diff --git a/drivers/usb/gadget/configfs.c b/drivers/usb/gadget/configfs.c
> >>>>>> index 15a607c..76b9983 100644
> >>>>>> --- a/drivers/usb/gadget/configfs.c
> >>>>>> +++ b/drivers/usb/gadget/configfs.c
> >>>>>> @@ -1374,6 +1374,7 @@ static int configfs_composite_bind(struct usb_gadget *gadget,
> >>>>>>  		struct usb_function *f;
> >>>>>>  		struct usb_function *tmp;
> >>>>>>  		struct gadget_config_name *cn;
> >>>>>> +		unsigned long ep_map = 0;
> >>>>>>  
> >>>>>>  		if (gadget_is_otg(gadget))
> >>>>>>  			c->descriptors = otg_desc;
> >>>>>> @@ -1403,7 +1404,28 @@ static int configfs_composite_bind(struct usb_gadget *gadget,
> >>>>>>  				list_add(&f->list, &cfg->func_list);
> >>>>>>  				goto err_purge_funcs;
> >>>>>>  			}
> >>>>>> +			if (f->fs_descriptors) {
> >>>>>> +				struct usb_descriptor_header **d;
> >>>>>> +
> >>>>>> +				d = f->fs_descriptors;
> >>>>>> +				for (; *d; ++d) {
> >>>>
> >>>> Hi Greg,
> >>>>
> >>>> Thanks for the review and feedback.
> >>>>
> >>>>>
> >>>>> With this check, there really is not a need to check for
> >>>>> f->fs_descriptors above in the if statement, right?
> >>>>>
> >>>>
> >>>> f->fs_descriptor will carry the table of descriptors that a particular
> >>>> function driver has assigned to it.  The for loop here, will dereference
> >>>> the individual descriptors within that descriptor array, so we need to
> >>>> first ensure the descriptor array is present before traversing through
> >>>> the individual entries/elements.
> >>>
> >>> Ah, it's a dereference of an array element.  Subtle.  Tricky.  Messy :(
> >>>
> >>>>>> +					struct usb_endpoint_descriptor *ep;
> >>>>>> +					int addr;
> >>>>>> +
> >>>>>> +					if ((*d)->bDescriptorType != USB_DT_ENDPOINT)
> >>>>>> +						continue;
> >>>>>> +
> >>>>>> +					ep = (struct usb_endpoint_descriptor *)*d;
> >>>>>> +					addr = ((ep->bEndpointAddress & 0x80) >> 3) |
> >>>>>> +						(ep->bEndpointAddress & 0x0f);
> >>>>>
> >>>>> Don't we have direction macros for this type of check?
> >>>>>
> >>>>
> >>>> I don't believe we have a macro which would be able to convert the
> >>>> bEndpointAddress field into the bit which needs to be set, assuming that
> >>>> the 32bit ep_map has the lower 16bits carrying OUT EPs, and the upper
> >>>> 16bits carrying the IN EPs.
> >>
> >> Hi Greg,
> >>
> >>>
> >>> We have macros to tell if an endpoint is IN or OUT, please use those.
> >>>
> >>> And this "cram the whole thing into 64 bits" is not obvious at all.
> >>> Just pass around the original pointer to the descriptors if someone
> >>> wants to use it or not, don't make up yet-another-data-structure here
> >>> for no good reason.  We aren't so memory constrained we need to pack
> >>> stuff into bits here.
> >>>
> >>
> >> Hmm ok, what I can do is to move this logic into the check_config()
> >> callback itself, which is implemented by the UDC driver.  So now, the
> >> DWC3 will have to do something similar to what is done here, ie loop the
> >> EP descriptors for each function to determine the number of IN endpoints
> >> being used.
> 
> Hi Greg,
> 
> > 
> > We have common USB core functions for this, why can't you just use them?
> > 
> 
> So, I've tried to use pre-existing mechanisms there in the USB core, but
> they are not populated correctly at the time of function binding.  I
> will highlight some of the things I've tried, and why they do not work.
>  If possible, if you could point which core functions can achieve what
> we are trying to do here, that would help as well.

We have functions to detect the IN/OUT of an endpoint, use them.

We also have functions that determine how many endpoints of each type
that a device has, why can you not use them as well?  Are they only
valid for driver structures, not gadget ones?

>   - f->endpoints - This is a bitmap which carries the endpoints used by
> a particular function driver.  This does not work, as this is set during
> receiving the SET_CONFIG packet.  (we need this during the function
> driver binding stage)
> 
>   - gadget->in_epnum/gadget->out_epnum - This carries the count of
> endpoints used per configuration.  This would be perfect, but this count
> is only incremented when we are not matching EPs using the EP name.  So
> in designs where the EP name is used to match, it can not be used.
> 
>  - gadget->ep_list - I can use this now in the check_config() to iterate
> through the list of eps to see which ones have been claimed for a
> particular configuration.
> 
> So just to re-iterate, the TXFIFO resize logic kicks in when the host
> sends the SET_CONFIG packet, which is the "end" of USB enumeration.  We
> had discussed a concern previously where, what if we run the resize
> logic, and there is not enough internal memory.  We'd end up with an
> enumerated device w/ certain functions broken.

Where are you running out of memory?  In the gadget kernel?  Or
somewhere else?

> This is where the check_config() comes into the picture.  It uses the
> number of endpoints collected during the bind() stage, and checks to
> make sure the resize logic can at least allocate 1 TXFIFO per endpoint.
>  If it can not, then it will fail the bind sequence.

That's fine, I am just worried about your crazy "pack all the bits into
a u64 for no good reason" logic here.

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-06-23  7:50                   ` Wesley Cheng
@ 2021-06-25 21:10                     ` Ferry Toth
  2021-06-25 21:19                       ` Wesley Cheng
  0 siblings, 1 reply; 73+ messages in thread
From: Ferry Toth @ 2021-06-25 21:10 UTC (permalink / raw)
  To: Wesley Cheng, balbi, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list
  Cc: linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	heikki.krogerus, andy.shevchenko

Hi

Op 23-06-2021 om 09:50 schreef Wesley Cheng:
>
> On 6/22/2021 3:12 PM, Ferry Toth wrote:
>> Hi
>>
>> Op 23-06-2021 om 00:00 schreef Wesley Cheng:
>>> On 6/22/2021 1:09 PM, Ferry Toth wrote:
>>>> Hi
>>>>
>>>> Op 22-06-2021 om 20:38 schreef Wesley Cheng:
>>>>> On 6/19/2021 5:40 AM, Ferry Toth wrote:
>>>>>> Hi
>>>>>>
>>>>>> Op 18-06-2021 om 00:25 schreef Wesley Cheng:
>>>>>>> Hi,
>>>>>>>
>>>>>>> On 6/17/2021 2:55 PM, Ferry Toth wrote:
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> Op 17-06-2021 om 23:48 schreef Wesley Cheng:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On 6/17/2021 2:01 PM, Ferry Toth wrote:
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> Op 17-06-2021 om 11:58 schreef Wesley Cheng:
>>>>>>>>>>> Changes in V10:
>>>>>>>>>>>       - Fixed compilation errors in config where OF is not used
>>>>>>>>>>> (error due to
>>>>>>>>>>>         unknown symbol for of_add_property()).  Add
>>>>>>>>>>> of_add_property()
>>>>>>>>>>> stub.
>>>>>>>>>>>       - Fixed compilation warning for incorrect argument being
>>>>>>>>>>> passed to
>>>>>>>>>>> dwc3_mdwidth
>>>>>>>>>> This fixes the OOPS I had in V9. I do not see any change in
>>>>>>>>>> performance
>>>>>>>>>> on Merrifield though.
>>>>>>>>> I see...thanks Ferry! With your testing, are you writing to the
>>>>>>>>> device's
>>>>>>>>> internal storage (ie UFS, eMMC, etc...), or did you use a
>>>>>>>>> ramdisk as
>>>>>>>>> well?
>>>>>>>> In this case I just tested the EEM path using iperf3.
>>>>>>>>
>>>>>>> Got it.  I don't believe f_eem will use a high enough (if at all)
>>>>>>> bMaxBurst value to change the TXFIFO size.
>>>>>>>
>>>>>>>>> If not with a ramdisk, we might want to give that a try to avoid
>>>>>>>>> the
>>>>>>>>> storage path being the bottleneck.  You can use "dd" to create an
>>>>>>>>> empty
>>>>>>>>> file, and then just use that as the LUN's backing file.
>>>>>>>>>
>>>>>>>>> echo ramdisk.img >
>>>>>>>>> /sys/kernel/config/usb_gadget/g1/functions/mass_storage.0/lun.0/file
>>>>>>>>>
>>>>>>>> Ah, why didn't I think of that. I have currently mass storage setup
>>>>>>>> with
>>>>>>>> eMMC but it seems that is indeed the bottleneck.
>>>>>>>>
>>>>>> I created a 64MB disk following the instructions here
>>>>>> http://www.linux-usb.org/gadget/file_storage.html (that seems a little
>>>>>> outdated, at least I can not start the first partition at sector 8,
>>>>>> but
>>>>>> minimum 2048), and added a test file on it.
>>>>>>
>>>>>> I then copy the file to /dev/shm prior to setting configfs (composite
>>>>>> device gser/eem/mass_storage/uac2).
>>>>>>
>>>>>> journal shows:
>>>>>>
>>>>>> kernel: Mass Storage Function, version: 2009/09/11
>>>>>> kernel: LUN: removable file: (no medium)
>>>>>>
>>>>>> I don't know what that means, because I see the test file on the
>>>>>> ramdisk.
>>>>>>
>>>>>> Then I again used gnome disks to benchmark (read/write 10MB):
>>>>>>
>>>>>> With V10 on top v5.13.0-rc5:
>>>>>>
>>>>>> R/W speed = 35.6/35.8MB/s, access time 0.35ms
>>>>>>
>>>>>> With no patches on top v5.12.0:
>>>>>>
>>>>>> R/W speed = 35.7/36.1MB/s, access time 0.35ms
>>>>> Hi Ferry,
>>>>>
>>>>>> I see no speed difference (and it's about the same as with the eMMC
>>>>>> backed disk). But the patches are causing a new call trace
>>>>>>
>>>>> Would you happen to know what DWC3 controller revision the device is
>>>>> using?  The callstack print occurs, because it looks like it ran out of
>>>>> internal memory, although there should be logic present for making sure
>>>>> that at least there is enough room for 1 FIFO per endpoint.  (possibly
>>>>> the logic/math depends on the controller revision)
>>> Hi Ferry,
>>>
>>>> Do you know where I could find that in a file on the device?
>>>>
>>> Maybe you can just dump the DWC3 registers?
>>> cat /sys/kernel/debug/usb/<controller name>/regdump
>> This isin host mode:
>>
>> ~# cat /sys/kernel/debug/usb/dwc3.0.auto/regdump
>> GSBUSCFG0 = 0x00000006
>> GSBUSCFG1 = 0x00000f00
>> GTXTHRCFG = 0x230a0000
>> GRXTHRCFG = 0x22800000
>> GCTL = 0x45801002
>> GEVTEN = 0x00000000
>> GSTS = 0x3e800001
>> GUCTL1 = 0x00000000
>> GSNPSID = 0x5533250a
>> GGPIO = 0x00000000
>> GUID = 0x00050d00
>> GUCTL = 0x0200ce00
>> GBUSERRADDR0 = 0x00000000
>> GBUSERRADDR1 = 0x00000000
> ...
> Hi Ferry,
>
> Great!  Thanks, that got me the information regarding the DWC3 revision
> you're using. (its version 2.50a)  I checked the DWC3 databook and looks
> like certain versions have some different math to calculate the TXFIFO
> size.  I have taken this into account and factored that in in the next
> revision.
>
> Is there a way you could collect the regdump in device mode w/ mass
> storage only composition after enumerating with the PC?  That would help
> confirm how the TXFIFO resizing logic is working on your platform.

Sorry for the wait. Yes I have serial console. So here is in device 
mode, mass_storage only, enumerated but not mounted:

GSBUSCFG0 = 0x00000006
GSBUSCFG1 = 0x00000f00
GTXTHRCFG = 0x230a0000
GRXTHRCFG = 0x02800000
GCTL = 0x45802002
GEVTEN = 0x00000000
GSTS = 0x3e800000
GUCTL1 = 0x00000000
GSNPSID = 0x5533250a
GGPIO = 0x00000000
GUID = 0x00050d00
GUCTL = 0x0200ce00
GBUSERRADDR0 = 0x00000000
GBUSERRADDR1 = 0x00000000
GPRTBIMAP0 = 0x00000000
GPRTBIMAP1 = 0x00000000
GHWPARAMS0 = 0x2020400a
GHWPARAMS1 = 0x0260c93b
GHWPARAMS2 = 0x008086a0
GHWPARAMS3 = 0x10420089
GHWPARAMS4 = 0x47a22004
GHWPARAMS5 = 0x04202088
GHWPARAMS6 = 0x0c00ac20
GHWPARAMS7 = 0x038807e6
GDBGFIFOSPACE = 0x00420000
GDBGLTSSM = 0x01090440
GDBGBMU = 0x20300000
GPRTBIMAP_HS0 = 0x00000000
GPRTBIMAP_HS1 = 0x00000000
GPRTBIMAP_FS0 = 0x00000000
GPRTBIMAP_FS1 = 0x00000000
GUSB2PHYCFG(0) = 0x0000a710
GUSB2PHYCFG(1) = 0x00000000
GUSB2PHYCFG(2) = 0x00000000
GUSB2PHYCFG(3) = 0x00000000
GUSB2PHYCFG(4) = 0x00000000
GUSB2PHYCFG(5) = 0x00000000
GUSB2PHYCFG(6) = 0x00000000
GUSB2PHYCFG(7) = 0x00000000
GUSB2PHYCFG(8) = 0x00000000
GUSB2PHYCFG(9) = 0x00000000
GUSB2PHYCFG(10) = 0x00000000
GUSB2PHYCFG(11) = 0x00000000
GUSB2PHYCFG(12) = 0x00000000
GUSB2PHYCFG(13) = 0x00000000
GUSB2PHYCFG(14) = 0x00000000
GUSB2PHYCFG(15) = 0x00000000
GUSB2I2CCTL(0) = 0x00000000
GUSB2I2CCTL(1) = 0x00000000
GUSB2I2CCTL(2) = 0x00000000
GUSB2I2CCTL(3) = 0x00000000
GUSB2I2CCTL(4) = 0x00000000
GUSB2I2CCTL(5) = 0x00000000
GUSB2I2CCTL(6) = 0x00000000
GUSB2I2CCTL(7) = 0x00000000
GUSB2I2CCTL(8) = 0x00000000
GUSB2I2CCTL(9) = 0x00000000
GUSB2I2CCTL(10) = 0x00000000
GUSB2I2CCTL(11) = 0x00000000
GUSB2I2CCTL(12) = 0x00000000
GUSB2I2CCTL(13) = 0x00000000
GUSB2I2CCTL(14) = 0x00000000
GUSB2I2CCTL(15) = 0x00000000
GUSB2PHYACC(0) = 0x014a0000
GUSB2PHYACC(1) = 0x00000000
GUSB2PHYACC(2) = 0x00000000
GUSB2PHYACC(3) = 0x00000000
GUSB2PHYACC(4) = 0x00000000
GUSB2PHYACC(5) = 0x00000000
GUSB2PHYACC(6) = 0x00000000
GUSB2PHYACC(7) = 0x00000000
GUSB2PHYACC(8) = 0x00000000
GUSB2PHYACC(9) = 0x00000000
GUSB2PHYACC(10) = 0x00000000
GUSB2PHYACC(11) = 0x00000000
GUSB2PHYACC(12) = 0x00000000
GUSB2PHYACC(13) = 0x00000000
GUSB2PHYACC(14) = 0x00000000
GUSB2PHYACC(15) = 0x00000000
GUSB3PIPECTL(0) = 0x02000002
GUSB3PIPECTL(1) = 0x00000000
GUSB3PIPECTL(2) = 0x00000000
GUSB3PIPECTL(3) = 0x00000000
GUSB3PIPECTL(4) = 0x00000000
GUSB3PIPECTL(5) = 0x00000000
GUSB3PIPECTL(6) = 0x00000000
GUSB3PIPECTL(7) = 0x00000000
GUSB3PIPECTL(8) = 0x00000000
GUSB3PIPECTL(9) = 0x00000000
GUSB3PIPECTL(10) = 0x00000000
GUSB3PIPECTL(11) = 0x00000000
GUSB3PIPECTL(12) = 0x00000000
GUSB3PIPECTL(13) = 0x00000000
GUSB3PIPECTL(14) = 0x00000000
GUSB3PIPECTL(15) = 0x00000000
GTXFIFOSIZ(0) = 0x00000042
GTXFIFOSIZ(1) = 0x00000000
GTXFIFOSIZ(2) = 0x00420083
GTXFIFOSIZ(3) = 0x00000000
GTXFIFOSIZ(4) = 0x00000000
GTXFIFOSIZ(5) = 0x00000000
GTXFIFOSIZ(6) = 0x00000000
GTXFIFOSIZ(7) = 0x00000000
GTXFIFOSIZ(8) = 0x00000000
GTXFIFOSIZ(9) = 0x00000000
GTXFIFOSIZ(10) = 0x00000000
GTXFIFOSIZ(11) = 0x00000000
GTXFIFOSIZ(12) = 0x00000000
GTXFIFOSIZ(13) = 0x00000000
GTXFIFOSIZ(14) = 0x00000000
GTXFIFOSIZ(15) = 0x00000000
GTXFIFOSIZ(16) = 0x00000000
GTXFIFOSIZ(17) = 0x00000000
GTXFIFOSIZ(18) = 0x00000000
GTXFIFOSIZ(19) = 0x00000000
GTXFIFOSIZ(20) = 0x00000000
GTXFIFOSIZ(21) = 0x00000000
GTXFIFOSIZ(22) = 0x00000000
GTXFIFOSIZ(23) = 0x00000000
GTXFIFOSIZ(24) = 0x00000000
GTXFIFOSIZ(25) = 0x00000000
GTXFIFOSIZ(26) = 0x00000000
GTXFIFOSIZ(27) = 0x00000000
GTXFIFOSIZ(28) = 0x00000000
GTXFIFOSIZ(29) = 0x00000000
GTXFIFOSIZ(30) = 0x00000000
GTXFIFOSIZ(31) = 0x00000000
GRXFIFOSIZ(0) = 0x00000385
GRXFIFOSIZ(1) = 0x03850000
GRXFIFOSIZ(2) = 0x03850000
GRXFIFOSIZ(3) = 0x00000000
GRXFIFOSIZ(4) = 0x00000000
GRXFIFOSIZ(5) = 0x00000000
GRXFIFOSIZ(6) = 0x00000000
GRXFIFOSIZ(7) = 0x00000000
GRXFIFOSIZ(8) = 0x00000000
GRXFIFOSIZ(9) = 0x00000000
GRXFIFOSIZ(10) = 0x00000000
GRXFIFOSIZ(11) = 0x00000000
GRXFIFOSIZ(12) = 0x00000000
GRXFIFOSIZ(13) = 0x00000000
GRXFIFOSIZ(14) = 0x00000000
GRXFIFOSIZ(15) = 0x00000000
GRXFIFOSIZ(16) = 0x00000000
GRXFIFOSIZ(17) = 0x00000000
GRXFIFOSIZ(18) = 0x00000000
GRXFIFOSIZ(19) = 0x00000000
GRXFIFOSIZ(20) = 0x00000000
GRXFIFOSIZ(21) = 0x00000000
GRXFIFOSIZ(22) = 0x00000000
GRXFIFOSIZ(23) = 0x00000000
GRXFIFOSIZ(24) = 0x00000000
GRXFIFOSIZ(25) = 0x00000000
GRXFIFOSIZ(26) = 0x00000000
GRXFIFOSIZ(27) = 0x00000000
GRXFIFOSIZ(28) = 0x00000000
GRXFIFOSIZ(29) = 0x00000000
GRXFIFOSIZ(30) = 0x00000000
GRXFIFOSIZ(31) = 0x00000000
GEVNTADRLO(0) = 0x08b15000
GEVNTADRHI(0) = 0x00000000
GEVNTSIZ(0) = 0x00001000
GEVNTCOUNT(0) = 0x00000000
GHWPARAMS8 = 0x00000c00
DCFG = 0x008e082c
DCTL = 0x80f00a00
DEVTEN = 0x00000257
DSTS = 0x00833f98
DGCMDPAR = 0x00000000
DGCMD = 0x00000000
DALEPENA = 0x00000027
DEPCMDPAR2(0) = 0x00000000
DEPCMDPAR1(0) = 0x06a41000
DEPCMDPAR0(0) = 0x00000000
DEPCMD(0) = 0x00000006
DEPCMDPAR2(1) = 0x00000000
DEPCMDPAR1(1) = 0x06a41000
DEPCMDPAR0(1) = 0x00000000
DEPCMD(1) = 0x00010006
DEPCMDPAR2(2) = 0x00000000
DEPCMDPAR1(2) = 0x00000000
DEPCMDPAR0(2) = 0x00000000
DEPCMD(2) = 0x00020007
DEPCMDPAR2(3) = 0x00000000
DEPCMDPAR1(3) = 0x00000000
DEPCMDPAR0(3) = 0x00000000
DEPCMD(3) = 0x00000000
DEPCMDPAR2(4) = 0x00000000
DEPCMDPAR1(4) = 0x00000000
DEPCMDPAR0(4) = 0x00000001
DEPCMD(4) = 0x00040002
DEPCMDPAR2(5) = 0x00000000
DEPCMDPAR1(5) = 0x00000000
DEPCMDPAR0(5) = 0x00000000
DEPCMD(5) = 0x00040007
DEPCMDPAR2(6) = 0x00000000
DEPCMDPAR1(6) = 0x00000000
DEPCMDPAR0(6) = 0x00000001
DEPCMD(6) = 0x00060002
DEPCMDPAR2(7) = 0x00000000
DEPCMDPAR1(7) = 0x00000000
DEPCMDPAR0(7) = 0x00000001
DEPCMD(7) = 0x00070002
DEPCMDPAR2(8) = 0x00000000
DEPCMDPAR1(8) = 0x00000000
DEPCMDPAR0(8) = 0x00000001
DEPCMD(8) = 0x00080002
DEPCMDPAR2(9) = 0x00000000
DEPCMDPAR1(9) = 0x00000000
DEPCMDPAR0(9) = 0x00000001
DEPCMD(9) = 0x00090002
DEPCMDPAR2(10) = 0x00000000
DEPCMDPAR1(10) = 0x00000000
DEPCMDPAR0(10) = 0x00000001
DEPCMD(10) = 0x000a0002
DEPCMDPAR2(11) = 0x00000000
DEPCMDPAR1(11) = 0x00000000
DEPCMDPAR0(11) = 0x00000001
DEPCMD(11) = 0x000b0002
DEPCMDPAR2(12) = 0x00000000
DEPCMDPAR1(12) = 0x00000000
DEPCMDPAR0(12) = 0x00000001
DEPCMD(12) = 0x000c0002
DEPCMDPAR2(13) = 0x00000000
DEPCMDPAR1(13) = 0x00000000
DEPCMDPAR0(13) = 0x00000001
DEPCMD(13) = 0x000d0002
DEPCMDPAR2(14) = 0x00000000
DEPCMDPAR1(14) = 0x00000000
DEPCMDPAR0(14) = 0x00000001
DEPCMD(14) = 0x000e0002
DEPCMDPAR2(15) = 0x00000000
DEPCMDPAR1(15) = 0x00000000
DEPCMDPAR0(15) = 0x00000001
DEPCMD(15) = 0x000f0002
DEPCMDPAR2(16) = 0x00000000
DEPCMDPAR1(16) = 0x00000000
DEPCMDPAR0(16) = 0x00000000
DEPCMD(16) = 0x00000000
DEPCMDPAR2(17) = 0x00000000
DEPCMDPAR1(17) = 0x00000000
DEPCMDPAR0(17) = 0x00000000
DEPCMD(17) = 0x00000000
DEPCMDPAR2(18) = 0x00000000
DEPCMDPAR1(18) = 0x00000000
DEPCMDPAR0(18) = 0x00000001
DEPCMD(18) = 0x00100002
DEPCMDPAR2(19) = 0x00000000
DEPCMDPAR1(19) = 0x00000000
DEPCMDPAR0(19) = 0x00000001
DEPCMD(19) = 0x00110002
DEPCMDPAR2(20) = 0x00000000
DEPCMDPAR1(20) = 0x00000000
DEPCMDPAR0(20) = 0x00000001
DEPCMD(20) = 0x00120002
DEPCMDPAR2(21) = 0x00000000
DEPCMDPAR1(21) = 0x00000000
DEPCMDPAR0(21) = 0x00000001
DEPCMD(21) = 0x00130002
DEPCMDPAR2(22) = 0x00000000
DEPCMDPAR1(22) = 0x00000000
DEPCMDPAR0(22) = 0x00000001
DEPCMD(22) = 0x00140002
DEPCMDPAR2(23) = 0x00000000
DEPCMDPAR1(23) = 0x00000000
DEPCMDPAR0(23) = 0x00000001
DEPCMD(23) = 0x00150002
DEPCMDPAR2(24) = 0x00000000
DEPCMDPAR1(24) = 0x00000000
DEPCMDPAR0(24) = 0x00000001
DEPCMD(24) = 0x00160002
DEPCMDPAR2(25) = 0x00000000
DEPCMDPAR1(25) = 0x00000000
DEPCMDPAR0(25) = 0x00000001
DEPCMD(25) = 0x00170002
DEPCMDPAR2(26) = 0x00000000
DEPCMDPAR1(26) = 0x00000000
DEPCMDPAR0(26) = 0x00000001
DEPCMD(26) = 0x00180002
DEPCMDPAR2(27) = 0x00000000
DEPCMDPAR1(27) = 0x00000000
DEPCMDPAR0(27) = 0x00000001
DEPCMD(27) = 0x00190002
DEPCMDPAR2(28) = 0x00000000
DEPCMDPAR1(28) = 0x00000000
DEPCMDPAR0(28) = 0x00000001
DEPCMD(28) = 0x001a0002
DEPCMDPAR2(29) = 0x00000000
DEPCMDPAR1(29) = 0x00000000
DEPCMDPAR0(29) = 0x00000001
DEPCMD(29) = 0x001b0002
DEPCMDPAR2(30) = 0x00000000
DEPCMDPAR1(30) = 0x00000000
DEPCMDPAR0(30) = 0x00000001
DEPCMD(30) = 0x001c0002
DEPCMDPAR2(31) = 0x00000000
DEPCMDPAR1(31) = 0x00000000
DEPCMDPAR0(31) = 0x00000001
DEPCMD(31) = 0x001d0002
OCFG = 0x00000000
OCTL = 0x00000040
OEVT = 0x80000000
OEVTEN = 0x00000000
OSTS = 0x0000201f

> Thanks
> Wesley Cheng
>
>>> Was going to ask for the same to confirm the TXFIFO sizes from your
>>> results below :).
>>>
>>>> Otherwise, I'm hoping Andy will know?
>>>>
>>>>> Also, is there a way to use just a mass storage only composition?
>>>>> Based
>>>>> on the above observation, that probably means that the mass storage
>>>>> interface wasn't resized at all, because the configuration took up a
>>>>> lot
>>>>> of the internal FIFO space.
>>>> Sure, it's configured through configfs. With only mass_storage I have:
>>>>
>>>> With V10 on top v5.13.0-rc5:
>>>>
>>>> R/W speed = 41,6/39,3MB/s, access time 0.33ms
>>>>
>>>> With no patches on top v5.12.0:
>>>>
>>>> R/W speed = 41,1/38,7MB/s, access time 0.38ms
>>>>
>>> Thanks Ferry!  Could you collect the regdump, so I can confirm the two
>>> things mentioned?
>>>
>>> Thanks
>>> Wesley Cheng
>>>
>>>>> Thanks
>>>>> Wesley Cheng
>>>>>
>>>>>> kernel: using random self ethernet address
>>>>>> kernel: using random host ethernet address
>>>>>> kernel: Mass Storage Function, version: 2009/09/11
>>>>>> kernel: LUN: removable file: (no medium)
>>>>>> kernel: usb0: HOST MAC aa:bb:cc:dd:ee:f2
>>>>>> kernel: usb0: MAC aa:bb:cc:dd:ee:f1
>>>>>> kernel: IPv6: ADDRCONF(NETDEV_CHANGE): usb0: link becomes ready
>>>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>>>> depth:115540359
>>>>>> kernel: ------------[ cut here ]------------
>>>>>> kernel: WARNING: CPU: 0 PID: 594 at drivers/usb/gadget/udc/core.c:278
>>>>>> usb_ep_queue+0x75/0x80
>>>>>> kernel: Modules linked in: usb_f_uac2 u_audio usb_f_mass_storage
>>>>>> usb_f_eem u_ether usb_f_serial u_serial libcomposite rfcomm
>>>>>> iptable_nat
>>>>>> bnep snd_sof_nocodec spi_pxa2>
>>>>>> kernel: CPU: 0 PID: 594 Comm: irq/14-dwc3 Not tainted
>>>>>> 5.13.0-rc5-edison-acpi-standard #1
>>>>>> kernel: Hardware name: Intel Corporation Merrifield/BODEGA BAY,
>>>>>> BIOS 542
>>>>>> 2015.01.21:18.19.48
>>>>>> kernel: RIP: 0010:usb_ep_queue+0x75/0x80
>>>>>> kernel: Code: 01 73 e4 48 8b 05 fb 63 06 01 48 85 c0 74 12 48 8b 78 08
>>>>>> 44 89 e9 4c 89 e2 48 89 ee e8 74 05 00 00 44 89 e8 5d 41 5c 41 5d c3
>>>>>> <0f> 0b 41 bd 94 ff ff ff >
>>>>>> kernel: RSP: 0000:ffff91eec083fc98 EFLAGS: 00010082
>>>>>> kernel: RAX: ffff8af20357d960 RBX: 0000000000000000 RCX:
>>>>>> ffff8af202f06400
>>>>>> kernel: RDX: 0000000000000a20 RSI: ffff8af208785780 RDI:
>>>>>> ffff8af202e9ae00
>>>>>> kernel: RBP: ffff8af202e9ae00 R08: 00000000000000c0 R09:
>>>>>> ffff8af208785780
>>>>>> kernel: R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
>>>>>> ffff8af208785780
>>>>>> kernel: R13: 0000000000000000 R14: ffff8af202e9ae00 R15:
>>>>>> ffff8af203e26cc0
>>>>>> kernel: FS:  0000000000000000(0000) GS:ffff8af23e200000(0000)
>>>>>> knlGS:0000000000000000
>>>>>> kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>> kernel: CR2: 000055e2c21f2100 CR3: 0000000003b38000 CR4:
>>>>>> 00000000001006f0
>>>>>> kernel: Call Trace:
>>>>>> kernel:  u_audio_start_playback+0x107/0x1a0 [u_audio]
>>>>>> kernel:  composite_setup+0x224/0x1ba0 [libcomposite]
>>>>>> kernel:  ? dwc3_gadget_ep_queue+0xf6/0x1a0
>>>>>> kernel:  ? usb_ep_queue+0x2a/0x80
>>>>>> kernel:  ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>> kernel:  configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>> kernel:  dwc3_ep0_interrupt+0x469/0xa80
>>>>>> kernel:  dwc3_thread_interrupt+0x8ee/0xf40
>>>>>> kernel:  ? __wake_up_common_lock+0x85/0xb0
>>>>>> kernel:  ? disable_irq_nosync+0x10/0x10
>>>>>> kernel:  irq_thread_fn+0x1b/0x60
>>>>>> kernel:  irq_thread+0xd6/0x170
>>>>>> kernel:  ? irq_thread_check_affinity+0x70/0x70
>>>>>> kernel:  ? irq_forced_thread_fn+0x70/0x70
>>>>>> kernel:  kthread+0x116/0x130
>>>>>> kernel:  ? kthread_create_worker_on_cpu+0x60/0x60
>>>>>> kernel:  ret_from_fork+0x22/0x30
>>>>>> kernel: ---[ end trace e5b9e28058c53584 ]---
>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>> kernel: dwc3 dwc3.0.auto: request 000000003c32dcc5 was not queued to
>>>>>> ep5in
>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to
>>>>>> ep5in
>>>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>>>> depth:115540359
>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to
>>>>>> ep5in
>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000036ac129 was not queued to
>>>>>> ep5in
>>>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>>>> depth:115540359
>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000ad1b8c18 was not queued to
>>>>>> ep5in
>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000fbc71244 was not queued to
>>>>>> ep5in
>>>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>>>> depth:115540359
>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000fbc71244 was not queued to
>>>>>> ep5in
>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000ad1b8c18 was not queued to
>>>>>> ep5in
>>>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>>>> depth:115540359
>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>> kernel: dwc3 dwc3.0.auto: request 000000003c32dcc5 was not queued to
>>>>>> ep5in
>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to
>>>>>> ep5in
>>>>>>
>>>>>> Removing uac2 from the config makes the call trace go away, but the
>>>>>> R/W
>>>>>> speed does not change.
>>>>>>
>>>>>>> :), not a problem...I've been working on getting the ideal set up for
>>>>>>> the performance profiling for awhile, so anything I can do to make
>>>>>>> sure
>>>>>>> we get some good results.
>>>>>>>
>>>>>>>> I'll try with a ramdisk and let you know.
>>>>>>>>
>>>>>>> Thanks again for the testing, Ferry.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Wesley Cheng
>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Wesley Cheng
>>>>>>>>>
>>>>>>>>>>> Changes in V9:
>>>>>>>>>>>       - Fixed incorrect patch in series.  Removed changes in
>>>>>>>>>>> DTSI, as
>>>>>>>>>>> dwc3-qcom will
>>>>>>>>>>>         add the property by default from the kernel.
>>>>>>>>>>>
>>>>>>>>>>> Changes in V8:
>>>>>>>>>>>       - Rebased to usb-testing
>>>>>>>>>>>       - Using devm_kzalloc for adding txfifo property in dwc3-qcom
>>>>>>>>>>>       - Removed DWC3 QCOM ACPI property for enabling the txfifo
>>>>>>>>>>> resize
>>>>>>>>>>>
>>>>>>>>>>> Changes in V7:
>>>>>>>>>>>       - Added a new property tx-fifo-max-num for limiting how much
>>>>>>>>>>> fifo
>>>>>>>>>>> space the
>>>>>>>>>>>         resizing logic can allocate for endpoints with large burst
>>>>>>>>>>> values.  This
>>>>>>>>>>>         can differ across platforms, and tie in closely with
>>>>>>>>>>> overall
>>>>>>>>>>> system latency.
>>>>>>>>>>>       - Added recommended checks for DWC32.
>>>>>>>>>>>       - Added changes to set the tx-fifo-resize property from
>>>>>>>>>>> dwc3-qcom by
>>>>>>>>>>> default
>>>>>>>>>>>         instead of modifying the current DTSI files.
>>>>>>>>>>>       - Added comments on all APIs/variables introduced.
>>>>>>>>>>>       - Updated the DWC3 YAML to include a better description
>>>>>>>>>>> of the
>>>>>>>>>>> tx-fifo-resize
>>>>>>>>>>>         property and added an entry for tx-fifo-max-num.
>>>>>>>>>>>
>>>>>>>>>>> Changes in V6:
>>>>>>>>>>>       - Rebased patches to usb-testing.
>>>>>>>>>>>       - Renamed to PATCH series instead of RFC.
>>>>>>>>>>>       - Checking for fs_descriptors instead of ss_descriptors for
>>>>>>>>>>> determining the
>>>>>>>>>>>         endpoint count for a particular configuration.
>>>>>>>>>>>       - Re-ordered patch series to fix patch dependencies.
>>>>>>>>>>>
>>>>>>>>>>> Changes in V5:
>>>>>>>>>>>       - Added check_config() logic, which is used to
>>>>>>>>>>> communicate the
>>>>>>>>>>> number of EPs
>>>>>>>>>>>         used in a particular configuration.  Based on this, the
>>>>>>>>>>> DWC3
>>>>>>>>>>> gadget driver
>>>>>>>>>>>         has the ability to know the maximum number of eps
>>>>>>>>>>> utilized in
>>>>>>>>>>> all
>>>>>>>>>>> configs.
>>>>>>>>>>>         This helps reduce unnecessary allocation to unused eps,
>>>>>>>>>>> and will
>>>>>>>>>>> catch fifo
>>>>>>>>>>>         allocation issues at bind() time.
>>>>>>>>>>>       - Fixed variable declaration to single line per variable,
>>>>>>>>>>> and
>>>>>>>>>>> reverse xmas.
>>>>>>>>>>>       - Created a helper for fifo clearing, which is used by ep0.c
>>>>>>>>>>>
>>>>>>>>>>> Changes in V4:
>>>>>>>>>>>       - Removed struct dwc3* as an argument for
>>>>>>>>>>> dwc3_gadget_resize_tx_fifos()
>>>>>>>>>>>       - Removed WARN_ON(1) in case we run out of fifo space
>>>>>>>>>>>       Changes in V3:
>>>>>>>>>>>       - Removed "Reviewed-by" tags
>>>>>>>>>>>       - Renamed series back to RFC
>>>>>>>>>>>       - Modified logic to ensure that fifo_size is reset if we
>>>>>>>>>>> pass the
>>>>>>>>>>> minimum
>>>>>>>>>>>         threshold.  Tested with binding multiple FDs requesting 6
>>>>>>>>>>> FIFOs.
>>>>>>>>>>>
>>>>>>>>>>> Changes in V2:
>>>>>>>>>>>       - Modified TXFIFO resizing logic to ensure that each EP is
>>>>>>>>>>> reserved a
>>>>>>>>>>>         FIFO.
>>>>>>>>>>>       - Removed dev_dbg() prints and fixed typos from patches
>>>>>>>>>>>       - Added some more description on the dt-bindings commit
>>>>>>>>>>> message
>>>>>>>>>>>
>>>>>>>>>>> Currently, there is no functionality to allow for resizing the
>>>>>>>>>>> TXFIFOs, and
>>>>>>>>>>> relying on the HW default setting for the TXFIFO depth.  In most
>>>>>>>>>>> cases, the
>>>>>>>>>>> HW default is probably sufficient, but for USB compositions that
>>>>>>>>>>> contain
>>>>>>>>>>> multiple functions that require EP bursting, the default settings
>>>>>>>>>>> might not be enough.  Also to note, the current SW will assign an
>>>>>>>>>>> EP to a
>>>>>>>>>>> function driver w/o checking to see if the TXFIFO size for that
>>>>>>>>>>> particular
>>>>>>>>>>> EP is large enough. (this is a problem if there are multiple HW
>>>>>>>>>>> defined
>>>>>>>>>>> values for the TXFIFO size)
>>>>>>>>>>>
>>>>>>>>>>> It is mentioned in the SNPS databook that a minimum of TX FIFO
>>>>>>>>>>> depth = 3
>>>>>>>>>>> is required for an EP that supports bursting.  Otherwise, there
>>>>>>>>>>> may be
>>>>>>>>>>> frequent occurences of bursts ending.  For high bandwidth
>>>>>>>>>>> functions,
>>>>>>>>>>> such as data tethering (protocols that support data aggregation),
>>>>>>>>>>> mass
>>>>>>>>>>> storage, and media transfer protocol (over FFS), the bMaxBurst
>>>>>>>>>>> value
>>>>>>>>>>> can be
>>>>>>>>>>> large, and a bigger TXFIFO depth may prove to be beneficial in
>>>>>>>>>>> terms
>>>>>>>>>>> of USB
>>>>>>>>>>> throughput. (which can be associated to system access latency,
>>>>>>>>>>> etc...)  It
>>>>>>>>>>> allows for a more consistent burst of traffic, w/o any
>>>>>>>>>>> interruptions, as
>>>>>>>>>>> data is readily available in the FIFO.
>>>>>>>>>>>
>>>>>>>>>>> With testing done using the mass storage function driver, the
>>>>>>>>>>> results
>>>>>>>>>>> show
>>>>>>>>>>> that with a larger TXFIFO depth, the bandwidth increased
>>>>>>>>>>> significantly.
>>>>>>>>>>>
>>>>>>>>>>> Test Parameters:
>>>>>>>>>>>       - Platform: Qualcomm SM8150
>>>>>>>>>>>       - bMaxBurst = 6
>>>>>>>>>>>       - USB req size = 256kB
>>>>>>>>>>>       - Num of USB reqs = 16
>>>>>>>>>>>       - USB Speed = Super-Speed
>>>>>>>>>>>       - Function Driver: Mass Storage (w/ ramdisk)
>>>>>>>>>>>       - Test Application: CrystalDiskMark
>>>>>>>>>>>
>>>>>>>>>>> Results:
>>>>>>>>>>>
>>>>>>>>>>> TXFIFO Depth = 3 max packets
>>>>>>>>>>>
>>>>>>>>>>> Test Case | Data Size | AVG tput (in MB/s)
>>>>>>>>>>> -------------------------------------------
>>>>>>>>>>> Sequential|1 GB x     |
>>>>>>>>>>> Read      |9 loops    | 193.60
>>>>>>>>>>>           |           | 195.86
>>>>>>>>>>>                |           | 184.77
>>>>>>>>>>>                |           | 193.60
>>>>>>>>>>> -------------------------------------------
>>>>>>>>>>>
>>>>>>>>>>> TXFIFO Depth = 6 max packets
>>>>>>>>>>>
>>>>>>>>>>> Test Case | Data Size | AVG tput (in MB/s)
>>>>>>>>>>> -------------------------------------------
>>>>>>>>>>> Sequential|1 GB x     |
>>>>>>>>>>> Read      |9 loops    | 287.35
>>>>>>>>>>>           |           | 304.94
>>>>>>>>>>>                |           | 289.64
>>>>>>>>>>>                |           | 293.61
>>>>>>>>>>> -------------------------------------------
>>>>>>>>>>>
>>>>>>>>>>> Wesley Cheng (6):
>>>>>>>>>>>        usb: gadget: udc: core: Introduce check_config to verify
>>>>>>>>>>> USB
>>>>>>>>>>>          configuration
>>>>>>>>>>>        usb: gadget: configfs: Check USB configuration before
>>>>>>>>>>> adding
>>>>>>>>>>>        usb: dwc3: Resize TX FIFOs to meet EP bursting requirements
>>>>>>>>>>>        of: Add stub for of_add_property()
>>>>>>>>>>>        usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by
>>>>>>>>>>> default
>>>>>>>>>>>        dt-bindings: usb: dwc3: Update dwc3 TX fifo properties
>>>>>>>>>>>
>>>>>>>>>>>       .../devicetree/bindings/usb/snps,dwc3.yaml         |  15 +-
>>>>>>>>>>>       drivers/usb/dwc3/core.c                            |   9 +
>>>>>>>>>>>       drivers/usb/dwc3/core.h                            |  15 ++
>>>>>>>>>>>       drivers/usb/dwc3/dwc3-qcom.c                       |   9 +
>>>>>>>>>>>       drivers/usb/dwc3/ep0.c                             |   2 +
>>>>>>>>>>>       drivers/usb/dwc3/gadget.c                          | 212
>>>>>>>>>>> +++++++++++++++++++++
>>>>>>>>>>>       drivers/usb/gadget/configfs.c                      |  22 +++
>>>>>>>>>>>       drivers/usb/gadget/udc/core.c                      |  25 +++
>>>>>>>>>>>       include/linux/of.h                                 |   5 +
>>>>>>>>>>>       include/linux/usb/gadget.h                         |   5 +
>>>>>>>>>>>       10 files changed, 317 insertions(+), 2 deletions(-)
>>>>>>>>>>>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-06-25 21:10                     ` Ferry Toth
@ 2021-06-25 21:19                       ` Wesley Cheng
  2021-06-25 22:31                         ` Ferry Toth
  0 siblings, 1 reply; 73+ messages in thread
From: Wesley Cheng @ 2021-06-25 21:19 UTC (permalink / raw)
  To: Ferry Toth, balbi, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list
  Cc: linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	heikki.krogerus, andy.shevchenko



On 6/25/2021 2:10 PM, Ferry Toth wrote:
> Hi
> 
> Op 23-06-2021 om 09:50 schreef Wesley Cheng:
>>
>> On 6/22/2021 3:12 PM, Ferry Toth wrote:
>>> Hi
>>>
>>> Op 23-06-2021 om 00:00 schreef Wesley Cheng:
>>>> On 6/22/2021 1:09 PM, Ferry Toth wrote:
>>>>> Hi
>>>>>
>>>>> Op 22-06-2021 om 20:38 schreef Wesley Cheng:
>>>>>> On 6/19/2021 5:40 AM, Ferry Toth wrote:
>>>>>>> Hi
>>>>>>>
>>>>>>> Op 18-06-2021 om 00:25 schreef Wesley Cheng:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On 6/17/2021 2:55 PM, Ferry Toth wrote:
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> Op 17-06-2021 om 23:48 schreef Wesley Cheng:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> On 6/17/2021 2:01 PM, Ferry Toth wrote:
>>>>>>>>>>> Hi
>>>>>>>>>>>
>>>>>>>>>>> Op 17-06-2021 om 11:58 schreef Wesley Cheng:
>>>>>>>>>>>> Changes in V10:
>>>>>>>>>>>>       - Fixed compilation errors in config where OF is not used
>>>>>>>>>>>> (error due to
>>>>>>>>>>>>         unknown symbol for of_add_property()).  Add
>>>>>>>>>>>> of_add_property()
>>>>>>>>>>>> stub.
>>>>>>>>>>>>       - Fixed compilation warning for incorrect argument being
>>>>>>>>>>>> passed to
>>>>>>>>>>>> dwc3_mdwidth
>>>>>>>>>>> This fixes the OOPS I had in V9. I do not see any change in
>>>>>>>>>>> performance
>>>>>>>>>>> on Merrifield though.
>>>>>>>>>> I see...thanks Ferry! With your testing, are you writing to the
>>>>>>>>>> device's
>>>>>>>>>> internal storage (ie UFS, eMMC, etc...), or did you use a
>>>>>>>>>> ramdisk as
>>>>>>>>>> well?
>>>>>>>>> In this case I just tested the EEM path using iperf3.
>>>>>>>>>
>>>>>>>> Got it.  I don't believe f_eem will use a high enough (if at all)
>>>>>>>> bMaxBurst value to change the TXFIFO size.
>>>>>>>>
>>>>>>>>>> If not with a ramdisk, we might want to give that a try to avoid
>>>>>>>>>> the
>>>>>>>>>> storage path being the bottleneck.  You can use "dd" to create an
>>>>>>>>>> empty
>>>>>>>>>> file, and then just use that as the LUN's backing file.
>>>>>>>>>>
>>>>>>>>>> echo ramdisk.img >
>>>>>>>>>> /sys/kernel/config/usb_gadget/g1/functions/mass_storage.0/lun.0/file
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Ah, why didn't I think of that. I have currently mass storage
>>>>>>>>> setup
>>>>>>>>> with
>>>>>>>>> eMMC but it seems that is indeed the bottleneck.
>>>>>>>>>
>>>>>>> I created a 64MB disk following the instructions here
>>>>>>> http://www.linux-usb.org/gadget/file_storage.html (that seems a
>>>>>>> little
>>>>>>> outdated, at least I can not start the first partition at sector 8,
>>>>>>> but
>>>>>>> minimum 2048), and added a test file on it.
>>>>>>>
>>>>>>> I then copy the file to /dev/shm prior to setting configfs
>>>>>>> (composite
>>>>>>> device gser/eem/mass_storage/uac2).
>>>>>>>
>>>>>>> journal shows:
>>>>>>>
>>>>>>> kernel: Mass Storage Function, version: 2009/09/11
>>>>>>> kernel: LUN: removable file: (no medium)
>>>>>>>
>>>>>>> I don't know what that means, because I see the test file on the
>>>>>>> ramdisk.
>>>>>>>
>>>>>>> Then I again used gnome disks to benchmark (read/write 10MB):
>>>>>>>
>>>>>>> With V10 on top v5.13.0-rc5:
>>>>>>>
>>>>>>> R/W speed = 35.6/35.8MB/s, access time 0.35ms
>>>>>>>
>>>>>>> With no patches on top v5.12.0:
>>>>>>>
>>>>>>> R/W speed = 35.7/36.1MB/s, access time 0.35ms
>>>>>> Hi Ferry,
>>>>>>
>>>>>>> I see no speed difference (and it's about the same as with the eMMC
>>>>>>> backed disk). But the patches are causing a new call trace
>>>>>>>
>>>>>> Would you happen to know what DWC3 controller revision the device is
>>>>>> using?  The callstack print occurs, because it looks like it ran
>>>>>> out of
>>>>>> internal memory, although there should be logic present for making
>>>>>> sure
>>>>>> that at least there is enough room for 1 FIFO per endpoint. 
>>>>>> (possibly
>>>>>> the logic/math depends on the controller revision)
>>>> Hi Ferry,
>>>>
>>>>> Do you know where I could find that in a file on the device?
>>>>>
>>>> Maybe you can just dump the DWC3 registers?
>>>> cat /sys/kernel/debug/usb/<controller name>/regdump
>>> This isin host mode:
>>>
>>> ~# cat /sys/kernel/debug/usb/dwc3.0.auto/regdump
>>> GSBUSCFG0 = 0x00000006
>>> GSBUSCFG1 = 0x00000f00
>>> GTXTHRCFG = 0x230a0000
>>> GRXTHRCFG = 0x22800000
>>> GCTL = 0x45801002
>>> GEVTEN = 0x00000000
>>> GSTS = 0x3e800001
>>> GUCTL1 = 0x00000000
>>> GSNPSID = 0x5533250a
>>> GGPIO = 0x00000000
>>> GUID = 0x00050d00
>>> GUCTL = 0x0200ce00
>>> GBUSERRADDR0 = 0x00000000
>>> GBUSERRADDR1 = 0x00000000
>> ...
>> Hi Ferry,
>>
>> Great!  Thanks, that got me the information regarding the DWC3 revision
>> you're using. (its version 2.50a)  I checked the DWC3 databook and looks
>> like certain versions have some different math to calculate the TXFIFO
>> size.  I have taken this into account and factored that in in the next
>> revision.
>>
>> Is there a way you could collect the regdump in device mode w/ mass
>> storage only composition after enumerating with the PC?  That would help
>> confirm how the TXFIFO resizing logic is working on your platform.
> 
> Sorry for the wait. Yes I have serial console. So here is in device
> mode, mass_storage only, enumerated but not mounted:
> 
> GSBUSCFG0 = 0x00000006
> GSBUSCFG1 = 0x00000f00
> GTXTHRCFG = 0x230a0000
> GRXTHRCFG = 0x02800000
> GCTL = 0x45802002
> GEVTEN = 0x00000000
...
> 

Hi Ferry,

Thanks!  So does your board support SSUSB by any chance?  Based on the
DSTS output, it seems that its currently operating in HSUSB mode.  I
guess we'd expect higher tput values as well in general if we are in
SSUSB mode.

The resize logic would only happen if the device was in SSUSB, as SSUSB
has endpoint bursting, which is the main incentive for the fifo resize.

Thanks
Wesley Cheng

>> Thanks
>> Wesley Cheng
>>
>>>> Was going to ask for the same to confirm the TXFIFO sizes from your
>>>> results below :).
>>>>
>>>>> Otherwise, I'm hoping Andy will know?
>>>>>
>>>>>> Also, is there a way to use just a mass storage only composition?
>>>>>> Based
>>>>>> on the above observation, that probably means that the mass storage
>>>>>> interface wasn't resized at all, because the configuration took up a
>>>>>> lot
>>>>>> of the internal FIFO space.
>>>>> Sure, it's configured through configfs. With only mass_storage I have:
>>>>>
>>>>> With V10 on top v5.13.0-rc5:
>>>>>
>>>>> R/W speed = 41,6/39,3MB/s, access time 0.33ms
>>>>>
>>>>> With no patches on top v5.12.0:
>>>>>
>>>>> R/W speed = 41,1/38,7MB/s, access time 0.38ms
>>>>>
>>>> Thanks Ferry!  Could you collect the regdump, so I can confirm the two
>>>> things mentioned?
>>>>
>>>> Thanks
>>>> Wesley Cheng
>>>>
>>>>>> Thanks
>>>>>> Wesley Cheng
>>>>>>
>>>>>>> kernel: using random self ethernet address
>>>>>>> kernel: using random host ethernet address
>>>>>>> kernel: Mass Storage Function, version: 2009/09/11
>>>>>>> kernel: LUN: removable file: (no medium)
>>>>>>> kernel: usb0: HOST MAC aa:bb:cc:dd:ee:f2
>>>>>>> kernel: usb0: MAC aa:bb:cc:dd:ee:f1
>>>>>>> kernel: IPv6: ADDRCONF(NETDEV_CHANGE): usb0: link becomes ready
>>>>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>>>>> depth:115540359
>>>>>>> kernel: ------------[ cut here ]------------
>>>>>>> kernel: WARNING: CPU: 0 PID: 594 at
>>>>>>> drivers/usb/gadget/udc/core.c:278
>>>>>>> usb_ep_queue+0x75/0x80
>>>>>>> kernel: Modules linked in: usb_f_uac2 u_audio usb_f_mass_storage
>>>>>>> usb_f_eem u_ether usb_f_serial u_serial libcomposite rfcomm
>>>>>>> iptable_nat
>>>>>>> bnep snd_sof_nocodec spi_pxa2>
>>>>>>> kernel: CPU: 0 PID: 594 Comm: irq/14-dwc3 Not tainted
>>>>>>> 5.13.0-rc5-edison-acpi-standard #1
>>>>>>> kernel: Hardware name: Intel Corporation Merrifield/BODEGA BAY,
>>>>>>> BIOS 542
>>>>>>> 2015.01.21:18.19.48
>>>>>>> kernel: RIP: 0010:usb_ep_queue+0x75/0x80
>>>>>>> kernel: Code: 01 73 e4 48 8b 05 fb 63 06 01 48 85 c0 74 12 48 8b
>>>>>>> 78 08
>>>>>>> 44 89 e9 4c 89 e2 48 89 ee e8 74 05 00 00 44 89 e8 5d 41 5c 41 5d c3
>>>>>>> <0f> 0b 41 bd 94 ff ff ff >
>>>>>>> kernel: RSP: 0000:ffff91eec083fc98 EFLAGS: 00010082
>>>>>>> kernel: RAX: ffff8af20357d960 RBX: 0000000000000000 RCX:
>>>>>>> ffff8af202f06400
>>>>>>> kernel: RDX: 0000000000000a20 RSI: ffff8af208785780 RDI:
>>>>>>> ffff8af202e9ae00
>>>>>>> kernel: RBP: ffff8af202e9ae00 R08: 00000000000000c0 R09:
>>>>>>> ffff8af208785780
>>>>>>> kernel: R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
>>>>>>> ffff8af208785780
>>>>>>> kernel: R13: 0000000000000000 R14: ffff8af202e9ae00 R15:
>>>>>>> ffff8af203e26cc0
>>>>>>> kernel: FS:  0000000000000000(0000) GS:ffff8af23e200000(0000)
>>>>>>> knlGS:0000000000000000
>>>>>>> kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>>> kernel: CR2: 000055e2c21f2100 CR3: 0000000003b38000 CR4:
>>>>>>> 00000000001006f0
>>>>>>> kernel: Call Trace:
>>>>>>> kernel:  u_audio_start_playback+0x107/0x1a0 [u_audio]
>>>>>>> kernel:  composite_setup+0x224/0x1ba0 [libcomposite]
>>>>>>> kernel:  ? dwc3_gadget_ep_queue+0xf6/0x1a0
>>>>>>> kernel:  ? usb_ep_queue+0x2a/0x80
>>>>>>> kernel:  ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>> kernel:  configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>> kernel:  dwc3_ep0_interrupt+0x469/0xa80
>>>>>>> kernel:  dwc3_thread_interrupt+0x8ee/0xf40
>>>>>>> kernel:  ? __wake_up_common_lock+0x85/0xb0
>>>>>>> kernel:  ? disable_irq_nosync+0x10/0x10
>>>>>>> kernel:  irq_thread_fn+0x1b/0x60
>>>>>>> kernel:  irq_thread+0xd6/0x170
>>>>>>> kernel:  ? irq_thread_check_affinity+0x70/0x70
>>>>>>> kernel:  ? irq_forced_thread_fn+0x70/0x70
>>>>>>> kernel:  kthread+0x116/0x130
>>>>>>> kernel:  ? kthread_create_worker_on_cpu+0x60/0x60
>>>>>>> kernel:  ret_from_fork+0x22/0x30
>>>>>>> kernel: ---[ end trace e5b9e28058c53584 ]---
>>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>>> kernel: dwc3 dwc3.0.auto: request 000000003c32dcc5 was not queued to
>>>>>>> ep5in
>>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to
>>>>>>> ep5in
>>>>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>>>>> depth:115540359
>>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to
>>>>>>> ep5in
>>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000036ac129 was not queued to
>>>>>>> ep5in
>>>>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>>>>> depth:115540359
>>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000ad1b8c18 was not queued to
>>>>>>> ep5in
>>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000fbc71244 was not queued to
>>>>>>> ep5in
>>>>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>>>>> depth:115540359
>>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000fbc71244 was not queued to
>>>>>>> ep5in
>>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000ad1b8c18 was not queued to
>>>>>>> ep5in
>>>>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>>>>> depth:115540359
>>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>>> kernel: dwc3 dwc3.0.auto: request 000000003c32dcc5 was not queued to
>>>>>>> ep5in
>>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to
>>>>>>> ep5in
>>>>>>>
>>>>>>> Removing uac2 from the config makes the call trace go away, but the
>>>>>>> R/W
>>>>>>> speed does not change.
>>>>>>>
>>>>>>>> :), not a problem...I've been working on getting the ideal set
>>>>>>>> up for
>>>>>>>> the performance profiling for awhile, so anything I can do to make
>>>>>>>> sure
>>>>>>>> we get some good results.
>>>>>>>>
>>>>>>>>> I'll try with a ramdisk and let you know.
>>>>>>>>>
>>>>>>>> Thanks again for the testing, Ferry.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Wesley Cheng
>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Wesley Cheng
>>>>>>>>>>
>>>>>>>>>>>> Changes in V9:
>>>>>>>>>>>>       - Fixed incorrect patch in series.  Removed changes in
>>>>>>>>>>>> DTSI, as
>>>>>>>>>>>> dwc3-qcom will
>>>>>>>>>>>>         add the property by default from the kernel.
>>>>>>>>>>>>
>>>>>>>>>>>> Changes in V8:
>>>>>>>>>>>>       - Rebased to usb-testing
>>>>>>>>>>>>       - Using devm_kzalloc for adding txfifo property in
>>>>>>>>>>>> dwc3-qcom
>>>>>>>>>>>>       - Removed DWC3 QCOM ACPI property for enabling the txfifo
>>>>>>>>>>>> resize
>>>>>>>>>>>>
>>>>>>>>>>>> Changes in V7:
>>>>>>>>>>>>       - Added a new property tx-fifo-max-num for limiting
>>>>>>>>>>>> how much
>>>>>>>>>>>> fifo
>>>>>>>>>>>> space the
>>>>>>>>>>>>         resizing logic can allocate for endpoints with large
>>>>>>>>>>>> burst
>>>>>>>>>>>> values.  This
>>>>>>>>>>>>         can differ across platforms, and tie in closely with
>>>>>>>>>>>> overall
>>>>>>>>>>>> system latency.
>>>>>>>>>>>>       - Added recommended checks for DWC32.
>>>>>>>>>>>>       - Added changes to set the tx-fifo-resize property from
>>>>>>>>>>>> dwc3-qcom by
>>>>>>>>>>>> default
>>>>>>>>>>>>         instead of modifying the current DTSI files.
>>>>>>>>>>>>       - Added comments on all APIs/variables introduced.
>>>>>>>>>>>>       - Updated the DWC3 YAML to include a better description
>>>>>>>>>>>> of the
>>>>>>>>>>>> tx-fifo-resize
>>>>>>>>>>>>         property and added an entry for tx-fifo-max-num.
>>>>>>>>>>>>
>>>>>>>>>>>> Changes in V6:
>>>>>>>>>>>>       - Rebased patches to usb-testing.
>>>>>>>>>>>>       - Renamed to PATCH series instead of RFC.
>>>>>>>>>>>>       - Checking for fs_descriptors instead of
>>>>>>>>>>>> ss_descriptors for
>>>>>>>>>>>> determining the
>>>>>>>>>>>>         endpoint count for a particular configuration.
>>>>>>>>>>>>       - Re-ordered patch series to fix patch dependencies.
>>>>>>>>>>>>
>>>>>>>>>>>> Changes in V5:
>>>>>>>>>>>>       - Added check_config() logic, which is used to
>>>>>>>>>>>> communicate the
>>>>>>>>>>>> number of EPs
>>>>>>>>>>>>         used in a particular configuration.  Based on this, the
>>>>>>>>>>>> DWC3
>>>>>>>>>>>> gadget driver
>>>>>>>>>>>>         has the ability to know the maximum number of eps
>>>>>>>>>>>> utilized in
>>>>>>>>>>>> all
>>>>>>>>>>>> configs.
>>>>>>>>>>>>         This helps reduce unnecessary allocation to unused eps,
>>>>>>>>>>>> and will
>>>>>>>>>>>> catch fifo
>>>>>>>>>>>>         allocation issues at bind() time.
>>>>>>>>>>>>       - Fixed variable declaration to single line per variable,
>>>>>>>>>>>> and
>>>>>>>>>>>> reverse xmas.
>>>>>>>>>>>>       - Created a helper for fifo clearing, which is used by
>>>>>>>>>>>> ep0.c
>>>>>>>>>>>>
>>>>>>>>>>>> Changes in V4:
>>>>>>>>>>>>       - Removed struct dwc3* as an argument for
>>>>>>>>>>>> dwc3_gadget_resize_tx_fifos()
>>>>>>>>>>>>       - Removed WARN_ON(1) in case we run out of fifo space
>>>>>>>>>>>>       Changes in V3:
>>>>>>>>>>>>       - Removed "Reviewed-by" tags
>>>>>>>>>>>>       - Renamed series back to RFC
>>>>>>>>>>>>       - Modified logic to ensure that fifo_size is reset if we
>>>>>>>>>>>> pass the
>>>>>>>>>>>> minimum
>>>>>>>>>>>>         threshold.  Tested with binding multiple FDs
>>>>>>>>>>>> requesting 6
>>>>>>>>>>>> FIFOs.
>>>>>>>>>>>>
>>>>>>>>>>>> Changes in V2:
>>>>>>>>>>>>       - Modified TXFIFO resizing logic to ensure that each
>>>>>>>>>>>> EP is
>>>>>>>>>>>> reserved a
>>>>>>>>>>>>         FIFO.
>>>>>>>>>>>>       - Removed dev_dbg() prints and fixed typos from patches
>>>>>>>>>>>>       - Added some more description on the dt-bindings commit
>>>>>>>>>>>> message
>>>>>>>>>>>>
>>>>>>>>>>>> Currently, there is no functionality to allow for resizing the
>>>>>>>>>>>> TXFIFOs, and
>>>>>>>>>>>> relying on the HW default setting for the TXFIFO depth.  In
>>>>>>>>>>>> most
>>>>>>>>>>>> cases, the
>>>>>>>>>>>> HW default is probably sufficient, but for USB compositions
>>>>>>>>>>>> that
>>>>>>>>>>>> contain
>>>>>>>>>>>> multiple functions that require EP bursting, the default
>>>>>>>>>>>> settings
>>>>>>>>>>>> might not be enough.  Also to note, the current SW will
>>>>>>>>>>>> assign an
>>>>>>>>>>>> EP to a
>>>>>>>>>>>> function driver w/o checking to see if the TXFIFO size for that
>>>>>>>>>>>> particular
>>>>>>>>>>>> EP is large enough. (this is a problem if there are multiple HW
>>>>>>>>>>>> defined
>>>>>>>>>>>> values for the TXFIFO size)
>>>>>>>>>>>>
>>>>>>>>>>>> It is mentioned in the SNPS databook that a minimum of TX FIFO
>>>>>>>>>>>> depth = 3
>>>>>>>>>>>> is required for an EP that supports bursting.  Otherwise, there
>>>>>>>>>>>> may be
>>>>>>>>>>>> frequent occurences of bursts ending.  For high bandwidth
>>>>>>>>>>>> functions,
>>>>>>>>>>>> such as data tethering (protocols that support data
>>>>>>>>>>>> aggregation),
>>>>>>>>>>>> mass
>>>>>>>>>>>> storage, and media transfer protocol (over FFS), the bMaxBurst
>>>>>>>>>>>> value
>>>>>>>>>>>> can be
>>>>>>>>>>>> large, and a bigger TXFIFO depth may prove to be beneficial in
>>>>>>>>>>>> terms
>>>>>>>>>>>> of USB
>>>>>>>>>>>> throughput. (which can be associated to system access latency,
>>>>>>>>>>>> etc...)  It
>>>>>>>>>>>> allows for a more consistent burst of traffic, w/o any
>>>>>>>>>>>> interruptions, as
>>>>>>>>>>>> data is readily available in the FIFO.
>>>>>>>>>>>>
>>>>>>>>>>>> With testing done using the mass storage function driver, the
>>>>>>>>>>>> results
>>>>>>>>>>>> show
>>>>>>>>>>>> that with a larger TXFIFO depth, the bandwidth increased
>>>>>>>>>>>> significantly.
>>>>>>>>>>>>
>>>>>>>>>>>> Test Parameters:
>>>>>>>>>>>>       - Platform: Qualcomm SM8150
>>>>>>>>>>>>       - bMaxBurst = 6
>>>>>>>>>>>>       - USB req size = 256kB
>>>>>>>>>>>>       - Num of USB reqs = 16
>>>>>>>>>>>>       - USB Speed = Super-Speed
>>>>>>>>>>>>       - Function Driver: Mass Storage (w/ ramdisk)
>>>>>>>>>>>>       - Test Application: CrystalDiskMark
>>>>>>>>>>>>
>>>>>>>>>>>> Results:
>>>>>>>>>>>>
>>>>>>>>>>>> TXFIFO Depth = 3 max packets
>>>>>>>>>>>>
>>>>>>>>>>>> Test Case | Data Size | AVG tput (in MB/s)
>>>>>>>>>>>> -------------------------------------------
>>>>>>>>>>>> Sequential|1 GB x     |
>>>>>>>>>>>> Read      |9 loops    | 193.60
>>>>>>>>>>>>           |           | 195.86
>>>>>>>>>>>>                |           | 184.77
>>>>>>>>>>>>                |           | 193.60
>>>>>>>>>>>> -------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> TXFIFO Depth = 6 max packets
>>>>>>>>>>>>
>>>>>>>>>>>> Test Case | Data Size | AVG tput (in MB/s)
>>>>>>>>>>>> -------------------------------------------
>>>>>>>>>>>> Sequential|1 GB x     |
>>>>>>>>>>>> Read      |9 loops    | 287.35
>>>>>>>>>>>>           |           | 304.94
>>>>>>>>>>>>                |           | 289.64
>>>>>>>>>>>>                |           | 293.61
>>>>>>>>>>>> -------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> Wesley Cheng (6):
>>>>>>>>>>>>        usb: gadget: udc: core: Introduce check_config to verify
>>>>>>>>>>>> USB
>>>>>>>>>>>>          configuration
>>>>>>>>>>>>        usb: gadget: configfs: Check USB configuration before
>>>>>>>>>>>> adding
>>>>>>>>>>>>        usb: dwc3: Resize TX FIFOs to meet EP bursting
>>>>>>>>>>>> requirements
>>>>>>>>>>>>        of: Add stub for of_add_property()
>>>>>>>>>>>>        usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by
>>>>>>>>>>>> default
>>>>>>>>>>>>        dt-bindings: usb: dwc3: Update dwc3 TX fifo properties
>>>>>>>>>>>>
>>>>>>>>>>>>       .../devicetree/bindings/usb/snps,dwc3.yaml         | 
>>>>>>>>>>>> 15 +-
>>>>>>>>>>>>       drivers/usb/dwc3/core.c                            |  
>>>>>>>>>>>> 9 +
>>>>>>>>>>>>       drivers/usb/dwc3/core.h                            | 
>>>>>>>>>>>> 15 ++
>>>>>>>>>>>>       drivers/usb/dwc3/dwc3-qcom.c                       |  
>>>>>>>>>>>> 9 +
>>>>>>>>>>>>       drivers/usb/dwc3/ep0.c                             |  
>>>>>>>>>>>> 2 +
>>>>>>>>>>>>       drivers/usb/dwc3/gadget.c                          | 212
>>>>>>>>>>>> +++++++++++++++++++++
>>>>>>>>>>>>       drivers/usb/gadget/configfs.c                      | 
>>>>>>>>>>>> 22 +++
>>>>>>>>>>>>       drivers/usb/gadget/udc/core.c                      | 
>>>>>>>>>>>> 25 +++
>>>>>>>>>>>>       include/linux/of.h                                 |  
>>>>>>>>>>>> 5 +
>>>>>>>>>>>>       include/linux/usb/gadget.h                         |  
>>>>>>>>>>>> 5 +
>>>>>>>>>>>>       10 files changed, 317 insertions(+), 2 deletions(-)
>>>>>>>>>>>>

-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-06-25 21:19                       ` Wesley Cheng
@ 2021-06-25 22:31                         ` Ferry Toth
  0 siblings, 0 replies; 73+ messages in thread
From: Ferry Toth @ 2021-06-25 22:31 UTC (permalink / raw)
  To: Wesley Cheng, balbi, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list
  Cc: linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	heikki.krogerus, andy.shevchenko

Hi

Op 25-06-2021 om 23:19 schreef Wesley Cheng:
>
> On 6/25/2021 2:10 PM, Ferry Toth wrote:
>> Hi
>>
>> Op 23-06-2021 om 09:50 schreef Wesley Cheng:
>>> On 6/22/2021 3:12 PM, Ferry Toth wrote:
>>>> Hi
>>>>
>>>> Op 23-06-2021 om 00:00 schreef Wesley Cheng:
>>>>> On 6/22/2021 1:09 PM, Ferry Toth wrote:
>>>>>> Hi
>>>>>>
>>>>>> Op 22-06-2021 om 20:38 schreef Wesley Cheng:
>>>>>>> On 6/19/2021 5:40 AM, Ferry Toth wrote:
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> Op 18-06-2021 om 00:25 schreef Wesley Cheng:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On 6/17/2021 2:55 PM, Ferry Toth wrote:
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> Op 17-06-2021 om 23:48 schreef Wesley Cheng:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> On 6/17/2021 2:01 PM, Ferry Toth wrote:
>>>>>>>>>>>> Hi
>>>>>>>>>>>>
>>>>>>>>>>>> Op 17-06-2021 om 11:58 schreef Wesley Cheng:
>>>>>>>>>>>>> Changes in V10:
>>>>>>>>>>>>>        - Fixed compilation errors in config where OF is not used
>>>>>>>>>>>>> (error due to
>>>>>>>>>>>>>          unknown symbol for of_add_property()).  Add
>>>>>>>>>>>>> of_add_property()
>>>>>>>>>>>>> stub.
>>>>>>>>>>>>>        - Fixed compilation warning for incorrect argument being
>>>>>>>>>>>>> passed to
>>>>>>>>>>>>> dwc3_mdwidth
>>>>>>>>>>>> This fixes the OOPS I had in V9. I do not see any change in
>>>>>>>>>>>> performance
>>>>>>>>>>>> on Merrifield though.
>>>>>>>>>>> I see...thanks Ferry! With your testing, are you writing to the
>>>>>>>>>>> device's
>>>>>>>>>>> internal storage (ie UFS, eMMC, etc...), or did you use a
>>>>>>>>>>> ramdisk as
>>>>>>>>>>> well?
>>>>>>>>>> In this case I just tested the EEM path using iperf3.
>>>>>>>>>>
>>>>>>>>> Got it.  I don't believe f_eem will use a high enough (if at all)
>>>>>>>>> bMaxBurst value to change the TXFIFO size.
>>>>>>>>>
>>>>>>>>>>> If not with a ramdisk, we might want to give that a try to avoid
>>>>>>>>>>> the
>>>>>>>>>>> storage path being the bottleneck.  You can use "dd" to create an
>>>>>>>>>>> empty
>>>>>>>>>>> file, and then just use that as the LUN's backing file.
>>>>>>>>>>>
>>>>>>>>>>> echo ramdisk.img >
>>>>>>>>>>> /sys/kernel/config/usb_gadget/g1/functions/mass_storage.0/lun.0/file
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Ah, why didn't I think of that. I have currently mass storage
>>>>>>>>>> setup
>>>>>>>>>> with
>>>>>>>>>> eMMC but it seems that is indeed the bottleneck.
>>>>>>>>>>
>>>>>>>> I created a 64MB disk following the instructions here
>>>>>>>> http://www.linux-usb.org/gadget/file_storage.html (that seems a
>>>>>>>> little
>>>>>>>> outdated, at least I can not start the first partition at sector 8,
>>>>>>>> but
>>>>>>>> minimum 2048), and added a test file on it.
>>>>>>>>
>>>>>>>> I then copy the file to /dev/shm prior to setting configfs
>>>>>>>> (composite
>>>>>>>> device gser/eem/mass_storage/uac2).
>>>>>>>>
>>>>>>>> journal shows:
>>>>>>>>
>>>>>>>> kernel: Mass Storage Function, version: 2009/09/11
>>>>>>>> kernel: LUN: removable file: (no medium)
>>>>>>>>
>>>>>>>> I don't know what that means, because I see the test file on the
>>>>>>>> ramdisk.
>>>>>>>>
>>>>>>>> Then I again used gnome disks to benchmark (read/write 10MB):
>>>>>>>>
>>>>>>>> With V10 on top v5.13.0-rc5:
>>>>>>>>
>>>>>>>> R/W speed = 35.6/35.8MB/s, access time 0.35ms
>>>>>>>>
>>>>>>>> With no patches on top v5.12.0:
>>>>>>>>
>>>>>>>> R/W speed = 35.7/36.1MB/s, access time 0.35ms
>>>>>>> Hi Ferry,
>>>>>>>
>>>>>>>> I see no speed difference (and it's about the same as with the eMMC
>>>>>>>> backed disk). But the patches are causing a new call trace
>>>>>>>>
>>>>>>> Would you happen to know what DWC3 controller revision the device is
>>>>>>> using?  The callstack print occurs, because it looks like it ran
>>>>>>> out of
>>>>>>> internal memory, although there should be logic present for making
>>>>>>> sure
>>>>>>> that at least there is enough room for 1 FIFO per endpoint.
>>>>>>> (possibly
>>>>>>> the logic/math depends on the controller revision)
>>>>> Hi Ferry,
>>>>>
>>>>>> Do you know where I could find that in a file on the device?
>>>>>>
>>>>> Maybe you can just dump the DWC3 registers?
>>>>> cat /sys/kernel/debug/usb/<controller name>/regdump
>>>> This isin host mode:
>>>>
>>>> ~# cat /sys/kernel/debug/usb/dwc3.0.auto/regdump
>>>> GSBUSCFG0 = 0x00000006
>>>> GSBUSCFG1 = 0x00000f00
>>>> GTXTHRCFG = 0x230a0000
>>>> GRXTHRCFG = 0x22800000
>>>> GCTL = 0x45801002
>>>> GEVTEN = 0x00000000
>>>> GSTS = 0x3e800001
>>>> GUCTL1 = 0x00000000
>>>> GSNPSID = 0x5533250a
>>>> GGPIO = 0x00000000
>>>> GUID = 0x00050d00
>>>> GUCTL = 0x0200ce00
>>>> GBUSERRADDR0 = 0x00000000
>>>> GBUSERRADDR1 = 0x00000000
>>> ...
>>> Hi Ferry,
>>>
>>> Great!  Thanks, that got me the information regarding the DWC3 revision
>>> you're using. (its version 2.50a)  I checked the DWC3 databook and looks
>>> like certain versions have some different math to calculate the TXFIFO
>>> size.  I have taken this into account and factored that in in the next
>>> revision.
>>>
>>> Is there a way you could collect the regdump in device mode w/ mass
>>> storage only composition after enumerating with the PC?  That would help
>>> confirm how the TXFIFO resizing logic is working on your platform.
>> Sorry for the wait. Yes I have serial console. So here is in device
>> mode, mass_storage only, enumerated but not mounted:
>>
>> GSBUSCFG0 = 0x00000006
>> GSBUSCFG1 = 0x00000f00
>> GTXTHRCFG = 0x230a0000
>> GRXTHRCFG = 0x02800000
>> GCTL = 0x45802002
>> GEVTEN = 0x00000000
> ...
> Hi Ferry,
>
> Thanks!  So does your board support SSUSB by any chance?  Based on the
> DSTS output, it seems that its currently operating in HSUSB mode.  I
> guess we'd expect higher tput values as well in general if we are in
> SSUSB mode.
Afaik indeed we have only HS.
> The resize logic would only happen if the device was in SSUSB, as SSUSB
> has endpoint bursting, which is the main incentive for the fifo resize.
>
> Thanks
> Wesley Cheng
>
>>> Thanks
>>> Wesley Cheng
>>>
>>>>> Was going to ask for the same to confirm the TXFIFO sizes from your
>>>>> results below :).
>>>>>
>>>>>> Otherwise, I'm hoping Andy will know?
>>>>>>
>>>>>>> Also, is there a way to use just a mass storage only composition?
>>>>>>> Based
>>>>>>> on the above observation, that probably means that the mass storage
>>>>>>> interface wasn't resized at all, because the configuration took up a
>>>>>>> lot
>>>>>>> of the internal FIFO space.
>>>>>> Sure, it's configured through configfs. With only mass_storage I have:
>>>>>>
>>>>>> With V10 on top v5.13.0-rc5:
>>>>>>
>>>>>> R/W speed = 41,6/39,3MB/s, access time 0.33ms
>>>>>>
>>>>>> With no patches on top v5.12.0:
>>>>>>
>>>>>> R/W speed = 41,1/38,7MB/s, access time 0.38ms
>>>>>>
>>>>> Thanks Ferry!  Could you collect the regdump, so I can confirm the two
>>>>> things mentioned?
>>>>>
>>>>> Thanks
>>>>> Wesley Cheng
>>>>>
>>>>>>> Thanks
>>>>>>> Wesley Cheng
>>>>>>>
>>>>>>>> kernel: using random self ethernet address
>>>>>>>> kernel: using random host ethernet address
>>>>>>>> kernel: Mass Storage Function, version: 2009/09/11
>>>>>>>> kernel: LUN: removable file: (no medium)
>>>>>>>> kernel: usb0: HOST MAC aa:bb:cc:dd:ee:f2
>>>>>>>> kernel: usb0: MAC aa:bb:cc:dd:ee:f1
>>>>>>>> kernel: IPv6: ADDRCONF(NETDEV_CHANGE): usb0: link becomes ready
>>>>>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>>>>>> depth:115540359
>>>>>>>> kernel: ------------[ cut here ]------------
>>>>>>>> kernel: WARNING: CPU: 0 PID: 594 at
>>>>>>>> drivers/usb/gadget/udc/core.c:278
>>>>>>>> usb_ep_queue+0x75/0x80
>>>>>>>> kernel: Modules linked in: usb_f_uac2 u_audio usb_f_mass_storage
>>>>>>>> usb_f_eem u_ether usb_f_serial u_serial libcomposite rfcomm
>>>>>>>> iptable_nat
>>>>>>>> bnep snd_sof_nocodec spi_pxa2>
>>>>>>>> kernel: CPU: 0 PID: 594 Comm: irq/14-dwc3 Not tainted
>>>>>>>> 5.13.0-rc5-edison-acpi-standard #1
>>>>>>>> kernel: Hardware name: Intel Corporation Merrifield/BODEGA BAY,
>>>>>>>> BIOS 542
>>>>>>>> 2015.01.21:18.19.48
>>>>>>>> kernel: RIP: 0010:usb_ep_queue+0x75/0x80
>>>>>>>> kernel: Code: 01 73 e4 48 8b 05 fb 63 06 01 48 85 c0 74 12 48 8b
>>>>>>>> 78 08
>>>>>>>> 44 89 e9 4c 89 e2 48 89 ee e8 74 05 00 00 44 89 e8 5d 41 5c 41 5d c3
>>>>>>>> <0f> 0b 41 bd 94 ff ff ff >
>>>>>>>> kernel: RSP: 0000:ffff91eec083fc98 EFLAGS: 00010082
>>>>>>>> kernel: RAX: ffff8af20357d960 RBX: 0000000000000000 RCX:
>>>>>>>> ffff8af202f06400
>>>>>>>> kernel: RDX: 0000000000000a20 RSI: ffff8af208785780 RDI:
>>>>>>>> ffff8af202e9ae00
>>>>>>>> kernel: RBP: ffff8af202e9ae00 R08: 00000000000000c0 R09:
>>>>>>>> ffff8af208785780
>>>>>>>> kernel: R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
>>>>>>>> ffff8af208785780
>>>>>>>> kernel: R13: 0000000000000000 R14: ffff8af202e9ae00 R15:
>>>>>>>> ffff8af203e26cc0
>>>>>>>> kernel: FS:  0000000000000000(0000) GS:ffff8af23e200000(0000)
>>>>>>>> knlGS:0000000000000000
>>>>>>>> kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>>>> kernel: CR2: 000055e2c21f2100 CR3: 0000000003b38000 CR4:
>>>>>>>> 00000000001006f0
>>>>>>>> kernel: Call Trace:
>>>>>>>> kernel:  u_audio_start_playback+0x107/0x1a0 [u_audio]
>>>>>>>> kernel:  composite_setup+0x224/0x1ba0 [libcomposite]
>>>>>>>> kernel:  ? dwc3_gadget_ep_queue+0xf6/0x1a0
>>>>>>>> kernel:  ? usb_ep_queue+0x2a/0x80
>>>>>>>> kernel:  ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>> kernel:  configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>> kernel:  dwc3_ep0_interrupt+0x469/0xa80
>>>>>>>> kernel:  dwc3_thread_interrupt+0x8ee/0xf40
>>>>>>>> kernel:  ? __wake_up_common_lock+0x85/0xb0
>>>>>>>> kernel:  ? disable_irq_nosync+0x10/0x10
>>>>>>>> kernel:  irq_thread_fn+0x1b/0x60
>>>>>>>> kernel:  irq_thread+0xd6/0x170
>>>>>>>> kernel:  ? irq_thread_check_affinity+0x70/0x70
>>>>>>>> kernel:  ? irq_forced_thread_fn+0x70/0x70
>>>>>>>> kernel:  kthread+0x116/0x130
>>>>>>>> kernel:  ? kthread_create_worker_on_cpu+0x60/0x60
>>>>>>>> kernel:  ret_from_fork+0x22/0x30
>>>>>>>> kernel: ---[ end trace e5b9e28058c53584 ]---
>>>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>>>> kernel: dwc3 dwc3.0.auto: request 000000003c32dcc5 was not queued to
>>>>>>>> ep5in
>>>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to
>>>>>>>> ep5in
>>>>>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>>>>>> depth:115540359
>>>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to
>>>>>>>> ep5in
>>>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000036ac129 was not queued to
>>>>>>>> ep5in
>>>>>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>>>>>> depth:115540359
>>>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000ad1b8c18 was not queued to
>>>>>>>> ep5in
>>>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000fbc71244 was not queued to
>>>>>>>> ep5in
>>>>>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>>>>>> depth:115540359
>>>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000fbc71244 was not queued to
>>>>>>>> ep5in
>>>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000ad1b8c18 was not queued to
>>>>>>>> ep5in
>>>>>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>>>>>> depth:115540359
>>>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>>>> kernel: dwc3 dwc3.0.auto: request 000000003c32dcc5 was not queued to
>>>>>>>> ep5in
>>>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to
>>>>>>>> ep5in
>>>>>>>>
>>>>>>>> Removing uac2 from the config makes the call trace go away, but the
>>>>>>>> R/W
>>>>>>>> speed does not change.
>>>>>>>>
>>>>>>>>> :), not a problem...I've been working on getting the ideal set
>>>>>>>>> up for
>>>>>>>>> the performance profiling for awhile, so anything I can do to make
>>>>>>>>> sure
>>>>>>>>> we get some good results.
>>>>>>>>>
>>>>>>>>>> I'll try with a ramdisk and let you know.
>>>>>>>>>>
>>>>>>>>> Thanks again for the testing, Ferry.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Wesley Cheng
>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Wesley Cheng
>>>>>>>>>>>
>>>>>>>>>>>>> Changes in V9:
>>>>>>>>>>>>>        - Fixed incorrect patch in series.  Removed changes in
>>>>>>>>>>>>> DTSI, as
>>>>>>>>>>>>> dwc3-qcom will
>>>>>>>>>>>>>          add the property by default from the kernel.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Changes in V8:
>>>>>>>>>>>>>        - Rebased to usb-testing
>>>>>>>>>>>>>        - Using devm_kzalloc for adding txfifo property in
>>>>>>>>>>>>> dwc3-qcom
>>>>>>>>>>>>>        - Removed DWC3 QCOM ACPI property for enabling the txfifo
>>>>>>>>>>>>> resize
>>>>>>>>>>>>>
>>>>>>>>>>>>> Changes in V7:
>>>>>>>>>>>>>        - Added a new property tx-fifo-max-num for limiting
>>>>>>>>>>>>> how much
>>>>>>>>>>>>> fifo
>>>>>>>>>>>>> space the
>>>>>>>>>>>>>          resizing logic can allocate for endpoints with large
>>>>>>>>>>>>> burst
>>>>>>>>>>>>> values.  This
>>>>>>>>>>>>>          can differ across platforms, and tie in closely with
>>>>>>>>>>>>> overall
>>>>>>>>>>>>> system latency.
>>>>>>>>>>>>>        - Added recommended checks for DWC32.
>>>>>>>>>>>>>        - Added changes to set the tx-fifo-resize property from
>>>>>>>>>>>>> dwc3-qcom by
>>>>>>>>>>>>> default
>>>>>>>>>>>>>          instead of modifying the current DTSI files.
>>>>>>>>>>>>>        - Added comments on all APIs/variables introduced.
>>>>>>>>>>>>>        - Updated the DWC3 YAML to include a better description
>>>>>>>>>>>>> of the
>>>>>>>>>>>>> tx-fifo-resize
>>>>>>>>>>>>>          property and added an entry for tx-fifo-max-num.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Changes in V6:
>>>>>>>>>>>>>        - Rebased patches to usb-testing.
>>>>>>>>>>>>>        - Renamed to PATCH series instead of RFC.
>>>>>>>>>>>>>        - Checking for fs_descriptors instead of
>>>>>>>>>>>>> ss_descriptors for
>>>>>>>>>>>>> determining the
>>>>>>>>>>>>>          endpoint count for a particular configuration.
>>>>>>>>>>>>>        - Re-ordered patch series to fix patch dependencies.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Changes in V5:
>>>>>>>>>>>>>        - Added check_config() logic, which is used to
>>>>>>>>>>>>> communicate the
>>>>>>>>>>>>> number of EPs
>>>>>>>>>>>>>          used in a particular configuration.  Based on this, the
>>>>>>>>>>>>> DWC3
>>>>>>>>>>>>> gadget driver
>>>>>>>>>>>>>          has the ability to know the maximum number of eps
>>>>>>>>>>>>> utilized in
>>>>>>>>>>>>> all
>>>>>>>>>>>>> configs.
>>>>>>>>>>>>>          This helps reduce unnecessary allocation to unused eps,
>>>>>>>>>>>>> and will
>>>>>>>>>>>>> catch fifo
>>>>>>>>>>>>>          allocation issues at bind() time.
>>>>>>>>>>>>>        - Fixed variable declaration to single line per variable,
>>>>>>>>>>>>> and
>>>>>>>>>>>>> reverse xmas.
>>>>>>>>>>>>>        - Created a helper for fifo clearing, which is used by
>>>>>>>>>>>>> ep0.c
>>>>>>>>>>>>>
>>>>>>>>>>>>> Changes in V4:
>>>>>>>>>>>>>        - Removed struct dwc3* as an argument for
>>>>>>>>>>>>> dwc3_gadget_resize_tx_fifos()
>>>>>>>>>>>>>        - Removed WARN_ON(1) in case we run out of fifo space
>>>>>>>>>>>>>        Changes in V3:
>>>>>>>>>>>>>        - Removed "Reviewed-by" tags
>>>>>>>>>>>>>        - Renamed series back to RFC
>>>>>>>>>>>>>        - Modified logic to ensure that fifo_size is reset if we
>>>>>>>>>>>>> pass the
>>>>>>>>>>>>> minimum
>>>>>>>>>>>>>          threshold.  Tested with binding multiple FDs
>>>>>>>>>>>>> requesting 6
>>>>>>>>>>>>> FIFOs.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Changes in V2:
>>>>>>>>>>>>>        - Modified TXFIFO resizing logic to ensure that each
>>>>>>>>>>>>> EP is
>>>>>>>>>>>>> reserved a
>>>>>>>>>>>>>          FIFO.
>>>>>>>>>>>>>        - Removed dev_dbg() prints and fixed typos from patches
>>>>>>>>>>>>>        - Added some more description on the dt-bindings commit
>>>>>>>>>>>>> message
>>>>>>>>>>>>>
>>>>>>>>>>>>> Currently, there is no functionality to allow for resizing the
>>>>>>>>>>>>> TXFIFOs, and
>>>>>>>>>>>>> relying on the HW default setting for the TXFIFO depth.  In
>>>>>>>>>>>>> most
>>>>>>>>>>>>> cases, the
>>>>>>>>>>>>> HW default is probably sufficient, but for USB compositions
>>>>>>>>>>>>> that
>>>>>>>>>>>>> contain
>>>>>>>>>>>>> multiple functions that require EP bursting, the default
>>>>>>>>>>>>> settings
>>>>>>>>>>>>> might not be enough.  Also to note, the current SW will
>>>>>>>>>>>>> assign an
>>>>>>>>>>>>> EP to a
>>>>>>>>>>>>> function driver w/o checking to see if the TXFIFO size for that
>>>>>>>>>>>>> particular
>>>>>>>>>>>>> EP is large enough. (this is a problem if there are multiple HW
>>>>>>>>>>>>> defined
>>>>>>>>>>>>> values for the TXFIFO size)
>>>>>>>>>>>>>
>>>>>>>>>>>>> It is mentioned in the SNPS databook that a minimum of TX FIFO
>>>>>>>>>>>>> depth = 3
>>>>>>>>>>>>> is required for an EP that supports bursting.  Otherwise, there
>>>>>>>>>>>>> may be
>>>>>>>>>>>>> frequent occurences of bursts ending.  For high bandwidth
>>>>>>>>>>>>> functions,
>>>>>>>>>>>>> such as data tethering (protocols that support data
>>>>>>>>>>>>> aggregation),
>>>>>>>>>>>>> mass
>>>>>>>>>>>>> storage, and media transfer protocol (over FFS), the bMaxBurst
>>>>>>>>>>>>> value
>>>>>>>>>>>>> can be
>>>>>>>>>>>>> large, and a bigger TXFIFO depth may prove to be beneficial in
>>>>>>>>>>>>> terms
>>>>>>>>>>>>> of USB
>>>>>>>>>>>>> throughput. (which can be associated to system access latency,
>>>>>>>>>>>>> etc...)  It
>>>>>>>>>>>>> allows for a more consistent burst of traffic, w/o any
>>>>>>>>>>>>> interruptions, as
>>>>>>>>>>>>> data is readily available in the FIFO.
>>>>>>>>>>>>>
>>>>>>>>>>>>> With testing done using the mass storage function driver, the
>>>>>>>>>>>>> results
>>>>>>>>>>>>> show
>>>>>>>>>>>>> that with a larger TXFIFO depth, the bandwidth increased
>>>>>>>>>>>>> significantly.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Test Parameters:
>>>>>>>>>>>>>        - Platform: Qualcomm SM8150
>>>>>>>>>>>>>        - bMaxBurst = 6
>>>>>>>>>>>>>        - USB req size = 256kB
>>>>>>>>>>>>>        - Num of USB reqs = 16
>>>>>>>>>>>>>        - USB Speed = Super-Speed
>>>>>>>>>>>>>        - Function Driver: Mass Storage (w/ ramdisk)
>>>>>>>>>>>>>        - Test Application: CrystalDiskMark
>>>>>>>>>>>>>
>>>>>>>>>>>>> Results:
>>>>>>>>>>>>>
>>>>>>>>>>>>> TXFIFO Depth = 3 max packets
>>>>>>>>>>>>>
>>>>>>>>>>>>> Test Case | Data Size | AVG tput (in MB/s)
>>>>>>>>>>>>> -------------------------------------------
>>>>>>>>>>>>> Sequential|1 GB x     |
>>>>>>>>>>>>> Read      |9 loops    | 193.60
>>>>>>>>>>>>>            |           | 195.86
>>>>>>>>>>>>>                 |           | 184.77
>>>>>>>>>>>>>                 |           | 193.60
>>>>>>>>>>>>> -------------------------------------------
>>>>>>>>>>>>>
>>>>>>>>>>>>> TXFIFO Depth = 6 max packets
>>>>>>>>>>>>>
>>>>>>>>>>>>> Test Case | Data Size | AVG tput (in MB/s)
>>>>>>>>>>>>> -------------------------------------------
>>>>>>>>>>>>> Sequential|1 GB x     |
>>>>>>>>>>>>> Read      |9 loops    | 287.35
>>>>>>>>>>>>>            |           | 304.94
>>>>>>>>>>>>>                 |           | 289.64
>>>>>>>>>>>>>                 |           | 293.61
>>>>>>>>>>>>> -------------------------------------------
>>>>>>>>>>>>>
>>>>>>>>>>>>> Wesley Cheng (6):
>>>>>>>>>>>>>         usb: gadget: udc: core: Introduce check_config to verify
>>>>>>>>>>>>> USB
>>>>>>>>>>>>>           configuration
>>>>>>>>>>>>>         usb: gadget: configfs: Check USB configuration before
>>>>>>>>>>>>> adding
>>>>>>>>>>>>>         usb: dwc3: Resize TX FIFOs to meet EP bursting
>>>>>>>>>>>>> requirements
>>>>>>>>>>>>>         of: Add stub for of_add_property()
>>>>>>>>>>>>>         usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by
>>>>>>>>>>>>> default
>>>>>>>>>>>>>         dt-bindings: usb: dwc3: Update dwc3 TX fifo properties
>>>>>>>>>>>>>
>>>>>>>>>>>>>        .../devicetree/bindings/usb/snps,dwc3.yaml         |
>>>>>>>>>>>>> 15 +-
>>>>>>>>>>>>>        drivers/usb/dwc3/core.c                            |
>>>>>>>>>>>>> 9 +
>>>>>>>>>>>>>        drivers/usb/dwc3/core.h                            |
>>>>>>>>>>>>> 15 ++
>>>>>>>>>>>>>        drivers/usb/dwc3/dwc3-qcom.c                       |
>>>>>>>>>>>>> 9 +
>>>>>>>>>>>>>        drivers/usb/dwc3/ep0.c                             |
>>>>>>>>>>>>> 2 +
>>>>>>>>>>>>>        drivers/usb/dwc3/gadget.c                          | 212
>>>>>>>>>>>>> +++++++++++++++++++++
>>>>>>>>>>>>>        drivers/usb/gadget/configfs.c                      |
>>>>>>>>>>>>> 22 +++
>>>>>>>>>>>>>        drivers/usb/gadget/udc/core.c                      |
>>>>>>>>>>>>> 25 +++
>>>>>>>>>>>>>        include/linux/of.h                                 |
>>>>>>>>>>>>> 5 +
>>>>>>>>>>>>>        include/linux/usb/gadget.h                         |
>>>>>>>>>>>>> 5 +
>>>>>>>>>>>>>        10 files changed, 317 insertions(+), 2 deletions(-)
>>>>>>>>>>>>>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-06-22 18:38           ` Wesley Cheng
  2021-06-22 20:09             ` Ferry Toth
@ 2021-07-22 18:43             ` Ferry Toth
  2021-07-23  6:59               ` Felipe Balbi
  1 sibling, 1 reply; 73+ messages in thread
From: Ferry Toth @ 2021-07-22 18:43 UTC (permalink / raw)
  To: Wesley Cheng, balbi, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list
  Cc: linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	heikki.krogerus, andy.shevchenko

Hi,

Op 22-06-2021 om 20:38 schreef Wesley Cheng:
> 
> 
> On 6/19/2021 5:40 AM, Ferry Toth wrote:
>> Hi
>>
>> Op 18-06-2021 om 00:25 schreef Wesley Cheng:
>>> Hi,
>>>
>>> On 6/17/2021 2:55 PM, Ferry Toth wrote:
>>>> Hi
>>>>
>>>> Op 17-06-2021 om 23:48 schreef Wesley Cheng:
>>>>> Hi,
>>>>>
>>>>> On 6/17/2021 2:01 PM, Ferry Toth wrote:
>>>>>> Hi
>>>>>>
>>>>>> Op 17-06-2021 om 11:58 schreef Wesley Cheng:
>>>>>>> Changes in V10:
>>>>>>>     - Fixed compilation errors in config where OF is not used
>>>>>>> (error due to
>>>>>>>       unknown symbol for of_add_property()).  Add of_add_property()
>>>>>>> stub.
>>>>>>>     - Fixed compilation warning for incorrect argument being passed to
>>>>>>> dwc3_mdwidth
>>>>>> This fixes the OOPS I had in V9. I do not see any change in
>>>>>> performance
>>>>>> on Merrifield though.
>>>>> I see...thanks Ferry! With your testing, are you writing to the
>>>>> device's
>>>>> internal storage (ie UFS, eMMC, etc...), or did you use a ramdisk as
>>>>> well?
>>>> In this case I just tested the EEM path using iperf3.
>>>>
>>> Got it.  I don't believe f_eem will use a high enough (if at all)
>>> bMaxBurst value to change the TXFIFO size.
>>>
>>>>> If not with a ramdisk, we might want to give that a try to avoid the
>>>>> storage path being the bottleneck.  You can use "dd" to create an empty
>>>>> file, and then just use that as the LUN's backing file.
>>>>>
>>>>> echo ramdisk.img >
>>>>> /sys/kernel/config/usb_gadget/g1/functions/mass_storage.0/lun.0/file
>>>> Ah, why didn't I think of that. I have currently mass storage setup with
>>>> eMMC but it seems that is indeed the bottleneck.
>>>>
>> I created a 64MB disk following the instructions here
>> http://www.linux-usb.org/gadget/file_storage.html (that seems a little
>> outdated, at least I can not start the first partition at sector 8, but
>> minimum 2048), and added a test file on it.
>>
>> I then copy the file to /dev/shm prior to setting configfs (composite
>> device gser/eem/mass_storage/uac2).
>>
>> journal shows:
>>
>> kernel: Mass Storage Function, version: 2009/09/11
>> kernel: LUN: removable file: (no medium)
>>
>> I don't know what that means, because I see the test file on the ramdisk.
>>
>> Then I again used gnome disks to benchmark (read/write 10MB):
>>
>> With V10 on top v5.13.0-rc5:
>>
>> R/W speed = 35.6/35.8MB/s, access time 0.35ms
>>
>> With no patches on top v5.12.0:
>>
>> R/W speed = 35.7/36.1MB/s, access time 0.35ms
> 
> Hi Ferry,
> 
>>
>> I see no speed difference (and it's about the same as with the eMMC
>> backed disk). But the patches are causing a new call trace
>>
> 
> Would you happen to know what DWC3 controller revision the device is
> using?  The callstack print occurs, because it looks like it ran out of
> internal memory, although there should be logic present for making sure
> that at least there is enough room for 1 FIFO per endpoint.  (possibly
> the logic/math depends on the controller revision)
> 
> Also, is there a way to use just a mass storage only composition?  Based
> on the above observation, that probably means that the mass storage
> interface wasn't resized at all, because the configuration took up a lot
> of the internal FIFO space.
> 
> Thanks
> Wesley Cheng
> 
>> kernel: using random self ethernet address
>> kernel: using random host ethernet address
>> kernel: Mass Storage Function, version: 2009/09/11
>> kernel: LUN: removable file: (no medium)
>> kernel: usb0: HOST MAC aa:bb:cc:dd:ee:f2
>> kernel: usb0: MAC aa:bb:cc:dd:ee:f1
>> kernel: IPv6: ADDRCONF(NETDEV_CHANGE): usb0: link becomes ready
>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>> depth:115540359
>> kernel: ------------[ cut here ]------------
>> kernel: WARNING: CPU: 0 PID: 594 at drivers/usb/gadget/udc/core.c:278
>> usb_ep_queue+0x75/0x80
>> kernel: Modules linked in: usb_f_uac2 u_audio usb_f_mass_storage
>> usb_f_eem u_ether usb_f_serial u_serial libcomposite rfcomm iptable_nat
>> bnep snd_sof_nocodec spi_pxa2>
>> kernel: CPU: 0 PID: 594 Comm: irq/14-dwc3 Not tainted
>> 5.13.0-rc5-edison-acpi-standard #1
>> kernel: Hardware name: Intel Corporation Merrifield/BODEGA BAY, BIOS 542
>> 2015.01.21:18.19.48
>> kernel: RIP: 0010:usb_ep_queue+0x75/0x80
>> kernel: Code: 01 73 e4 48 8b 05 fb 63 06 01 48 85 c0 74 12 48 8b 78 08
>> 44 89 e9 4c 89 e2 48 89 ee e8 74 05 00 00 44 89 e8 5d 41 5c 41 5d c3
>> <0f> 0b 41 bd 94 ff ff ff >
>> kernel: RSP: 0000:ffff91eec083fc98 EFLAGS: 00010082
>> kernel: RAX: ffff8af20357d960 RBX: 0000000000000000 RCX: ffff8af202f06400
>> kernel: RDX: 0000000000000a20 RSI: ffff8af208785780 RDI: ffff8af202e9ae00
>> kernel: RBP: ffff8af202e9ae00 R08: 00000000000000c0 R09: ffff8af208785780
>> kernel: R10: 00000000ffffe000 R11: 3fffffffffffffff R12: ffff8af208785780
>> kernel: R13: 0000000000000000 R14: ffff8af202e9ae00 R15: ffff8af203e26cc0
>> kernel: FS:  0000000000000000(0000) GS:ffff8af23e200000(0000)
>> knlGS:0000000000000000
>> kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> kernel: CR2: 000055e2c21f2100 CR3: 0000000003b38000 CR4: 00000000001006f0
>> kernel: Call Trace:
>> kernel:  u_audio_start_playback+0x107/0x1a0 [u_audio]
>> kernel:  composite_setup+0x224/0x1ba0 [libcomposite]
>> kernel:  ? dwc3_gadget_ep_queue+0xf6/0x1a0
>> kernel:  ? usb_ep_queue+0x2a/0x80
>> kernel:  ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>> kernel:  configfs_composite_setup+0x6b/0x90 [libcomposite]
>> kernel:  dwc3_ep0_interrupt+0x469/0xa80
>> kernel:  dwc3_thread_interrupt+0x8ee/0xf40
>> kernel:  ? __wake_up_common_lock+0x85/0xb0
>> kernel:  ? disable_irq_nosync+0x10/0x10
>> kernel:  irq_thread_fn+0x1b/0x60
>> kernel:  irq_thread+0xd6/0x170
>> kernel:  ? irq_thread_check_affinity+0x70/0x70
>> kernel:  ? irq_forced_thread_fn+0x70/0x70
>> kernel:  kthread+0x116/0x130
>> kernel:  ? kthread_create_worker_on_cpu+0x60/0x60
>> kernel:  ret_from_fork+0x22/0x30
>> kernel: ---[ end trace e5b9e28058c53584 ]---
>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>> kernel: dwc3 dwc3.0.auto: request 000000003c32dcc5 was not queued to ep5in
>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to ep5in
>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>> depth:115540359
>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to ep5in
>> kernel: dwc3 dwc3.0.auto: request 00000000036ac129 was not queued to ep5in
>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>> depth:115540359
>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>> kernel: dwc3 dwc3.0.auto: request 00000000ad1b8c18 was not queued to ep5in
>> kernel: dwc3 dwc3.0.auto: request 00000000fbc71244 was not queued to ep5in
>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>> depth:115540359
>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>> kernel: dwc3 dwc3.0.auto: request 00000000fbc71244 was not queued to ep5in
>> kernel: dwc3 dwc3.0.auto: request 00000000ad1b8c18 was not queued to ep5in
>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>> depth:115540359
>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>> kernel: dwc3 dwc3.0.auto: request 000000003c32dcc5 was not queued to ep5in
>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to ep5in
>>
>> Removing uac2 from the config makes the call trace go away, but the R/W
>> speed does not change.

I am testing with 5.14.0-rc2 and now related error appears (not in rc1). 
Disabling uac2 solves it still. Any idea what it could be?

BUG: unable to handle page fault for address: 0000000500000000
#PF: supervisor instruction fetch in kernel mode
#PF: error_code(0x0010) - not-present page
PGD 0 P4D 0
Oops: 0010 [#1] SMP PTI
CPU: 0 PID: 494 Comm: irq/14-dwc3 Not tainted 
5.14.0-rc2-edison-acpi-standard #1
Hardware name: Intel Corporation Merrifield/BODEGA BAY, BIOS 542 
2015.01.21:18.19.48
RIP: 0010:0x500000000
Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI: ffff8cd546aed200
RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09: 0000000000000001
R10: ffff8cd541fd28c0 R11: 0000000000000000 R12: ffff8cd547342828
R13: ffff8cd546aed248 R14: 0000000000000000 R15: ffff8cd548b1d000
FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000500000000 CR3: 000000000311e000 CR4: 00000000001006f0
Call Trace:
  ? dwc3_remove_requests.constprop.0+0x14d/0x170
  ? __dwc3_gadget_ep_disable+0x7a/0x160
  ? dwc3_gadget_ep_disable+0x3d/0xd0
  ? usb_ep_disable+0x1c/0x70
  ? u_audio_stop_capture+0x79/0x120 [u_audio]
  ? afunc_set_alt+0x73/0x80 [usb_f_uac2]
  ? composite_setup+0x224/0x1b90 [libcomposite]
  ? __dwc3_gadget_kick_transfer+0x160/0x400
  ? dwc3_gadget_ep_queue+0xf3/0x1a0
  ? configfs_composite_setup+0x6b/0x90 [libcomposite]
  ? configfs_composite_setup+0x6b/0x90 [libcomposite]
  ? dwc3_ep0_interrupt+0x459/0xa40
  ? dwc3_thread_interrupt+0x8ee/0xf40
  ? __schedule+0x235/0x6c0
  ? disable_irq_nosync+0x10/0x10
  ? irq_thread_fn+0x1b/0x60
  ? irq_thread+0xc0/0x160
  ? irq_thread_check_affinity+0x70/0x70
  ? irq_forced_thread_fn+0x70/0x70
  ? kthread+0x122/0x140
  ? set_kthread_struct+0x40/0x40
  ? ret_from_fork+0x22/0x30

>>> :), not a problem...I've been working on getting the ideal set up for
>>> the performance profiling for awhile, so anything I can do to make sure
>>> we get some good results.
>>>
>>>> I'll try with a ramdisk and let you know.
>>>>
>>> Thanks again for the testing, Ferry.
>>>
>>> Thanks
>>> Wesley Cheng
>>>
>>>>> Thanks
>>>>> Wesley Cheng
>>>>>
>>>>>>> Changes in V9:
>>>>>>>     - Fixed incorrect patch in series.  Removed changes in DTSI, as
>>>>>>> dwc3-qcom will
>>>>>>>       add the property by default from the kernel.
>>>>>>>
>>>>>>> Changes in V8:
>>>>>>>     - Rebased to usb-testing
>>>>>>>     - Using devm_kzalloc for adding txfifo property in dwc3-qcom
>>>>>>>     - Removed DWC3 QCOM ACPI property for enabling the txfifo resize
>>>>>>>
>>>>>>> Changes in V7:
>>>>>>>     - Added a new property tx-fifo-max-num for limiting how much fifo
>>>>>>> space the
>>>>>>>       resizing logic can allocate for endpoints with large burst
>>>>>>> values.  This
>>>>>>>       can differ across platforms, and tie in closely with overall
>>>>>>> system latency.
>>>>>>>     - Added recommended checks for DWC32.
>>>>>>>     - Added changes to set the tx-fifo-resize property from
>>>>>>> dwc3-qcom by
>>>>>>> default
>>>>>>>       instead of modifying the current DTSI files.
>>>>>>>     - Added comments on all APIs/variables introduced.
>>>>>>>     - Updated the DWC3 YAML to include a better description of the
>>>>>>> tx-fifo-resize
>>>>>>>       property and added an entry for tx-fifo-max-num.
>>>>>>>
>>>>>>> Changes in V6:
>>>>>>>     - Rebased patches to usb-testing.
>>>>>>>     - Renamed to PATCH series instead of RFC.
>>>>>>>     - Checking for fs_descriptors instead of ss_descriptors for
>>>>>>> determining the
>>>>>>>       endpoint count for a particular configuration.
>>>>>>>     - Re-ordered patch series to fix patch dependencies.
>>>>>>>
>>>>>>> Changes in V5:
>>>>>>>     - Added check_config() logic, which is used to communicate the
>>>>>>> number of EPs
>>>>>>>       used in a particular configuration.  Based on this, the DWC3
>>>>>>> gadget driver
>>>>>>>       has the ability to know the maximum number of eps utilized in
>>>>>>> all
>>>>>>> configs.
>>>>>>>       This helps reduce unnecessary allocation to unused eps, and will
>>>>>>> catch fifo
>>>>>>>       allocation issues at bind() time.
>>>>>>>     - Fixed variable declaration to single line per variable, and
>>>>>>> reverse xmas.
>>>>>>>     - Created a helper for fifo clearing, which is used by ep0.c
>>>>>>>
>>>>>>> Changes in V4:
>>>>>>>     - Removed struct dwc3* as an argument for
>>>>>>> dwc3_gadget_resize_tx_fifos()
>>>>>>>     - Removed WARN_ON(1) in case we run out of fifo space
>>>>>>>     Changes in V3:
>>>>>>>     - Removed "Reviewed-by" tags
>>>>>>>     - Renamed series back to RFC
>>>>>>>     - Modified logic to ensure that fifo_size is reset if we pass the
>>>>>>> minimum
>>>>>>>       threshold.  Tested with binding multiple FDs requesting 6 FIFOs.
>>>>>>>
>>>>>>> Changes in V2:
>>>>>>>     - Modified TXFIFO resizing logic to ensure that each EP is
>>>>>>> reserved a
>>>>>>>       FIFO.
>>>>>>>     - Removed dev_dbg() prints and fixed typos from patches
>>>>>>>     - Added some more description on the dt-bindings commit message
>>>>>>>
>>>>>>> Currently, there is no functionality to allow for resizing the
>>>>>>> TXFIFOs, and
>>>>>>> relying on the HW default setting for the TXFIFO depth.  In most
>>>>>>> cases, the
>>>>>>> HW default is probably sufficient, but for USB compositions that
>>>>>>> contain
>>>>>>> multiple functions that require EP bursting, the default settings
>>>>>>> might not be enough.  Also to note, the current SW will assign an
>>>>>>> EP to a
>>>>>>> function driver w/o checking to see if the TXFIFO size for that
>>>>>>> particular
>>>>>>> EP is large enough. (this is a problem if there are multiple HW
>>>>>>> defined
>>>>>>> values for the TXFIFO size)
>>>>>>>
>>>>>>> It is mentioned in the SNPS databook that a minimum of TX FIFO
>>>>>>> depth = 3
>>>>>>> is required for an EP that supports bursting.  Otherwise, there
>>>>>>> may be
>>>>>>> frequent occurences of bursts ending.  For high bandwidth functions,
>>>>>>> such as data tethering (protocols that support data aggregation),
>>>>>>> mass
>>>>>>> storage, and media transfer protocol (over FFS), the bMaxBurst value
>>>>>>> can be
>>>>>>> large, and a bigger TXFIFO depth may prove to be beneficial in terms
>>>>>>> of USB
>>>>>>> throughput. (which can be associated to system access latency,
>>>>>>> etc...)  It
>>>>>>> allows for a more consistent burst of traffic, w/o any
>>>>>>> interruptions, as
>>>>>>> data is readily available in the FIFO.
>>>>>>>
>>>>>>> With testing done using the mass storage function driver, the results
>>>>>>> show
>>>>>>> that with a larger TXFIFO depth, the bandwidth increased
>>>>>>> significantly.
>>>>>>>
>>>>>>> Test Parameters:
>>>>>>>     - Platform: Qualcomm SM8150
>>>>>>>     - bMaxBurst = 6
>>>>>>>     - USB req size = 256kB
>>>>>>>     - Num of USB reqs = 16
>>>>>>>     - USB Speed = Super-Speed
>>>>>>>     - Function Driver: Mass Storage (w/ ramdisk)
>>>>>>>     - Test Application: CrystalDiskMark
>>>>>>>
>>>>>>> Results:
>>>>>>>
>>>>>>> TXFIFO Depth = 3 max packets
>>>>>>>
>>>>>>> Test Case | Data Size | AVG tput (in MB/s)
>>>>>>> -------------------------------------------
>>>>>>> Sequential|1 GB x     |
>>>>>>> Read      |9 loops    | 193.60
>>>>>>>         |           | 195.86
>>>>>>>              |           | 184.77
>>>>>>>              |           | 193.60
>>>>>>> -------------------------------------------
>>>>>>>
>>>>>>> TXFIFO Depth = 6 max packets
>>>>>>>
>>>>>>> Test Case | Data Size | AVG tput (in MB/s)
>>>>>>> -------------------------------------------
>>>>>>> Sequential|1 GB x     |
>>>>>>> Read      |9 loops    | 287.35
>>>>>>>         |           | 304.94
>>>>>>>              |           | 289.64
>>>>>>>              |           | 293.61
>>>>>>> -------------------------------------------
>>>>>>>
>>>>>>> Wesley Cheng (6):
>>>>>>>      usb: gadget: udc: core: Introduce check_config to verify USB
>>>>>>>        configuration
>>>>>>>      usb: gadget: configfs: Check USB configuration before adding
>>>>>>>      usb: dwc3: Resize TX FIFOs to meet EP bursting requirements
>>>>>>>      of: Add stub for of_add_property()
>>>>>>>      usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by default
>>>>>>>      dt-bindings: usb: dwc3: Update dwc3 TX fifo properties
>>>>>>>
>>>>>>>     .../devicetree/bindings/usb/snps,dwc3.yaml         |  15 +-
>>>>>>>     drivers/usb/dwc3/core.c                            |   9 +
>>>>>>>     drivers/usb/dwc3/core.h                            |  15 ++
>>>>>>>     drivers/usb/dwc3/dwc3-qcom.c                       |   9 +
>>>>>>>     drivers/usb/dwc3/ep0.c                             |   2 +
>>>>>>>     drivers/usb/dwc3/gadget.c                          | 212
>>>>>>> +++++++++++++++++++++
>>>>>>>     drivers/usb/gadget/configfs.c                      |  22 +++
>>>>>>>     drivers/usb/gadget/udc/core.c                      |  25 +++
>>>>>>>     include/linux/of.h                                 |   5 +
>>>>>>>     include/linux/usb/gadget.h                         |   5 +
>>>>>>>     10 files changed, 317 insertions(+), 2 deletions(-)
>>>>>>>
> 


^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-07-22 18:43             ` Ferry Toth
@ 2021-07-23  6:59               ` Felipe Balbi
       [not found]                 ` <d9aef50c-4bd1-4957-13d8-0b6a14b9fcd0@gmail.com>
  0 siblings, 1 reply; 73+ messages in thread
From: Felipe Balbi @ 2021-07-23  6:59 UTC (permalink / raw)
  To: Ferry Toth, Wesley Cheng, gregkh, robh+dt, agross,
	bjorn.andersson, frowand.list
  Cc: linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	heikki.krogerus, andy.shevchenko

[-- Attachment #1: Type: text/plain, Size: 3121 bytes --]


Hi,

Ferry Toth <fntoth@gmail.com> writes:
>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>> kernel: dwc3 dwc3.0.auto: request 00000000fbc71244 was not queued to ep5in
>>> kernel: dwc3 dwc3.0.auto: request 00000000ad1b8c18 was not queued to ep5in
>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>> depth:115540359
>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>> kernel: dwc3 dwc3.0.auto: request 000000003c32dcc5 was not queued to ep5in
>>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to ep5in
>>>
>>> Removing uac2 from the config makes the call trace go away, but the R/W
>>> speed does not change.
>
> I am testing with 5.14.0-rc2 and now related error appears (not in rc1). 
> Disabling uac2 solves it still. Any idea what it could be?
>
> BUG: unable to handle page fault for address: 0000000500000000
> #PF: supervisor instruction fetch in kernel mode
> #PF: error_code(0x0010) - not-present page
> PGD 0 P4D 0
> Oops: 0010 [#1] SMP PTI
> CPU: 0 PID: 494 Comm: irq/14-dwc3 Not tainted 
> 5.14.0-rc2-edison-acpi-standard #1

(cool that you're running on ACPI heh)

> Hardware name: Intel Corporation Merrifield/BODEGA BAY, BIOS 542 
> 2015.01.21:18.19.48
> RIP: 0010:0x500000000
> Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
> RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
> RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX: 0000000000000000
> RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI: ffff8cd546aed200
> RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09: 0000000000000001
> R10: ffff8cd541fd28c0 R11: 0000000000000000 R12: ffff8cd547342828
> R13: ffff8cd546aed248 R14: 0000000000000000 R15: ffff8cd548b1d000
> FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000500000000 CR3: 000000000311e000 CR4: 00000000001006f0
> Call Trace:
>   ? dwc3_remove_requests.constprop.0+0x14d/0x170
>   ? __dwc3_gadget_ep_disable+0x7a/0x160
>   ? dwc3_gadget_ep_disable+0x3d/0xd0
>   ? usb_ep_disable+0x1c/0x70
>   ? u_audio_stop_capture+0x79/0x120 [u_audio]
>   ? afunc_set_alt+0x73/0x80 [usb_f_uac2]
>   ? composite_setup+0x224/0x1b90 [libcomposite]
>   ? __dwc3_gadget_kick_transfer+0x160/0x400
>   ? dwc3_gadget_ep_queue+0xf3/0x1a0
>   ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>   ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>   ? dwc3_ep0_interrupt+0x459/0xa40
>   ? dwc3_thread_interrupt+0x8ee/0xf40
>   ? __schedule+0x235/0x6c0
>   ? disable_irq_nosync+0x10/0x10
>   ? irq_thread_fn+0x1b/0x60
>   ? irq_thread+0xc0/0x160
>   ? irq_thread_check_affinity+0x70/0x70
>   ? irq_forced_thread_fn+0x70/0x70
>   ? kthread+0x122/0x140
>   ? set_kthread_struct+0x40/0x40
>   ? ret_from_fork+0x22/0x30

Do you mind enabling dwc3 traces and collecting them? Trying to figure
out how we got here.

-- 
balbi

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
       [not found]                 ` <d9aef50c-4bd1-4957-13d8-0b6a14b9fcd0@gmail.com>
@ 2021-07-23 11:23                   ` Felipe Balbi
  2021-07-24 20:59                     ` Ferry Toth
  0 siblings, 1 reply; 73+ messages in thread
From: Felipe Balbi @ 2021-07-23 11:23 UTC (permalink / raw)
  To: Ferry Toth, Wesley Cheng, gregkh, robh+dt, agross,
	bjorn.andersson, frowand.list
  Cc: linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	heikki.krogerus, andy.shevchenko

[-- Attachment #1: Type: text/plain, Size: 3937 bytes --]

Ferry Toth <fntoth@gmail.com> writes:

> Hi
>
> Op 23-07-2021 om 08:59 schreef Felipe Balbi:
>> Hi,
>>
>> Ferry Toth <fntoth@gmail.com> writes:
>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>> kernel: dwc3 dwc3.0.auto: request 00000000fbc71244 was not queued to ep5in
>>>>> kernel: dwc3 dwc3.0.auto: request 00000000ad1b8c18 was not queued to ep5in
>>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>>> depth:115540359
>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>> kernel: dwc3 dwc3.0.auto: request 000000003c32dcc5 was not queued to ep5in
>>>>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to ep5in
>>>>>
>>>>> Removing uac2 from the config makes the call trace go away, but the R/W
>>>>> speed does not change.
>>> I am testing with 5.14.0-rc2 and now related error appears (not in rc1).
>>> Disabling uac2 solves it still. Any idea what it could be?
>>>
>>> BUG: unable to handle page fault for address: 0000000500000000
>>> #PF: supervisor instruction fetch in kernel mode
>>> #PF: error_code(0x0010) - not-present page
>>> PGD 0 P4D 0
>>> Oops: 0010 [#1] SMP PTI
>>> CPU: 0 PID: 494 Comm: irq/14-dwc3 Not tainted
>>> 5.14.0-rc2-edison-acpi-standard #1
>> (cool that you're running on ACPI heh)
> Thanks to Andy this is Edison-Arduino board with ACPI.
>>> Hardware name: Intel Corporation Merrifield/BODEGA BAY, BIOS 542
>>> 2015.01.21:18.19.48
>>> RIP: 0010:0x500000000
>>> Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
>>> RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
>>> RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX: 0000000000000000
>>> RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI: ffff8cd546aed200
>>> RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09: 0000000000000001
>>> R10: ffff8cd541fd28c0 R11: 0000000000000000 R12: ffff8cd547342828
>>> R13: ffff8cd546aed248 R14: 0000000000000000 R15: ffff8cd548b1d000
>>> FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000) knlGS:0000000000000000
>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> CR2: 0000000500000000 CR3: 000000000311e000 CR4: 00000000001006f0
>>> Call Trace:
>>>    ? dwc3_remove_requests.constprop.0+0x14d/0x170
>>>    ? __dwc3_gadget_ep_disable+0x7a/0x160
>>>    ? dwc3_gadget_ep_disable+0x3d/0xd0
>>>    ? usb_ep_disable+0x1c/0x
>>>    ? u_audio_stop_capture+0x79/0x120 [u_audio]
>>>    ? afunc_set_alt+0x73/0x80 [usb_f_uac2]
>>>    ? composite_setup+0x224/0x1b90 [libcomposite]
>>>    ? __dwc3_gadget_kick_transfer+0x160/0x400
>>>    ? dwc3_gadget_ep_queue+0xf3/0x1a0
>>>    ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>    ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>    ? dwc3_ep0_interrupt+0x459/0xa40
>>>    ? dwc3_thread_interrupt+0x8ee/0xf40
>>>    ? __schedule+0x235/0x6c0
>>>    ? disable_irq_nosync+0x10/0x10
>>>    ? irq_thread_fn+0x1b/0x60
>>>    ? irq_thread+0xc0/0x160
>>>    ? irq_thread_check_affinity+0x70/0x70
>>>    ? irq_forced_thread_fn+0x70/0x70
>>>    ? kthread+0x122/0x140
>>>    ? set_kthread_struct+0x40/0x40
>>>    ? ret_from_fork+0x22/0x30
>> Do you mind enabling dwc3 traces and collecting them? Trying to figure
>> out how we got here.
>>
> I'll try if I can get the same error by booting with USB in host mode 
> and then switch to device mode. If so I can enable traces and collect as 
> you explained me before.
>
> I'll try before monday, as then I fly for a holiday and will not be 
> available before rc5.

you can enable all of those with kernel cmdline :-)

https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html

you need ftrace_dump_on_oops=1 and also need the correct options on
trace_buf_size and trace_event.

-- 
balbi

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-07-23 11:23                   ` Felipe Balbi
@ 2021-07-24 20:59                     ` Ferry Toth
  2021-07-24 21:19                       ` Andy Shevchenko
  2021-07-25  6:05                       ` Felipe Balbi
  0 siblings, 2 replies; 73+ messages in thread
From: Ferry Toth @ 2021-07-24 20:59 UTC (permalink / raw)
  To: Felipe Balbi, Wesley Cheng, gregkh, robh+dt, agross,
	bjorn.andersson, frowand.list
  Cc: linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	heikki.krogerus, andy.shevchenko

Hi,

Op 23-07-2021 om 13:23 schreef Felipe Balbi:
> Ferry Toth <fntoth@gmail.com> writes:
>
>> Hi
>>
>> Op 23-07-2021 om 08:59 schreef Felipe Balbi:
>>> Hi,
>>>
>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000fbc71244 was not queued to ep5in
>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000ad1b8c18 was not queued to ep5in
>>>>>> kernel: dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
>>>>>> depth:115540359
>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>> kernel: configfs-gadget gadget: u_audio_start_playback:451 Error!
>>>>>> kernel: dwc3 dwc3.0.auto: request 000000003c32dcc5 was not queued to ep5in
>>>>>> kernel: dwc3 dwc3.0.auto: request 00000000b2512aa9 was not queued to ep5in
>>>>>>
>>>>>> Removing uac2 from the config makes the call trace go away, but the R/W
>>>>>> speed does not change.
>>>> I am testing with 5.14.0-rc2 and now related error appears (not in rc1).
>>>> Disabling uac2 solves it still. Any idea what it could be?
>>>>
>>>> BUG: unable to handle page fault for address: 0000000500000000
>>>> #PF: supervisor instruction fetch in kernel mode
>>>> #PF: error_code(0x0010) - not-present page
>>>> PGD 0 P4D 0
>>>> Oops: 0010 [#1] SMP PTI
>>>> CPU: 0 PID: 494 Comm: irq/14-dwc3 Not tainted
>>>> 5.14.0-rc2-edison-acpi-standard #1
>>> (cool that you're running on ACPI heh)
>> Thanks to Andy this is Edison-Arduino board with ACPI.
>>>> Hardware name: Intel Corporation Merrifield/BODEGA BAY, BIOS 542
>>>> 2015.01.21:18.19.48
>>>> RIP: 0010:0x500000000
>>>> Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
>>>> RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
>>>> RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX: 0000000000000000
>>>> RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI: ffff8cd546aed200
>>>> RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09: 0000000000000001
>>>> R10: ffff8cd541fd28c0 R11: 0000000000000000 R12: ffff8cd547342828
>>>> R13: ffff8cd546aed248 R14: 0000000000000000 R15: ffff8cd548b1d000
>>>> FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000) knlGS:0000000000000000
>>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>> CR2: 0000000500000000 CR3: 000000000311e000 CR4: 00000000001006f0
>>>> Call Trace:
>>>>     ? dwc3_remove_requests.constprop.0+0x14d/0x170
>>>>     ? __dwc3_gadget_ep_disable+0x7a/0x160
>>>>     ? dwc3_gadget_ep_disable+0x3d/0xd0
>>>>     ? usb_ep_disable+0x1c/0x
>>>>     ? u_audio_stop_capture+0x79/0x120 [u_audio]
>>>>     ? afunc_set_alt+0x73/0x80 [usb_f_uac2]
>>>>     ? composite_setup+0x224/0x1b90 [libcomposite]
>>>>     ? __dwc3_gadget_kick_transfer+0x160/0x400
>>>>     ? dwc3_gadget_ep_queue+0xf3/0x1a0
>>>>     ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>     ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>     ? dwc3_ep0_interrupt+0x459/0xa40
>>>>     ? dwc3_thread_interrupt+0x8ee/0xf40
>>>>     ? __schedule+0x235/0x6c0
>>>>     ? disable_irq_nosync+0x10/0x10
>>>>     ? irq_thread_fn+0x1b/0x60
>>>>     ? irq_thread+0xc0/0x160
>>>>     ? irq_thread_check_affinity+0x70/0x70
>>>>     ? irq_forced_thread_fn+0x70/0x70
>>>>     ? kthread+0x122/0x140
>>>>     ? set_kthread_struct+0x40/0x40
>>>>     ? ret_from_fork+0x22/0x30
>>> Do you mind enabling dwc3 traces and collecting them? Trying to figure
>>> out how we got here.
>>>
>> I'll try if I can get the same error by booting with USB in host mode
>> and then switch to device mode. If so I can enable traces and collect as
>> you explained me before.
>>
>> I'll try before monday, as then I fly for a holiday and will not be
>> available before rc5.
> you can enable all of those with kernel cmdline :-)
>
> https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
>
> you need ftrace_dump_on_oops=1 and also need the correct options on
> trace_buf_size and trace_event.
>
On Edison-Arduino I have a switch to go to device mode, after which udev 
triggers a script configure gadgets through configfs.

I tried to log following these instructions:

https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs  <https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs>

Unfortunately the kernel crashes so badly I can not get to the ` cp 
/t/trace /root/trace.txt` line (after a while the watchdog kicks).

What to do next?


BTW there is a secondary problem not related to dwc3: the console is not 
working well and loosing chars. I can connect through wifi/ssh though.


^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-07-24 20:59                     ` Ferry Toth
@ 2021-07-24 21:19                       ` Andy Shevchenko
  2021-07-24 22:56                         ` Ferry Toth
  2021-07-25  6:05                       ` Felipe Balbi
  1 sibling, 1 reply; 73+ messages in thread
From: Andy Shevchenko @ 2021-07-24 21:19 UTC (permalink / raw)
  To: Ferry Toth
  Cc: Felipe Balbi, Wesley Cheng, Greg Kroah-Hartman, Rob Herring,
	Andy Gross, Bjorn Andersson, Frank Rowand, USB,
	Linux Kernel Mailing List, devicetree, linux-arm-msm, Jack Pham,
	Krogerus, Heikki

On Sat, Jul 24, 2021 at 11:59 PM Ferry Toth <fntoth@gmail.com> wrote:

> BTW there is a secondary problem not related to dwc3: the console is not
> working well and loosing chars. I can connect through wifi/ssh though.

You mean even on the kernel before any crash happened due to DWC3 stuff?

-- 
With Best Regards,
Andy Shevchenko

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-07-24 21:19                       ` Andy Shevchenko
@ 2021-07-24 22:56                         ` Ferry Toth
  0 siblings, 0 replies; 73+ messages in thread
From: Ferry Toth @ 2021-07-24 22:56 UTC (permalink / raw)
  To: Andy Shevchenko
  Cc: Felipe Balbi, Wesley Cheng, Greg Kroah-Hartman, Rob Herring,
	Andy Gross, Bjorn Andersson, Frank Rowand, USB,
	Linux Kernel Mailing List, devicetree, linux-arm-msm, Jack Pham,
	Krogerus, Heikki

Hi

Op 24-07-2021 om 23:19 schreef Andy Shevchenko:
> On Sat, Jul 24, 2021 at 11:59 PM Ferry Toth<fntoth@gmail.com>  wrote:
>
>> BTW there is a secondary problem not related to dwc3: the console is not
>> working well and loosing chars. I can connect through wifi/ssh though.
> You mean even on the kernel before any crash happened due to DWC3 stuff?

Exactly.

And this was working fine with rc1.

I do have patches (as you know) for improving dma for the hsu. But, for 
the console afaik dma is disabled.


^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-07-24 20:59                     ` Ferry Toth
  2021-07-24 21:19                       ` Andy Shevchenko
@ 2021-07-25  6:05                       ` Felipe Balbi
  2021-07-25 13:32                         ` Ferry Toth
  1 sibling, 1 reply; 73+ messages in thread
From: Felipe Balbi @ 2021-07-25  6:05 UTC (permalink / raw)
  To: Ferry Toth
  Cc: Wesley Cheng, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list, linux-usb, linux-kernel, devicetree, linux-arm-msm,
	jackp, heikki.krogerus, andy.shevchenko


Hi,

Ferry Toth <fntoth@gmail.com> writes:
>>>>> Hardware name: Intel Corporation Merrifield/BODEGA BAY, BIOS 542
>>>>> 2015.01.21:18.19.48
>>>>> RIP: 0010:0x500000000
>>>>> Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
>>>>> RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
>>>>> RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX: 0000000000000000
>>>>> RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI: ffff8cd546aed200
>>>>> RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09: 0000000000000001
>>>>> R10: ffff8cd541fd28c0 R11: 0000000000000000 R12: ffff8cd547342828
>>>>> R13: ffff8cd546aed248 R14: 0000000000000000 R15: ffff8cd548b1d000
>>>>> FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000) knlGS:0000000000000000
>>>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>> CR2: 0000000500000000 CR3: 000000000311e000 CR4: 00000000001006f0
>>>>> Call Trace:
>>>>>     ? dwc3_remove_requests.constprop.0+0x14d/0x170
>>>>>     ? __dwc3_gadget_ep_disable+0x7a/0x160
>>>>>     ? dwc3_gadget_ep_disable+0x3d/0xd0
>>>>>     ? usb_ep_disable+0x1c/0x
>>>>>     ? u_audio_stop_capture+0x79/0x120 [u_audio]
>>>>>     ? afunc_set_alt+0x73/0x80 [usb_f_uac2]
>>>>>     ? composite_setup+0x224/0x1b90 [libcomposite]
>>>>>     ? __dwc3_gadget_kick_transfer+0x160/0x400
>>>>>     ? dwc3_gadget_ep_queue+0xf3/0x1a0
>>>>>     ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>     ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>     ? dwc3_ep0_interrupt+0x459/0xa40
>>>>>     ? dwc3_thread_interrupt+0x8ee/0xf40
>>>>>     ? __schedule+0x235/0x6c0
>>>>>     ? disable_irq_nosync+0x10/0x10
>>>>>     ? irq_thread_fn+0x1b/0x60
>>>>>     ? irq_thread+0xc0/0x160
>>>>>     ? irq_thread_check_affinity+0x70/0x70
>>>>>     ? irq_forced_thread_fn+0x70/0x70
>>>>>     ? kthread+0x122/0x140
>>>>>     ? set_kthread_struct+0x40/0x40
>>>>>     ? ret_from_fork+0x22/0x30
>>>> Do you mind enabling dwc3 traces and collecting them? Trying to figure
>>>> out how we got here.
>>>>
>>> I'll try if I can get the same error by booting with USB in host mode
>>> and then switch to device mode. If so I can enable traces and collect as
>>> you explained me before.
>>>
>>> I'll try before monday, as then I fly for a holiday and will not be
>>> available before rc5.
>> you can enable all of those with kernel cmdline :-)
>>
>> https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
>>
>> you need ftrace_dump_on_oops=1 and also need the correct options on
>> trace_buf_size and trace_event.
>>
> On Edison-Arduino I have a switch to go to device mode, after which
> udev triggers a script configure gadgets through configfs.
>
> I tried to log following these instructions:
>
> https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs  <https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs>
>
> Unfortunately the kernel crashes so badly I can not get to the ` cp
> /t/trace /root/trace.txt` line (after a while the watchdog kicks).
>
> What to do next?

Pass ftrace_dump_on_oops to kernel cmdline.

-- 
balbi

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-07-25  6:05                       ` Felipe Balbi
@ 2021-07-25 13:32                         ` Ferry Toth
  2021-07-25 14:05                           ` Felipe Balbi
  0 siblings, 1 reply; 73+ messages in thread
From: Ferry Toth @ 2021-07-25 13:32 UTC (permalink / raw)
  To: Felipe Balbi
  Cc: Wesley Cheng, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list, linux-usb, linux-kernel, devicetree, linux-arm-msm,
	jackp, heikki.krogerus, andy.shevchenko

[-- Attachment #1: Type: text/plain, Size: 3369 bytes --]

Hi

Op 25-07-2021 om 08:05 schreef Felipe Balbi:
> Hi,
>
> Ferry Toth <fntoth@gmail.com> writes:
>>>>>> Hardware name: Intel Corporation Merrifield/BODEGA BAY, BIOS 542
>>>>>> 2015.01.21:18.19.48
>>>>>> RIP: 0010:0x500000000
>>>>>> Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
>>>>>> RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
>>>>>> RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX: 0000000000000000
>>>>>> RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI: ffff8cd546aed200
>>>>>> RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09: 0000000000000001
>>>>>> R10: ffff8cd541fd28c0 R11: 0000000000000000 R12: ffff8cd547342828
>>>>>> R13: ffff8cd546aed248 R14: 0000000000000000 R15: ffff8cd548b1d000
>>>>>> FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000) knlGS:0000000000000000
>>>>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>> CR2: 0000000500000000 CR3: 000000000311e000 CR4: 00000000001006f0
>>>>>> Call Trace:
>>>>>>      ? dwc3_remove_requests.constprop.0+0x14d/0x170
>>>>>>      ? __dwc3_gadget_ep_disable+0x7a/0x160
>>>>>>      ? dwc3_gadget_ep_disable+0x3d/0xd0
>>>>>>      ? usb_ep_disable+0x1c/0x
>>>>>>      ? u_audio_stop_capture+0x79/0x120 [u_audio]
>>>>>>      ? afunc_set_alt+0x73/0x80 [usb_f_uac2]
>>>>>>      ? composite_setup+0x224/0x1b90 [libcomposite]
>>>>>>      ? __dwc3_gadget_kick_transfer+0x160/0x400
>>>>>>      ? dwc3_gadget_ep_queue+0xf3/0x1a0
>>>>>>      ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>      ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>      ? dwc3_ep0_interrupt+0x459/0xa40
>>>>>>      ? dwc3_thread_interrupt+0x8ee/0xf40
>>>>>>      ? __schedule+0x235/0x6c0
>>>>>>      ? disable_irq_nosync+0x10/0x10
>>>>>>      ? irq_thread_fn+0x1b/0x60
>>>>>>      ? irq_thread+0xc0/0x160
>>>>>>      ? irq_thread_check_affinity+0x70/0x70
>>>>>>      ? irq_forced_thread_fn+0x70/0x70
>>>>>>      ? kthread+0x122/0x140
>>>>>>      ? set_kthread_struct+0x40/0x40
>>>>>>      ? ret_from_fork+0x22/0x30
>>>>> Do you mind enabling dwc3 traces and collecting them? Trying to figure
>>>>> out how we got here.
>>>>>
>>>> I'll try if I can get the same error by booting with USB in host mode
>>>> and then switch to device mode. If so I can enable traces and collect as
>>>> you explained me before.
>>>>
>>>> I'll try before monday, as then I fly for a holiday and will not be
>>>> available before rc5.
>>> you can enable all of those with kernel cmdline :-)
>>>
>>> https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
>>>
>>> you need ftrace_dump_on_oops=1 and also need the correct options on
>>> trace_buf_size and trace_event.
>>>
>> On Edison-Arduino I have a switch to go to device mode, after which
>> udev triggers a script configure gadgets through configfs.
>>
>> I tried to log following these instructions:
>>
>> https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs  <https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs>
>>
>> Unfortunately the kernel crashes so badly I can not get to the ` cp
>> /t/trace /root/trace.txt` line (after a while the watchdog kicks).
>>
>> What to do next?
> Pass ftrace_dump_on_oops to kernel cmdline.
>
No sure if I did this right, on oops everything is pushed to console 
(115k2 serial), I hope nothing essential is lost.

I copied the screen buffer to file see attached.


[-- Attachment #2: dump-on-oops.7z --]
[-- Type: application/x-7z-compressed, Size: 35551 bytes --]

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-07-25 13:32                         ` Ferry Toth
@ 2021-07-25 14:05                           ` Felipe Balbi
  2021-07-25 16:54                             ` Ferry Toth
  0 siblings, 1 reply; 73+ messages in thread
From: Felipe Balbi @ 2021-07-25 14:05 UTC (permalink / raw)
  To: Ferry Toth
  Cc: Wesley Cheng, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list, linux-usb, linux-kernel, devicetree, linux-arm-msm,
	jackp, heikki.krogerus, andy.shevchenko


Hi,

Ferry Toth <fntoth@gmail.com> writes:

> Hi
>
> Op 25-07-2021 om 08:05 schreef Felipe Balbi:
>> Hi,
>>
>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>> Hardware name: Intel Corporation Merrifield/BODEGA BAY, BIOS 542
>>>>>>> 2015.01.21:18.19.48
>>>>>>> RIP: 0010:0x500000000
>>>>>>> Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
>>>>>>> RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
>>>>>>> RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX: 0000000000000000
>>>>>>> RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI: ffff8cd546aed200
>>>>>>> RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09: 0000000000000001
>>>>>>> R10: ffff8cd541fd28c0 R11: 0000000000000000 R12: ffff8cd547342828
>>>>>>> R13: ffff8cd546aed248 R14: 0000000000000000 R15: ffff8cd548b1d000
>>>>>>> FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000) knlGS:0000000000000000
>>>>>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>>> CR2: 0000000500000000 CR3: 000000000311e000 CR4: 00000000001006f0
>>>>>>> Call Trace:
>>>>>>>      ? dwc3_remove_requests.constprop.0+0x14d/0x170
>>>>>>>      ? __dwc3_gadget_ep_disable+0x7a/0x160
>>>>>>>      ? dwc3_gadget_ep_disable+0x3d/0xd0
>>>>>>>      ? usb_ep_disable+0x1c/0x
>>>>>>>      ? u_audio_stop_capture+0x79/0x120 [u_audio]
>>>>>>>      ? afunc_set_alt+0x73/0x80 [usb_f_uac2]

So this is triggered by a SetInterface request...

>>>>>>>      ? composite_setup+0x224/0x1b90 [libcomposite]
>>>>>>>      ? __dwc3_gadget_kick_transfer+0x160/0x400
>>>>>>>      ? dwc3_gadget_ep_queue+0xf3/0x1a0
>>>>>>>      ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>      ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>      ? dwc3_ep0_interrupt+0x459/0xa40
>>>>>>>      ? dwc3_thread_interrupt+0x8ee/0xf40
>>>>>>>      ? __schedule+0x235/0x6c0
>>>>>>>      ? disable_irq_nosync+0x10/0x10
>>>>>>>      ? irq_thread_fn+0x1b/0x60
>>>>>>>      ? irq_thread+0xc0/0x160
>>>>>>>      ? irq_thread_check_affinity+0x70/0x70
>>>>>>>      ? irq_forced_thread_fn+0x70/0x70
>>>>>>>      ? kthread+0x122/0x140
>>>>>>>      ? set_kthread_struct+0x40/0x40
>>>>>>>      ? ret_from_fork+0x22/0x30
>>>>>> Do you mind enabling dwc3 traces and collecting them? Trying to figure
>>>>>> out how we got here.
>>>>>>
>>>>> I'll try if I can get the same error by booting with USB in host mode
>>>>> and then switch to device mode. If so I can enable traces and collect as
>>>>> you explained me before.
>>>>>
>>>>> I'll try before monday, as then I fly for a holiday and will not be
>>>>> available before rc5.
>>>> you can enable all of those with kernel cmdline :-)
>>>>
>>>> https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
>>>>
>>>> you need ftrace_dump_on_oops=1 and also need the correct options on
>>>> trace_buf_size and trace_event.
>>>>
>>> On Edison-Arduino I have a switch to go to device mode, after which
>>> udev triggers a script configure gadgets through configfs.
>>>
>>> I tried to log following these instructions:
>>>
>>> https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs  <https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs>
>>>
>>> Unfortunately the kernel crashes so badly I can not get to the ` cp
>>> /t/trace /root/trace.txt` line (after a while the watchdog kicks).
>>>
>>> What to do next?
>> Pass ftrace_dump_on_oops to kernel cmdline.
>>
> No sure if I did this right, on oops everything is pushed to console
> (115k2 serial), I hope nothing essential is lost.
>
> I copied the screen buffer to file see attached.

Thank you, I bet it took quite a some time :-) Anyway, looking at
the logs around Set Interface requests, we can track every endpoint
that's disabled. I'll take a guess and assume we're failing at the last
Set Interface, that means we should have something odd with ep6in, but
everything looks fine in the trace output:

[   75.823107] irq/14-d-596       0d... 42789194us : dwc3_gadget_ep_enable: ep6in: mps 192/346 streams 16 burst 0 ring 0/0 flags E:swbp:<
[   75.835472] irq/14-d-596       0d... 42789198us : dwc3_alloc_request: ep6in: req 0000000002c71409 length 0/0 zsI ==> 0
[   75.846416] irq/14-d-596       0d... 42789202us : dwc3_ep_queue: ep6in: req 0000000002c71409 length 0/192 zsI ==> -115
[   75.857360] irq/14-d-596       0d... 42789204us : dwc3_alloc_request: ep6in: req 00000000a324f5d0 length 0/0 zsI ==> 0
[   75.868301] irq/14-d-596       0d... 42789206us : dwc3_ep_queue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -115
[   75.879244] irq/14-d-596       0d... 42789209us : dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0] (Not Active) [Status Phase]
[   75.891880] irq/14-d-596       0d... 42789211us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
[   75.989131] irq/14-d-596       0d... 42789224us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
[   76.096261] irq/14-d-596       0d... 42789272us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
[   76.107834] irq/14-d-596       0d... 42789275us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
[   76.122944] irq/14-d-596       0d... 42789277us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 zsI ==> 0
[   76.134160] irq/14-d-596       0d... 42789280us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
[   76.231322] irq/14-d-596       0d... 42789292us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
[   76.297418] kworker/-23        0d... 42789670us : dwc3_ep_queue: ep3in: req 0000000029586135 length 0/96 ZsI ==> -115
[   76.308278] kworker/-23        0d... 42789695us : dwc3_prepare_trb: ep3in: trb 00000000b81213d6 (E1:D0) buf 0000000003b7a800 size 96 ctrl 00000811 (Hlcs:sC:normal)
[   76.395294] kworker/-23        0d... 42789707us : dwc3_gadget_ep_cmd: ep3in: cmd 'Update Transfer' [60007] params 00000000 00000000 00000000 --> status: Successful
[   76.471900] irq/14-d-596       0d... 42789842us : dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
[   76.489308] irq/14-d-596       0d... 42789845us : dwc3_ctrl_req: Set Interface(Intf = 5, Alt.Setting = 0)
[   76.505650] irq/14-d-596       0d... 42789851us : dwc3_ep_dequeue: ep6in: req 0000000002c71409 length 0/192 zsI ==> -115
[   76.523315] irq/14-d-596       0d... 42789854us : dwc3_gadget_giveback: ep6in: req 0000000002c71409 length 0/192 zsI ==> -104
[   76.541427] irq/14-d-596       0d... 42789857us : dwc3_free_request: ep6in: req 0000000002c71409 length 0/192 zsI ==> -104
[   76.559267] irq/14-d-596       0d... 42789859us : dwc3_ep_dequeue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -115
[   76.576937] irq/14-d-596       0d... 42789861us : dwc3_gadget_giveback: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -104
[   76.595046] irq/14-d-596       0d... 42789862us : dwc3_free_request: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -104
[   76.612892] irq/14-d-596       0d... 42789865us : dwc3_gadget_ep_disable: ep6in: mps 192/346 streams 16 burst 0 ring 0/0 flags E:swbp:<
[   76.665535] irq/14-d-596       0d... 42789873us : dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0] (Not Active) [Status Phase]
[   76.684716] irq/14-d-596       0d... 42789875us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
[   76.819195] irq/14-d-596       0d... 42789886us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
[   76.926324] irq/14-d-596       0d... 42789930us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
[   76.937892] irq/14-d-596       0d... 42789933us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
[   76.953003] irq/14-d-596       0d... 42789935us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 zsI ==> 0
[   76.964217] irq/14-d-596       0d... 42789938us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
[   77.061379] irq/14-d-596       0d... 42789950us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
[   77.168595] irq/14-d-596       0d... 42790509us : dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
[   77.180159] irq/14-d-596       0d... 42790512us : dwc3_ctrl_req: Get String Descriptor(Index = 18, Length = 255)
[   77.190578] irq/14-d-596       0d... 42790537us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 0000000003b68000 size 36 ctrl 00000c53 (HLcs:SC:data)
[   77.287648] irq/14-d-596       0d... 42790550us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
[   77.333107] irq/14-d-596       0d... 42790557us : dwc3_event: event (000010c2): ep0in: Transfer Not Ready [0] (Not Active) [Data Phase]
[   77.407223] irq/14-d-596       0d... 42790575us : dwc3_event: event (000090c2): ep0in: Transfer Not Ready [0] (Active) [Data Phase]
[   77.480985] irq/14-d-596       0d... 42790588us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Data Phase]
[   77.492376] irq/14-d-596       0d... 42790590us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 0000000003b68000 size 0 ctrl 00000c52 (hLcs:SC:data)
[   77.507221] irq/14-d-596       0d... 42790595us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 36/36 ZsI ==> 0
[   77.518609] irq/14-d-596       0d... 42790597us : dwc3_event: event (000020c0): ep0out: Transfer Not Ready [0] (Not Active) [Status Phase]
[   77.531332] irq/14-d-596       0d... 42790598us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c43 (HLcs:SC:status3)
[   77.628669] irq/14-d-596       0d... 42790609us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful

Do you mind adding a few prints in dwc3_remove_requests to tell us which
endpoint is being processed? Then we'll know for sure which one caused
the crash.

-- 
balbi

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-07-25 14:05                           ` Felipe Balbi
@ 2021-07-25 16:54                             ` Ferry Toth
  2021-07-26  5:57                               ` Felipe Balbi
  0 siblings, 1 reply; 73+ messages in thread
From: Ferry Toth @ 2021-07-25 16:54 UTC (permalink / raw)
  To: Felipe Balbi
  Cc: Wesley Cheng, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list, linux-usb, linux-kernel, devicetree, linux-arm-msm,
	jackp, heikki.krogerus, andy.shevchenko

Hi

Op 25-07-2021 om 16:05 schreef Felipe Balbi:
> Hi,
>
> Ferry Toth <fntoth@gmail.com> writes:
>
>> Hi
>>
>> Op 25-07-2021 om 08:05 schreef Felipe Balbi:
>>> Hi,
>>>
>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>> Hardware name: Intel Corporation Merrifield/BODEGA BAY, BIOS 542
>>>>>>>> 2015.01.21:18.19.48
>>>>>>>> RIP: 0010:0x500000000
>>>>>>>> Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
>>>>>>>> RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
>>>>>>>> RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX: 0000000000000000
>>>>>>>> RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI: ffff8cd546aed200
>>>>>>>> RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09: 0000000000000001
>>>>>>>> R10: ffff8cd541fd28c0 R11: 0000000000000000 R12: ffff8cd547342828
>>>>>>>> R13: ffff8cd546aed248 R14: 0000000000000000 R15: ffff8cd548b1d000
>>>>>>>> FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000) knlGS:0000000000000000
>>>>>>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>>>> CR2: 0000000500000000 CR3: 000000000311e000 CR4: 00000000001006f0
>>>>>>>> Call Trace:
>>>>>>>>       ? dwc3_remove_requests.constprop.0+0x14d/0x170
>>>>>>>>       ? __dwc3_gadget_ep_disable+0x7a/0x160
>>>>>>>>       ? dwc3_gadget_ep_disable+0x3d/0xd0
>>>>>>>>       ? usb_ep_disable+0x1c/0x
>>>>>>>>       ? u_audio_stop_capture+0x79/0x120 [u_audio]
>>>>>>>>       ? afunc_set_alt+0x73/0x80 [usb_f_uac2]
> So this is triggered by a SetInterface request...
>
>>>>>>>>       ? composite_setup+0x224/0x1b90 [libcomposite]
>>>>>>>>       ? __dwc3_gadget_kick_transfer+0x160/0x400
>>>>>>>>       ? dwc3_gadget_ep_queue+0xf3/0x1a0
>>>>>>>>       ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>>       ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>>       ? dwc3_ep0_interrupt+0x459/0xa40
>>>>>>>>       ? dwc3_thread_interrupt+0x8ee/0xf40
>>>>>>>>       ? __schedule+0x235/0x6c0
>>>>>>>>       ? disable_irq_nosync+0x10/0x10
>>>>>>>>       ? irq_thread_fn+0x1b/0x60
>>>>>>>>       ? irq_thread+0xc0/0x160
>>>>>>>>       ? irq_thread_check_affinity+0x70/0x70
>>>>>>>>       ? irq_forced_thread_fn+0x70/0x70
>>>>>>>>       ? kthread+0x122/0x140
>>>>>>>>       ? set_kthread_struct+0x40/0x40
>>>>>>>>       ? ret_from_fork+0x22/0x30
>>>>>>> Do you mind enabling dwc3 traces and collecting them? Trying to figure
>>>>>>> out how we got here.
>>>>>>>
>>>>>> I'll try if I can get the same error by booting with USB in host mode
>>>>>> and then switch to device mode. If so I can enable traces and collect as
>>>>>> you explained me before.
>>>>>>
>>>>>> I'll try before monday, as then I fly for a holiday and will not be
>>>>>> available before rc5.
>>>>> you can enable all of those with kernel cmdline :-)
>>>>>
>>>>> https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
>>>>>
>>>>> you need ftrace_dump_on_oops=1 and also need the correct options on
>>>>> trace_buf_size and trace_event.
>>>>>
>>>> On Edison-Arduino I have a switch to go to device mode, after which
>>>> udev triggers a script configure gadgets through configfs.
>>>>
>>>> I tried to log following these instructions:
>>>>
>>>> https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs  <https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs>
>>>>
>>>> Unfortunately the kernel crashes so badly I can not get to the ` cp
>>>> /t/trace /root/trace.txt` line (after a while the watchdog kicks).
>>>>
>>>> What to do next?
>>> Pass ftrace_dump_on_oops to kernel cmdline.
>>>
>> No sure if I did this right, on oops everything is pushed to console
>> (115k2 serial), I hope nothing essential is lost.
>>
>> I copied the screen buffer to file see attached.
> Thank you, I bet it took quite a some time :-) Anyway, looking at
> the logs around Set Interface requests, we can track every endpoint
> that's disabled. I'll take a guess and assume we're failing at the last
> Set Interface, that means we should have something odd with ep6in, but
> everything looks fine in the trace output:
>
> [   75.823107] irq/14-d-596       0d... 42789194us : dwc3_gadget_ep_enable: ep6in: mps 192/346 streams 16 burst 0 ring 0/0 flags E:swbp:<
> [   75.835472] irq/14-d-596       0d... 42789198us : dwc3_alloc_request: ep6in: req 0000000002c71409 length 0/0 zsI ==> 0
> [   75.846416] irq/14-d-596       0d... 42789202us : dwc3_ep_queue: ep6in: req 0000000002c71409 length 0/192 zsI ==> -115
> [   75.857360] irq/14-d-596       0d... 42789204us : dwc3_alloc_request: ep6in: req 00000000a324f5d0 length 0/0 zsI ==> 0
> [   75.868301] irq/14-d-596       0d... 42789206us : dwc3_ep_queue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -115
> [   75.879244] irq/14-d-596       0d... 42789209us : dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0] (Not Active) [Status Phase]
> [   75.891880] irq/14-d-596       0d... 42789211us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
> [   75.989131] irq/14-d-596       0d... 42789224us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
> [   76.096261] irq/14-d-596       0d... 42789272us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
> [   76.107834] irq/14-d-596       0d... 42789275us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
> [   76.122944] irq/14-d-596       0d... 42789277us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 zsI ==> 0
> [   76.134160] irq/14-d-596       0d... 42789280us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
> [   76.231322] irq/14-d-596       0d... 42789292us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
> [   76.297418] kworker/-23        0d... 42789670us : dwc3_ep_queue: ep3in: req 0000000029586135 length 0/96 ZsI ==> -115
> [   76.308278] kworker/-23        0d... 42789695us : dwc3_prepare_trb: ep3in: trb 00000000b81213d6 (E1:D0) buf 0000000003b7a800 size 96 ctrl 00000811 (Hlcs:sC:normal)
> [   76.395294] kworker/-23        0d... 42789707us : dwc3_gadget_ep_cmd: ep3in: cmd 'Update Transfer' [60007] params 00000000 00000000 00000000 --> status: Successful
> [   76.471900] irq/14-d-596       0d... 42789842us : dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
> [   76.489308] irq/14-d-596       0d... 42789845us : dwc3_ctrl_req: Set Interface(Intf = 5, Alt.Setting = 0)
> [   76.505650] irq/14-d-596       0d... 42789851us : dwc3_ep_dequeue: ep6in: req 0000000002c71409 length 0/192 zsI ==> -115
> [   76.523315] irq/14-d-596       0d... 42789854us : dwc3_gadget_giveback: ep6in: req 0000000002c71409 length 0/192 zsI ==> -104
> [   76.541427] irq/14-d-596       0d... 42789857us : dwc3_free_request: ep6in: req 0000000002c71409 length 0/192 zsI ==> -104
> [   76.559267] irq/14-d-596       0d... 42789859us : dwc3_ep_dequeue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -115
> [   76.576937] irq/14-d-596       0d... 42789861us : dwc3_gadget_giveback: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -104
> [   76.595046] irq/14-d-596       0d... 42789862us : dwc3_free_request: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -104
> [   76.612892] irq/14-d-596       0d... 42789865us : dwc3_gadget_ep_disable: ep6in: mps 192/346 streams 16 burst 0 ring 0/0 flags E:swbp:<
> [   76.665535] irq/14-d-596       0d... 42789873us : dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0] (Not Active) [Status Phase]
> [   76.684716] irq/14-d-596       0d... 42789875us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
> [   76.819195] irq/14-d-596       0d... 42789886us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
> [   76.926324] irq/14-d-596       0d... 42789930us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
> [   76.937892] irq/14-d-596       0d... 42789933us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
> [   76.953003] irq/14-d-596       0d... 42789935us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 zsI ==> 0
> [   76.964217] irq/14-d-596       0d... 42789938us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
> [   77.061379] irq/14-d-596       0d... 42789950us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
> [   77.168595] irq/14-d-596       0d... 42790509us : dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
> [   77.180159] irq/14-d-596       0d... 42790512us : dwc3_ctrl_req: Get String Descriptor(Index = 18, Length = 255)
> [   77.190578] irq/14-d-596       0d... 42790537us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 0000000003b68000 size 36 ctrl 00000c53 (HLcs:SC:data)
> [   77.287648] irq/14-d-596       0d... 42790550us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
> [   77.333107] irq/14-d-596       0d... 42790557us : dwc3_event: event (000010c2): ep0in: Transfer Not Ready [0] (Not Active) [Data Phase]
> [   77.407223] irq/14-d-596       0d... 42790575us : dwc3_event: event (000090c2): ep0in: Transfer Not Ready [0] (Active) [Data Phase]
> [   77.480985] irq/14-d-596       0d... 42790588us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Data Phase]
> [   77.492376] irq/14-d-596       0d... 42790590us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 0000000003b68000 size 0 ctrl 00000c52 (hLcs:SC:data)
> [   77.507221] irq/14-d-596       0d... 42790595us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 36/36 ZsI ==> 0
> [   77.518609] irq/14-d-596       0d... 42790597us : dwc3_event: event (000020c0): ep0out: Transfer Not Ready [0] (Not Active) [Status Phase]
> [   77.531332] irq/14-d-596       0d... 42790598us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c43 (HLcs:SC:status3)
> [   77.628669] irq/14-d-596       0d... 42790609us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>
> Do you mind adding a few prints in dwc3_remove_requests to tell us which
> endpoint is being processed? Then we'll know for sure which one caused
> the crash.
>
I wouldn't mind but am leaving on a holiday, won't have time until 6 aug.

But as I am using configfs (excerpt follows) and just disabling the last 
2 line resolves the issue, I'm guessing uac2 is the issue. Or exceeding 
the available resources.

# Create directory structure
mkdir "${GADGET_BASE_DIR}"
cd "${GADGET_BASE_DIR}"
mkdir -p configs/c.1/strings/0x409
mkdir -p strings/0x409

# Serial device
mkdir functions/gser.usb0
ln -s functions/gser.usb0 configs/c.1/
###

# Ethernet device
mkdir functions/eem.usb0
echo "${DEV_ETH_ADDR}" > functions/eem.usb0/dev_addr
echo "${HOST_ETH_ADDR}" > functions/eem.usb0/host_addr
ln -s functions/eem.usb0 configs/c.1/

# Mass Storage device
mkdir functions/mass_storage.usb0
echo 1 > functions/mass_storage.usb0/stall
echo 0 > functions/mass_storage.usb0/lun.0/cdrom
echo 0 > functions/mass_storage.usb0/lun.0/ro
echo 0 > functions/mass_storage.usb0/lun.0/nofua
echo "${USBDISK}" > functions/mass_storage.usb0/lun.0/file
ln -s functions/mass_storage.usb0 configs/c.1/

# UAC2 device
mkdir functions/uac2.usb0
ln -s functions/uac2.usb0 configs/c.1


^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-07-25 16:54                             ` Ferry Toth
@ 2021-07-26  5:57                               ` Felipe Balbi
  2021-07-26 14:33                                 ` Wesley Cheng
  2021-08-15 20:56                                 ` Ferry Toth
  0 siblings, 2 replies; 73+ messages in thread
From: Felipe Balbi @ 2021-07-26  5:57 UTC (permalink / raw)
  To: Ferry Toth
  Cc: Wesley Cheng, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list, linux-usb, linux-kernel, devicetree, linux-arm-msm,
	jackp, heikki.krogerus, andy.shevchenko


Hi,

Ferry Toth <fntoth@gmail.com> writes:
>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>> Hardware name: Intel Corporation Merrifield/BODEGA BAY, BIOS 542
>>>>>>>>> 2015.01.21:18.19.48
>>>>>>>>> RIP: 0010:0x500000000
>>>>>>>>> Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
>>>>>>>>> RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
>>>>>>>>> RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX: 0000000000000000
>>>>>>>>> RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI: ffff8cd546aed200
>>>>>>>>> RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09: 0000000000000001
>>>>>>>>> R10: ffff8cd541fd28c0 R11: 0000000000000000 R12: ffff8cd547342828
>>>>>>>>> R13: ffff8cd546aed248 R14: 0000000000000000 R15: ffff8cd548b1d000
>>>>>>>>> FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000) knlGS:0000000000000000
>>>>>>>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>>>>> CR2: 0000000500000000 CR3: 000000000311e000 CR4: 00000000001006f0
>>>>>>>>> Call Trace:
>>>>>>>>>       ? dwc3_remove_requests.constprop.0+0x14d/0x170
>>>>>>>>>       ? __dwc3_gadget_ep_disable+0x7a/0x160
>>>>>>>>>       ? dwc3_gadget_ep_disable+0x3d/0xd0
>>>>>>>>>       ? usb_ep_disable+0x1c/0x
>>>>>>>>>       ? u_audio_stop_capture+0x79/0x120 [u_audio]
>>>>>>>>>       ? afunc_set_alt+0x73/0x80 [usb_f_uac2]
>> So this is triggered by a SetInterface request...
>>
>>>>>>>>>       ? composite_setup+0x224/0x1b90 [libcomposite]
>>>>>>>>>       ? __dwc3_gadget_kick_transfer+0x160/0x400
>>>>>>>>>       ? dwc3_gadget_ep_queue+0xf3/0x1a0
>>>>>>>>>       ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>>>       ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>>>       ? dwc3_ep0_interrupt+0x459/0xa40
>>>>>>>>>       ? dwc3_thread_interrupt+0x8ee/0xf40
>>>>>>>>>       ? __schedule+0x235/0x6c0
>>>>>>>>>       ? disable_irq_nosync+0x10/0x10
>>>>>>>>>       ? irq_thread_fn+0x1b/0x60
>>>>>>>>>       ? irq_thread+0xc0/0x160
>>>>>>>>>       ? irq_thread_check_affinity+0x70/0x70
>>>>>>>>>       ? irq_forced_thread_fn+0x70/0x70
>>>>>>>>>       ? kthread+0x122/0x140
>>>>>>>>>       ? set_kthread_struct+0x40/0x40
>>>>>>>>>       ? ret_from_fork+0x22/0x30
>>>>>>>> Do you mind enabling dwc3 traces and collecting them? Trying to figure
>>>>>>>> out how we got here.
>>>>>>>>
>>>>>>> I'll try if I can get the same error by booting with USB in host mode
>>>>>>> and then switch to device mode. If so I can enable traces and collect as
>>>>>>> you explained me before.
>>>>>>>
>>>>>>> I'll try before monday, as then I fly for a holiday and will not be
>>>>>>> available before rc5.
>>>>>> you can enable all of those with kernel cmdline :-)
>>>>>>
>>>>>> https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
>>>>>>
>>>>>> you need ftrace_dump_on_oops=1 and also need the correct options on
>>>>>> trace_buf_size and trace_event.
>>>>>>
>>>>> On Edison-Arduino I have a switch to go to device mode, after which
>>>>> udev triggers a script configure gadgets through configfs.
>>>>>
>>>>> I tried to log following these instructions:
>>>>>
>>>>> https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs  <https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs>
>>>>>
>>>>> Unfortunately the kernel crashes so badly I can not get to the ` cp
>>>>> /t/trace /root/trace.txt` line (after a while the watchdog kicks).
>>>>>
>>>>> What to do next?
>>>> Pass ftrace_dump_on_oops to kernel cmdline.
>>>>
>>> No sure if I did this right, on oops everything is pushed to console
>>> (115k2 serial), I hope nothing essential is lost.
>>>
>>> I copied the screen buffer to file see attached.
>> Thank you, I bet it took quite a some time :-) Anyway, looking at
>> the logs around Set Interface requests, we can track every endpoint
>> that's disabled. I'll take a guess and assume we're failing at the last
>> Set Interface, that means we should have something odd with ep6in, but
>> everything looks fine in the trace output:
>>
>> [   75.823107] irq/14-d-596       0d... 42789194us : dwc3_gadget_ep_enable: ep6in: mps 192/346 streams 16 burst 0 ring 0/0 flags E:swbp:<
>> [   75.835472] irq/14-d-596       0d... 42789198us : dwc3_alloc_request: ep6in: req 0000000002c71409 length 0/0 zsI ==> 0
>> [   75.846416] irq/14-d-596       0d... 42789202us : dwc3_ep_queue: ep6in: req 0000000002c71409 length 0/192 zsI ==> -115
>> [   75.857360] irq/14-d-596       0d... 42789204us : dwc3_alloc_request: ep6in: req 00000000a324f5d0 length 0/0 zsI ==> 0
>> [   75.868301] irq/14-d-596       0d... 42789206us : dwc3_ep_queue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -115
>> [   75.879244] irq/14-d-596       0d... 42789209us : dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0] (Not Active) [Status Phase]
>> [   75.891880] irq/14-d-596       0d... 42789211us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>> [   75.989131] irq/14-d-596       0d... 42789224us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>> [   76.096261] irq/14-d-596       0d... 42789272us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
>> [   76.107834] irq/14-d-596       0d... 42789275us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>> [   76.122944] irq/14-d-596       0d... 42789277us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 zsI ==> 0
>> [   76.134160] irq/14-d-596       0d... 42789280us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>> [   76.231322] irq/14-d-596       0d... 42789292us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>> [   76.297418] kworker/-23        0d... 42789670us : dwc3_ep_queue: ep3in: req 0000000029586135 length 0/96 ZsI ==> -115
>> [   76.308278] kworker/-23        0d... 42789695us : dwc3_prepare_trb: ep3in: trb 00000000b81213d6 (E1:D0) buf 0000000003b7a800 size 96 ctrl 00000811 (Hlcs:sC:normal)
>> [   76.395294] kworker/-23        0d... 42789707us : dwc3_gadget_ep_cmd: ep3in: cmd 'Update Transfer' [60007] params 00000000 00000000 00000000 --> status: Successful
>> [   76.471900] irq/14-d-596       0d... 42789842us : dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
>> [   76.489308] irq/14-d-596       0d... 42789845us : dwc3_ctrl_req: Set Interface(Intf = 5, Alt.Setting = 0)
>> [   76.505650] irq/14-d-596       0d... 42789851us : dwc3_ep_dequeue: ep6in: req 0000000002c71409 length 0/192 zsI ==> -115
>> [   76.523315] irq/14-d-596       0d... 42789854us : dwc3_gadget_giveback: ep6in: req 0000000002c71409 length 0/192 zsI ==> -104
>> [   76.541427] irq/14-d-596       0d... 42789857us : dwc3_free_request: ep6in: req 0000000002c71409 length 0/192 zsI ==> -104
>> [   76.559267] irq/14-d-596       0d... 42789859us : dwc3_ep_dequeue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -115
>> [   76.576937] irq/14-d-596       0d... 42789861us : dwc3_gadget_giveback: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -104
>> [   76.595046] irq/14-d-596       0d... 42789862us : dwc3_free_request: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -104
>> [   76.612892] irq/14-d-596       0d... 42789865us : dwc3_gadget_ep_disable: ep6in: mps 192/346 streams 16 burst 0 ring 0/0 flags E:swbp:<
>> [   76.665535] irq/14-d-596       0d... 42789873us : dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0] (Not Active) [Status Phase]
>> [   76.684716] irq/14-d-596       0d... 42789875us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>> [   76.819195] irq/14-d-596       0d... 42789886us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>> [   76.926324] irq/14-d-596       0d... 42789930us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
>> [   76.937892] irq/14-d-596       0d... 42789933us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>> [   76.953003] irq/14-d-596       0d... 42789935us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 zsI ==> 0
>> [   76.964217] irq/14-d-596       0d... 42789938us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>> [   77.061379] irq/14-d-596       0d... 42789950us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>> [   77.168595] irq/14-d-596       0d... 42790509us : dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
>> [   77.180159] irq/14-d-596       0d... 42790512us : dwc3_ctrl_req: Get String Descriptor(Index = 18, Length = 255)
>> [   77.190578] irq/14-d-596       0d... 42790537us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 0000000003b68000 size 36 ctrl 00000c53 (HLcs:SC:data)
>> [   77.287648] irq/14-d-596       0d... 42790550us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>> [   77.333107] irq/14-d-596       0d... 42790557us : dwc3_event: event (000010c2): ep0in: Transfer Not Ready [0] (Not Active) [Data Phase]
>> [   77.407223] irq/14-d-596       0d... 42790575us : dwc3_event: event (000090c2): ep0in: Transfer Not Ready [0] (Active) [Data Phase]
>> [   77.480985] irq/14-d-596       0d... 42790588us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Data Phase]
>> [   77.492376] irq/14-d-596       0d... 42790590us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 0000000003b68000 size 0 ctrl 00000c52 (hLcs:SC:data)
>> [   77.507221] irq/14-d-596       0d... 42790595us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 36/36 ZsI ==> 0
>> [   77.518609] irq/14-d-596       0d... 42790597us : dwc3_event: event (000020c0): ep0out: Transfer Not Ready [0] (Not Active) [Status Phase]
>> [   77.531332] irq/14-d-596       0d... 42790598us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c43 (HLcs:SC:status3)
>> [   77.628669] irq/14-d-596       0d... 42790609us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>
>> Do you mind adding a few prints in dwc3_remove_requests to tell us which
>> endpoint is being processed? Then we'll know for sure which one caused
>> the crash.
>>
> I wouldn't mind but am leaving on a holiday, won't have time until 6 aug.

not a problem, we'll still be here when you're back :-)

> But as I am using configfs (excerpt follows) and just disabling the
> last 2 line resolves the issue, I'm guessing uac2 is the issue. Or
> exceeding the available resources.
>
> # Create directory structure
> mkdir "${GADGET_BASE_DIR}"
> cd "${GADGET_BASE_DIR}"
> mkdir -p configs/c.1/strings/0x409
> mkdir -p strings/0x409
>
> # Serial device
> mkdir functions/gser.usb0
> ln -s functions/gser.usb0 configs/c.1/
> ###
>
> # Ethernet device
> mkdir functions/eem.usb0
> echo "${DEV_ETH_ADDR}" > functions/eem.usb0/dev_addr
> echo "${HOST_ETH_ADDR}" > functions/eem.usb0/host_addr
> ln -s functions/eem.usb0 configs/c.1/
>
> # Mass Storage device
> mkdir functions/mass_storage.usb0
> echo 1 > functions/mass_storage.usb0/stall
> echo 0 > functions/mass_storage.usb0/lun.0/cdrom
> echo 0 > functions/mass_storage.usb0/lun.0/ro
> echo 0 > functions/mass_storage.usb0/lun.0/nofua
> echo "${USBDISK}" > functions/mass_storage.usb0/lun.0/file
> ln -s functions/mass_storage.usb0 configs/c.1/
>
> # UAC2 device
> mkdir functions/uac2.usb0
> ln -s functions/uac2.usb0 configs/c.1

Right, either there's an actual bug in uac2, or we're running out of
FIFO space.

-- 
balbi

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-07-26  5:57                               ` Felipe Balbi
@ 2021-07-26 14:33                                 ` Wesley Cheng
  2021-07-26 21:51                                   ` Ferry Toth
  2021-08-15 20:56                                 ` Ferry Toth
  1 sibling, 1 reply; 73+ messages in thread
From: Wesley Cheng @ 2021-07-26 14:33 UTC (permalink / raw)
  To: Felipe Balbi, Ferry Toth
  Cc: gregkh, robh+dt, agross, bjorn.andersson, frowand.list,
	linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	heikki.krogerus, andy.shevchenko

Hi Ferry,

On 7/25/2021 10:57 PM, Felipe Balbi wrote:
> 
> Hi,
> 
> Ferry Toth <fntoth@gmail.com> writes:
>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>>> Hardware name: Intel Corporation Merrifield/BODEGA BAY, BIOS 542
>>>>>>>>>> 2015.01.21:18.19.48
>>>>>>>>>> RIP: 0010:0x500000000
>>>>>>>>>> Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
>>>>>>>>>> RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
>>>>>>>>>> RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX: 0000000000000000
>>>>>>>>>> RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI: ffff8cd546aed200
>>>>>>>>>> RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09: 0000000000000001
>>>>>>>>>> R10: ffff8cd541fd28c0 R11: 0000000000000000 R12: ffff8cd547342828
>>>>>>>>>> R13: ffff8cd546aed248 R14: 0000000000000000 R15: ffff8cd548b1d000
>>>>>>>>>> FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000) knlGS:0000000000000000
>>>>>>>>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>>>>>> CR2: 0000000500000000 CR3: 000000000311e000 CR4: 00000000001006f0
>>>>>>>>>> Call Trace:
>>>>>>>>>>       ? dwc3_remove_requests.constprop.0+0x14d/0x170
>>>>>>>>>>       ? __dwc3_gadget_ep_disable+0x7a/0x160
>>>>>>>>>>       ? dwc3_gadget_ep_disable+0x3d/0xd0
>>>>>>>>>>       ? usb_ep_disable+0x1c/0x
>>>>>>>>>>       ? u_audio_stop_capture+0x79/0x120 [u_audio]
>>>>>>>>>>       ? afunc_set_alt+0x73/0x80 [usb_f_uac2]
>>> So this is triggered by a SetInterface request...
>>>
>>>>>>>>>>       ? composite_setup+0x224/0x1b90 [libcomposite]
>>>>>>>>>>       ? __dwc3_gadget_kick_transfer+0x160/0x400
>>>>>>>>>>       ? dwc3_gadget_ep_queue+0xf3/0x1a0
>>>>>>>>>>       ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>>>>       ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>>>>       ? dwc3_ep0_interrupt+0x459/0xa40
>>>>>>>>>>       ? dwc3_thread_interrupt+0x8ee/0xf40
>>>>>>>>>>       ? __schedule+0x235/0x6c0
>>>>>>>>>>       ? disable_irq_nosync+0x10/0x10
>>>>>>>>>>       ? irq_thread_fn+0x1b/0x60
>>>>>>>>>>       ? irq_thread+0xc0/0x160
>>>>>>>>>>       ? irq_thread_check_affinity+0x70/0x70
>>>>>>>>>>       ? irq_forced_thread_fn+0x70/0x70
>>>>>>>>>>       ? kthread+0x122/0x140
>>>>>>>>>>       ? set_kthread_struct+0x40/0x40
>>>>>>>>>>       ? ret_from_fork+0x22/0x30
>>>>>>>>> Do you mind enabling dwc3 traces and collecting them? Trying to figure
>>>>>>>>> out how we got here.
>>>>>>>>>
>>>>>>>> I'll try if I can get the same error by booting with USB in host mode
>>>>>>>> and then switch to device mode. If so I can enable traces and collect as
>>>>>>>> you explained me before.
>>>>>>>>
>>>>>>>> I'll try before monday, as then I fly for a holiday and will not be
>>>>>>>> available before rc5.
>>>>>>> you can enable all of those with kernel cmdline :-)
>>>>>>>
>>>>>>> https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
>>>>>>>
>>>>>>> you need ftrace_dump_on_oops=1 and also need the correct options on
>>>>>>> trace_buf_size and trace_event.
>>>>>>>
>>>>>> On Edison-Arduino I have a switch to go to device mode, after which
>>>>>> udev triggers a script configure gadgets through configfs.
>>>>>>
>>>>>> I tried to log following these instructions:
>>>>>>
>>>>>> https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs  <https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs>
>>>>>>
>>>>>> Unfortunately the kernel crashes so badly I can not get to the ` cp
>>>>>> /t/trace /root/trace.txt` line (after a while the watchdog kicks).
>>>>>>
>>>>>> What to do next?
>>>>> Pass ftrace_dump_on_oops to kernel cmdline.
>>>>>
>>>> No sure if I did this right, on oops everything is pushed to console
>>>> (115k2 serial), I hope nothing essential is lost.
>>>>
>>>> I copied the screen buffer to file see attached.
>>> Thank you, I bet it took quite a some time :-) Anyway, looking at
>>> the logs around Set Interface requests, we can track every endpoint
>>> that's disabled. I'll take a guess and assume we're failing at the last
>>> Set Interface, that means we should have something odd with ep6in, but
>>> everything looks fine in the trace output:
>>>
>>> [   75.823107] irq/14-d-596       0d... 42789194us : dwc3_gadget_ep_enable: ep6in: mps 192/346 streams 16 burst 0 ring 0/0 flags E:swbp:<
>>> [   75.835472] irq/14-d-596       0d... 42789198us : dwc3_alloc_request: ep6in: req 0000000002c71409 length 0/0 zsI ==> 0
>>> [   75.846416] irq/14-d-596       0d... 42789202us : dwc3_ep_queue: ep6in: req 0000000002c71409 length 0/192 zsI ==> -115
>>> [   75.857360] irq/14-d-596       0d... 42789204us : dwc3_alloc_request: ep6in: req 00000000a324f5d0 length 0/0 zsI ==> 0
>>> [   75.868301] irq/14-d-596       0d... 42789206us : dwc3_ep_queue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -115
>>> [   75.879244] irq/14-d-596       0d... 42789209us : dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0] (Not Active) [Status Phase]
>>> [   75.891880] irq/14-d-596       0d... 42789211us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>> [   75.989131] irq/14-d-596       0d... 42789224us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>> [   76.096261] irq/14-d-596       0d... 42789272us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
>>> [   76.107834] irq/14-d-596       0d... 42789275us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>> [   76.122944] irq/14-d-596       0d... 42789277us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 zsI ==> 0
>>> [   76.134160] irq/14-d-596       0d... 42789280us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>> [   76.231322] irq/14-d-596       0d... 42789292us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>> [   76.297418] kworker/-23        0d... 42789670us : dwc3_ep_queue: ep3in: req 0000000029586135 length 0/96 ZsI ==> -115
>>> [   76.308278] kworker/-23        0d... 42789695us : dwc3_prepare_trb: ep3in: trb 00000000b81213d6 (E1:D0) buf 0000000003b7a800 size 96 ctrl 00000811 (Hlcs:sC:normal)
>>> [   76.395294] kworker/-23        0d... 42789707us : dwc3_gadget_ep_cmd: ep3in: cmd 'Update Transfer' [60007] params 00000000 00000000 00000000 --> status: Successful
>>> [   76.471900] irq/14-d-596       0d... 42789842us : dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
>>> [   76.489308] irq/14-d-596       0d... 42789845us : dwc3_ctrl_req: Set Interface(Intf = 5, Alt.Setting = 0)
>>> [   76.505650] irq/14-d-596       0d... 42789851us : dwc3_ep_dequeue: ep6in: req 0000000002c71409 length 0/192 zsI ==> -115
>>> [   76.523315] irq/14-d-596       0d... 42789854us : dwc3_gadget_giveback: ep6in: req 0000000002c71409 length 0/192 zsI ==> -104
>>> [   76.541427] irq/14-d-596       0d... 42789857us : dwc3_free_request: ep6in: req 0000000002c71409 length 0/192 zsI ==> -104
>>> [   76.559267] irq/14-d-596       0d... 42789859us : dwc3_ep_dequeue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -115
>>> [   76.576937] irq/14-d-596       0d... 42789861us : dwc3_gadget_giveback: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -104
>>> [   76.595046] irq/14-d-596       0d... 42789862us : dwc3_free_request: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -104
>>> [   76.612892] irq/14-d-596       0d... 42789865us : dwc3_gadget_ep_disable: ep6in: mps 192/346 streams 16 burst 0 ring 0/0 flags E:swbp:<
>>> [   76.665535] irq/14-d-596       0d... 42789873us : dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0] (Not Active) [Status Phase]
>>> [   76.684716] irq/14-d-596       0d... 42789875us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>> [   76.819195] irq/14-d-596       0d... 42789886us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>> [   76.926324] irq/14-d-596       0d... 42789930us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
>>> [   76.937892] irq/14-d-596       0d... 42789933us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>> [   76.953003] irq/14-d-596       0d... 42789935us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 zsI ==> 0
>>> [   76.964217] irq/14-d-596       0d... 42789938us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>> [   77.061379] irq/14-d-596       0d... 42789950us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>> [   77.168595] irq/14-d-596       0d... 42790509us : dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
>>> [   77.180159] irq/14-d-596       0d... 42790512us : dwc3_ctrl_req: Get String Descriptor(Index = 18, Length = 255)
>>> [   77.190578] irq/14-d-596       0d... 42790537us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 0000000003b68000 size 36 ctrl 00000c53 (HLcs:SC:data)
>>> [   77.287648] irq/14-d-596       0d... 42790550us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>> [   77.333107] irq/14-d-596       0d... 42790557us : dwc3_event: event (000010c2): ep0in: Transfer Not Ready [0] (Not Active) [Data Phase]
>>> [   77.407223] irq/14-d-596       0d... 42790575us : dwc3_event: event (000090c2): ep0in: Transfer Not Ready [0] (Active) [Data Phase]
>>> [   77.480985] irq/14-d-596       0d... 42790588us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Data Phase]
>>> [   77.492376] irq/14-d-596       0d... 42790590us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 0000000003b68000 size 0 ctrl 00000c52 (hLcs:SC:data)
>>> [   77.507221] irq/14-d-596       0d... 42790595us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 36/36 ZsI ==> 0
>>> [   77.518609] irq/14-d-596       0d... 42790597us : dwc3_event: event (000020c0): ep0out: Transfer Not Ready [0] (Not Active) [Status Phase]
>>> [   77.531332] irq/14-d-596       0d... 42790598us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c43 (HLcs:SC:status3)
>>> [   77.628669] irq/14-d-596       0d... 42790609us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>>
>>> Do you mind adding a few prints in dwc3_remove_requests to tell us which
>>> endpoint is being processed? Then we'll know for sure which one caused
>>> the crash.
>>>
>> I wouldn't mind but am leaving on a holiday, won't have time until 6 aug.
> 
> not a problem, we'll still be here when you're back :-)
> 
>> But as I am using configfs (excerpt follows) and just disabling the
>> last 2 line resolves the issue, I'm guessing uac2 is the issue. Or
>> exceeding the available resources.
>>
>> # Create directory structure
>> mkdir "${GADGET_BASE_DIR}"
>> cd "${GADGET_BASE_DIR}"
>> mkdir -p configs/c.1/strings/0x409
>> mkdir -p strings/0x409
>>
>> # Serial device
>> mkdir functions/gser.usb0
>> ln -s functions/gser.usb0 configs/c.1/
>> ###
>>
>> # Ethernet device
>> mkdir functions/eem.usb0
>> echo "${DEV_ETH_ADDR}" > functions/eem.usb0/dev_addr
>> echo "${HOST_ETH_ADDR}" > functions/eem.usb0/host_addr
>> ln -s functions/eem.usb0 configs/c.1/
>>
>> # Mass Storage device
>> mkdir functions/mass_storage.usb0
>> echo 1 > functions/mass_storage.usb0/stall
>> echo 0 > functions/mass_storage.usb0/lun.0/cdrom
>> echo 0 > functions/mass_storage.usb0/lun.0/ro
>> echo 0 > functions/mass_storage.usb0/lun.0/nofua
>> echo "${USBDISK}" > functions/mass_storage.usb0/lun.0/file
>> ln -s functions/mass_storage.usb0 configs/c.1/
>>
>> # UAC2 device
>> mkdir functions/uac2.usb0
>> ln -s functions/uac2.usb0 configs/c.1
> 
> Right, either there's an actual bug in uac2, or we're running out of
> FIFO space.
> 

Are you enabling the TXFIFO flag here again?  If we suspect that we're
running out of FIFO space, then:
- First, making sure we're explicitly enabling the TXFIFO resize flag.
- Second, we should see the same warning we saw previously:
	dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
depth:115540359


Thanks
Wesley Cheng

-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-07-26 14:33                                 ` Wesley Cheng
@ 2021-07-26 21:51                                   ` Ferry Toth
  0 siblings, 0 replies; 73+ messages in thread
From: Ferry Toth @ 2021-07-26 21:51 UTC (permalink / raw)
  To: Wesley Cheng, Felipe Balbi
  Cc: gregkh, robh+dt, agross, bjorn.andersson, frowand.list,
	linux-usb, linux-kernel, devicetree, linux-arm-msm, jackp,
	heikki.krogerus, andy.shevchenko

Hi

Op 26-07-2021 om 16:33 schreef Wesley Cheng:
> Hi Ferry,
> 
> On 7/25/2021 10:57 PM, Felipe Balbi wrote:
>>
>> Hi,
>>
>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>>>> Hardware name: Intel Corporation Merrifield/BODEGA BAY, BIOS 542
>>>>>>>>>>> 2015.01.21:18.19.48
>>>>>>>>>>> RIP: 0010:0x500000000
>>>>>>>>>>> Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
>>>>>>>>>>> RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
>>>>>>>>>>> RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX: 0000000000000000
>>>>>>>>>>> RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI: ffff8cd546aed200
>>>>>>>>>>> RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09: 0000000000000001
>>>>>>>>>>> R10: ffff8cd541fd28c0 R11: 0000000000000000 R12: ffff8cd547342828
>>>>>>>>>>> R13: ffff8cd546aed248 R14: 0000000000000000 R15: ffff8cd548b1d000
>>>>>>>>>>> FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000) knlGS:0000000000000000
>>>>>>>>>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>>>>>>> CR2: 0000000500000000 CR3: 000000000311e000 CR4: 00000000001006f0
>>>>>>>>>>> Call Trace:
>>>>>>>>>>>        ? dwc3_remove_requests.constprop.0+0x14d/0x170
>>>>>>>>>>>        ? __dwc3_gadget_ep_disable+0x7a/0x160
>>>>>>>>>>>        ? dwc3_gadget_ep_disable+0x3d/0xd0
>>>>>>>>>>>        ? usb_ep_disable+0x1c/0x
>>>>>>>>>>>        ? u_audio_stop_capture+0x79/0x120 [u_audio]
>>>>>>>>>>>        ? afunc_set_alt+0x73/0x80 [usb_f_uac2]
>>>> So this is triggered by a SetInterface request...
>>>>
>>>>>>>>>>>        ? composite_setup+0x224/0x1b90 [libcomposite]
>>>>>>>>>>>        ? __dwc3_gadget_kick_transfer+0x160/0x400
>>>>>>>>>>>        ? dwc3_gadget_ep_queue+0xf3/0x1a0
>>>>>>>>>>>        ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>>>>>        ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>>>>>        ? dwc3_ep0_interrupt+0x459/0xa40
>>>>>>>>>>>        ? dwc3_thread_interrupt+0x8ee/0xf40
>>>>>>>>>>>        ? __schedule+0x235/0x6c0
>>>>>>>>>>>        ? disable_irq_nosync+0x10/0x10
>>>>>>>>>>>        ? irq_thread_fn+0x1b/0x60
>>>>>>>>>>>        ? irq_thread+0xc0/0x160
>>>>>>>>>>>        ? irq_thread_check_affinity+0x70/0x70
>>>>>>>>>>>        ? irq_forced_thread_fn+0x70/0x70
>>>>>>>>>>>        ? kthread+0x122/0x140
>>>>>>>>>>>        ? set_kthread_struct+0x40/0x40
>>>>>>>>>>>        ? ret_from_fork+0x22/0x30
>>>>>>>>>> Do you mind enabling dwc3 traces and collecting them? Trying to figure
>>>>>>>>>> out how we got here.
>>>>>>>>>>
>>>>>>>>> I'll try if I can get the same error by booting with USB in host mode
>>>>>>>>> and then switch to device mode. If so I can enable traces and collect as
>>>>>>>>> you explained me before.
>>>>>>>>>
>>>>>>>>> I'll try before monday, as then I fly for a holiday and will not be
>>>>>>>>> available before rc5.
>>>>>>>> you can enable all of those with kernel cmdline :-)
>>>>>>>>
>>>>>>>> https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
>>>>>>>>
>>>>>>>> you need ftrace_dump_on_oops=1 and also need the correct options on
>>>>>>>> trace_buf_size and trace_event.
>>>>>>>>
>>>>>>> On Edison-Arduino I have a switch to go to device mode, after which
>>>>>>> udev triggers a script configure gadgets through configfs.
>>>>>>>
>>>>>>> I tried to log following these instructions:
>>>>>>>
>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs  <https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs>
>>>>>>>
>>>>>>> Unfortunately the kernel crashes so badly I can not get to the ` cp
>>>>>>> /t/trace /root/trace.txt` line (after a while the watchdog kicks).
>>>>>>>
>>>>>>> What to do next?
>>>>>> Pass ftrace_dump_on_oops to kernel cmdline.
>>>>>>
>>>>> No sure if I did this right, on oops everything is pushed to console
>>>>> (115k2 serial), I hope nothing essential is lost.
>>>>>
>>>>> I copied the screen buffer to file see attached.
>>>> Thank you, I bet it took quite a some time :-) Anyway, looking at
>>>> the logs around Set Interface requests, we can track every endpoint
>>>> that's disabled. I'll take a guess and assume we're failing at the last
>>>> Set Interface, that means we should have something odd with ep6in, but
>>>> everything looks fine in the trace output:
>>>>
>>>> [   75.823107] irq/14-d-596       0d... 42789194us : dwc3_gadget_ep_enable: ep6in: mps 192/346 streams 16 burst 0 ring 0/0 flags E:swbp:<
>>>> [   75.835472] irq/14-d-596       0d... 42789198us : dwc3_alloc_request: ep6in: req 0000000002c71409 length 0/0 zsI ==> 0
>>>> [   75.846416] irq/14-d-596       0d... 42789202us : dwc3_ep_queue: ep6in: req 0000000002c71409 length 0/192 zsI ==> -115
>>>> [   75.857360] irq/14-d-596       0d... 42789204us : dwc3_alloc_request: ep6in: req 00000000a324f5d0 length 0/0 zsI ==> 0
>>>> [   75.868301] irq/14-d-596       0d... 42789206us : dwc3_ep_queue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -115
>>>> [   75.879244] irq/14-d-596       0d... 42789209us : dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0] (Not Active) [Status Phase]
>>>> [   75.891880] irq/14-d-596       0d... 42789211us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>> [   75.989131] irq/14-d-596       0d... 42789224us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>>> [   76.096261] irq/14-d-596       0d... 42789272us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
>>>> [   76.107834] irq/14-d-596       0d... 42789275us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>> [   76.122944] irq/14-d-596       0d... 42789277us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 zsI ==> 0
>>>> [   76.134160] irq/14-d-596       0d... 42789280us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>> [   76.231322] irq/14-d-596       0d... 42789292us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>>> [   76.297418] kworker/-23        0d... 42789670us : dwc3_ep_queue: ep3in: req 0000000029586135 length 0/96 ZsI ==> -115
>>>> [   76.308278] kworker/-23        0d... 42789695us : dwc3_prepare_trb: ep3in: trb 00000000b81213d6 (E1:D0) buf 0000000003b7a800 size 96 ctrl 00000811 (Hlcs:sC:normal)
>>>> [   76.395294] kworker/-23        0d... 42789707us : dwc3_gadget_ep_cmd: ep3in: cmd 'Update Transfer' [60007] params 00000000 00000000 00000000 --> status: Successful
>>>> [   76.471900] irq/14-d-596       0d... 42789842us : dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
>>>> [   76.489308] irq/14-d-596       0d... 42789845us : dwc3_ctrl_req: Set Interface(Intf = 5, Alt.Setting = 0)
>>>> [   76.505650] irq/14-d-596       0d... 42789851us : dwc3_ep_dequeue: ep6in: req 0000000002c71409 length 0/192 zsI ==> -115
>>>> [   76.523315] irq/14-d-596       0d... 42789854us : dwc3_gadget_giveback: ep6in: req 0000000002c71409 length 0/192 zsI ==> -104
>>>> [   76.541427] irq/14-d-596       0d... 42789857us : dwc3_free_request: ep6in: req 0000000002c71409 length 0/192 zsI ==> -104
>>>> [   76.559267] irq/14-d-596       0d... 42789859us : dwc3_ep_dequeue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -115
>>>> [   76.576937] irq/14-d-596       0d... 42789861us : dwc3_gadget_giveback: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -104
>>>> [   76.595046] irq/14-d-596       0d... 42789862us : dwc3_free_request: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -104
>>>> [   76.612892] irq/14-d-596       0d... 42789865us : dwc3_gadget_ep_disable: ep6in: mps 192/346 streams 16 burst 0 ring 0/0 flags E:swbp:<
>>>> [   76.665535] irq/14-d-596       0d... 42789873us : dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0] (Not Active) [Status Phase]
>>>> [   76.684716] irq/14-d-596       0d... 42789875us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>> [   76.819195] irq/14-d-596       0d... 42789886us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>>> [   76.926324] irq/14-d-596       0d... 42789930us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
>>>> [   76.937892] irq/14-d-596       0d... 42789933us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>> [   76.953003] irq/14-d-596       0d... 42789935us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 zsI ==> 0
>>>> [   76.964217] irq/14-d-596       0d... 42789938us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>> [   77.061379] irq/14-d-596       0d... 42789950us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>>> [   77.168595] irq/14-d-596       0d... 42790509us : dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
>>>> [   77.180159] irq/14-d-596       0d... 42790512us : dwc3_ctrl_req: Get String Descriptor(Index = 18, Length = 255)
>>>> [   77.190578] irq/14-d-596       0d... 42790537us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 0000000003b68000 size 36 ctrl 00000c53 (HLcs:SC:data)
>>>> [   77.287648] irq/14-d-596       0d... 42790550us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>>> [   77.333107] irq/14-d-596       0d... 42790557us : dwc3_event: event (000010c2): ep0in: Transfer Not Ready [0] (Not Active) [Data Phase]
>>>> [   77.407223] irq/14-d-596       0d... 42790575us : dwc3_event: event (000090c2): ep0in: Transfer Not Ready [0] (Active) [Data Phase]
>>>> [   77.480985] irq/14-d-596       0d... 42790588us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Data Phase]
>>>> [   77.492376] irq/14-d-596       0d... 42790590us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 0000000003b68000 size 0 ctrl 00000c52 (hLcs:SC:data)
>>>> [   77.507221] irq/14-d-596       0d... 42790595us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 36/36 ZsI ==> 0
>>>> [   77.518609] irq/14-d-596       0d... 42790597us : dwc3_event: event (000020c0): ep0out: Transfer Not Ready [0] (Not Active) [Status Phase]
>>>> [   77.531332] irq/14-d-596       0d... 42790598us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c43 (HLcs:SC:status3)
>>>> [   77.628669] irq/14-d-596       0d... 42790609us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>>>
>>>> Do you mind adding a few prints in dwc3_remove_requests to tell us which
>>>> endpoint is being processed? Then we'll know for sure which one caused
>>>> the crash.
>>>>
>>> I wouldn't mind but am leaving on a holiday, won't have time until 6 aug.
>>
>> not a problem, we'll still be here when you're back :-)
>>
>>> But as I am using configfs (excerpt follows) and just disabling the
>>> last 2 line resolves the issue, I'm guessing uac2 is the issue. Or
>>> exceeding the available resources.
>>>
>>> # Create directory structure
>>> mkdir "${GADGET_BASE_DIR}"
>>> cd "${GADGET_BASE_DIR}"
>>> mkdir -p configs/c.1/strings/0x409
>>> mkdir -p strings/0x409
>>>
>>> # Serial device
>>> mkdir functions/gser.usb0
>>> ln -s functions/gser.usb0 configs/c.1/
>>> ###
>>>
>>> # Ethernet device
>>> mkdir functions/eem.usb0
>>> echo "${DEV_ETH_ADDR}" > functions/eem.usb0/dev_addr
>>> echo "${HOST_ETH_ADDR}" > functions/eem.usb0/host_addr
>>> ln -s functions/eem.usb0 configs/c.1/
>>>
>>> # Mass Storage device
>>> mkdir functions/mass_storage.usb0
>>> echo 1 > functions/mass_storage.usb0/stall
>>> echo 0 > functions/mass_storage.usb0/lun.0/cdrom
>>> echo 0 > functions/mass_storage.usb0/lun.0/ro
>>> echo 0 > functions/mass_storage.usb0/lun.0/nofua
>>> echo "${USBDISK}" > functions/mass_storage.usb0/lun.0/file
>>> ln -s functions/mass_storage.usb0 configs/c.1/
>>>
>>> # UAC2 device
>>> mkdir functions/uac2.usb0
>>> ln -s functions/uac2.usb0 configs/c.1
>>
>> Right, either there's an actual bug in uac2, or we're running out of
>> FIFO space.
>>
> 
> Are you enabling the TXFIFO flag here again?  If we suspect that we're
> running out of FIFO space, then:
> - First, making sure we're explicitly enabling the TXFIFO resize flag.
> - Second, we should see the same warning we saw previously:
> 	dwc3 dwc3.0.auto: Fifosize(2154) > RAM size(2022) ep5in
> depth:115540359

No, I'm building using a Yocto recipe:
https://github.com/htot/meta-intel-edison/blob/gatesgarth/meta-intel-edison-bsp/recipes-kernel/linux/linux-yocto_5.14-rc2.bb

patches are line #64 and on, just one old dwc3 related (edison) patch.

> 
> Thanks
> Wesley Cheng
> 

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-07-26  5:57                               ` Felipe Balbi
  2021-07-26 14:33                                 ` Wesley Cheng
@ 2021-08-15 20:56                                 ` Ferry Toth
  2021-08-16  5:18                                   ` Felipe Balbi
  1 sibling, 1 reply; 73+ messages in thread
From: Ferry Toth @ 2021-08-15 20:56 UTC (permalink / raw)
  To: Felipe Balbi
  Cc: Wesley Cheng, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list, linux-usb, linux-kernel, devicetree, linux-arm-msm,
	jackp, heikki.krogerus, andy.shevchenko

Hi,

Op 26-07-2021 om 07:57 schreef Felipe Balbi:
> Hi,
>
> Ferry Toth <fntoth@gmail.com> writes:
>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>>> Hardware name: Intel Corporation Merrifield/BODEGA BAY, BIOS 542
>>>>>>>>>> 2015.01.21:18.19.48
>>>>>>>>>> RIP: 0010:0x500000000
>>>>>>>>>> Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
>>>>>>>>>> RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
>>>>>>>>>> RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX: 0000000000000000
>>>>>>>>>> RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI: ffff8cd546aed200
>>>>>>>>>> RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09: 0000000000000001
>>>>>>>>>> R10: ffff8cd541fd28c0 R11: 0000000000000000 R12: ffff8cd547342828
>>>>>>>>>> R13: ffff8cd546aed248 R14: 0000000000000000 R15: ffff8cd548b1d000
>>>>>>>>>> FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000) knlGS:0000000000000000
>>>>>>>>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>>>>>> CR2: 0000000500000000 CR3: 000000000311e000 CR4: 00000000001006f0
>>>>>>>>>> Call Trace:
>>>>>>>>>>        ? dwc3_remove_requests.constprop.0+0x14d/0x170
>>>>>>>>>>        ? __dwc3_gadget_ep_disable+0x7a/0x160
>>>>>>>>>>        ? dwc3_gadget_ep_disable+0x3d/0xd0
>>>>>>>>>>        ? usb_ep_disable+0x1c/0x
>>>>>>>>>>        ? u_audio_stop_capture+0x79/0x120 [u_audio]
>>>>>>>>>>        ? afunc_set_alt+0x73/0x80 [usb_f_uac2]
>>> So this is triggered by a SetInterface request...
>>>
>>>>>>>>>>        ? composite_setup+0x224/0x1b90 [libcomposite]
>>>>>>>>>>        ? __dwc3_gadget_kick_transfer+0x160/0x400
>>>>>>>>>>        ? dwc3_gadget_ep_queue+0xf3/0x1a0
>>>>>>>>>>        ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>>>>        ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>>>>        ? dwc3_ep0_interrupt+0x459/0xa40
>>>>>>>>>>        ? dwc3_thread_interrupt+0x8ee/0xf40
>>>>>>>>>>        ? __schedule+0x235/0x6c0
>>>>>>>>>>        ? disable_irq_nosync+0x10/0x10
>>>>>>>>>>        ? irq_thread_fn+0x1b/0x60
>>>>>>>>>>        ? irq_thread+0xc0/0x160
>>>>>>>>>>        ? irq_thread_check_affinity+0x70/0x70
>>>>>>>>>>        ? irq_forced_thread_fn+0x70/0x70
>>>>>>>>>>        ? kthread+0x122/0x140
>>>>>>>>>>        ? set_kthread_struct+0x40/0x40
>>>>>>>>>>        ? ret_from_fork+0x22/0x30
>>>>>>>>> Do you mind enabling dwc3 traces and collecting them? Trying to figure
>>>>>>>>> out how we got here.
>>>>>>>>>
>>>>>>>> I'll try if I can get the same error by booting with USB in host mode
>>>>>>>> and then switch to device mode. If so I can enable traces and collect as
>>>>>>>> you explained me before.
>>>>>>>>
>>>>>>>> I'll try before monday, as then I fly for a holiday and will not be
>>>>>>>> available before rc5.
>>>>>>> you can enable all of those with kernel cmdline :-)
>>>>>>>
>>>>>>> https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
>>>>>>>
>>>>>>> you need ftrace_dump_on_oops=1 and also need the correct options on
>>>>>>> trace_buf_size and trace_event.
>>>>>>>
>>>>>> On Edison-Arduino I have a switch to go to device mode, after which
>>>>>> udev triggers a script configure gadgets through configfs.
>>>>>>
>>>>>> I tried to log following these instructions:
>>>>>>
>>>>>> https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs  <https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs>
>>>>>>
>>>>>> Unfortunately the kernel crashes so badly I can not get to the ` cp
>>>>>> /t/trace /root/trace.txt` line (after a while the watchdog kicks).
>>>>>>
>>>>>> What to do next?
>>>>> Pass ftrace_dump_on_oops to kernel cmdline.
>>>>>
>>>> No sure if I did this right, on oops everything is pushed to console
>>>> (115k2 serial), I hope nothing essential is lost.
>>>>
>>>> I copied the screen buffer to file see attached.
>>> Thank you, I bet it took quite a some time :-) Anyway, looking at
>>> the logs around Set Interface requests, we can track every endpoint
>>> that's disabled. I'll take a guess and assume we're failing at the last
>>> Set Interface, that means we should have something odd with ep6in, but
>>> everything looks fine in the trace output:
>>>
>>> [   75.823107] irq/14-d-596       0d... 42789194us : dwc3_gadget_ep_enable: ep6in: mps 192/346 streams 16 burst 0 ring 0/0 flags E:swbp:<
>>> [   75.835472] irq/14-d-596       0d... 42789198us : dwc3_alloc_request: ep6in: req 0000000002c71409 length 0/0 zsI ==> 0
>>> [   75.846416] irq/14-d-596       0d... 42789202us : dwc3_ep_queue: ep6in: req 0000000002c71409 length 0/192 zsI ==> -115
>>> [   75.857360] irq/14-d-596       0d... 42789204us : dwc3_alloc_request: ep6in: req 00000000a324f5d0 length 0/0 zsI ==> 0
>>> [   75.868301] irq/14-d-596       0d... 42789206us : dwc3_ep_queue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -115
>>> [   75.879244] irq/14-d-596       0d... 42789209us : dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0] (Not Active) [Status Phase]
>>> [   75.891880] irq/14-d-596       0d... 42789211us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>> [   75.989131] irq/14-d-596       0d... 42789224us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>> [   76.096261] irq/14-d-596       0d... 42789272us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
>>> [   76.107834] irq/14-d-596       0d... 42789275us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>> [   76.122944] irq/14-d-596       0d... 42789277us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 zsI ==> 0
>>> [   76.134160] irq/14-d-596       0d... 42789280us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>> [   76.231322] irq/14-d-596       0d... 42789292us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>> [   76.297418] kworker/-23        0d... 42789670us : dwc3_ep_queue: ep3in: req 0000000029586135 length 0/96 ZsI ==> -115
>>> [   76.308278] kworker/-23        0d... 42789695us : dwc3_prepare_trb: ep3in: trb 00000000b81213d6 (E1:D0) buf 0000000003b7a800 size 96 ctrl 00000811 (Hlcs:sC:normal)
>>> [   76.395294] kworker/-23        0d... 42789707us : dwc3_gadget_ep_cmd: ep3in: cmd 'Update Transfer' [60007] params 00000000 00000000 00000000 --> status: Successful
>>> [   76.471900] irq/14-d-596       0d... 42789842us : dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
>>> [   76.489308] irq/14-d-596       0d... 42789845us : dwc3_ctrl_req: Set Interface(Intf = 5, Alt.Setting = 0)
>>> [   76.505650] irq/14-d-596       0d... 42789851us : dwc3_ep_dequeue: ep6in: req 0000000002c71409 length 0/192 zsI ==> -115
>>> [   76.523315] irq/14-d-596       0d... 42789854us : dwc3_gadget_giveback: ep6in: req 0000000002c71409 length 0/192 zsI ==> -104
>>> [   76.541427] irq/14-d-596       0d... 42789857us : dwc3_free_request: ep6in: req 0000000002c71409 length 0/192 zsI ==> -104
>>> [   76.559267] irq/14-d-596       0d... 42789859us : dwc3_ep_dequeue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -115
>>> [   76.576937] irq/14-d-596       0d... 42789861us : dwc3_gadget_giveback: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -104
>>> [   76.595046] irq/14-d-596       0d... 42789862us : dwc3_free_request: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -104
>>> [   76.612892] irq/14-d-596       0d... 42789865us : dwc3_gadget_ep_disable: ep6in: mps 192/346 streams 16 burst 0 ring 0/0 flags E:swbp:<
>>> [   76.665535] irq/14-d-596       0d... 42789873us : dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0] (Not Active) [Status Phase]
>>> [   76.684716] irq/14-d-596       0d... 42789875us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>> [   76.819195] irq/14-d-596       0d... 42789886us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>> [   76.926324] irq/14-d-596       0d... 42789930us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
>>> [   76.937892] irq/14-d-596       0d... 42789933us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>> [   76.953003] irq/14-d-596       0d... 42789935us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 zsI ==> 0
>>> [   76.964217] irq/14-d-596       0d... 42789938us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>> [   77.061379] irq/14-d-596       0d... 42789950us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>> [   77.168595] irq/14-d-596       0d... 42790509us : dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
>>> [   77.180159] irq/14-d-596       0d... 42790512us : dwc3_ctrl_req: Get String Descriptor(Index = 18, Length = 255)
>>> [   77.190578] irq/14-d-596       0d... 42790537us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 0000000003b68000 size 36 ctrl 00000c53 (HLcs:SC:data)
>>> [   77.287648] irq/14-d-596       0d... 42790550us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>> [   77.333107] irq/14-d-596       0d... 42790557us : dwc3_event: event (000010c2): ep0in: Transfer Not Ready [0] (Not Active) [Data Phase]
>>> [   77.407223] irq/14-d-596       0d... 42790575us : dwc3_event: event (000090c2): ep0in: Transfer Not Ready [0] (Active) [Data Phase]
>>> [   77.480985] irq/14-d-596       0d... 42790588us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Data Phase]
>>> [   77.492376] irq/14-d-596       0d... 42790590us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 0000000003b68000 size 0 ctrl 00000c52 (hLcs:SC:data)
>>> [   77.507221] irq/14-d-596       0d... 42790595us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 36/36 ZsI ==> 0
>>> [   77.518609] irq/14-d-596       0d... 42790597us : dwc3_event: event (000020c0): ep0out: Transfer Not Ready [0] (Not Active) [Status Phase]
>>> [   77.531332] irq/14-d-596       0d... 42790598us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c43 (HLcs:SC:status3)
>>> [   77.628669] irq/14-d-596       0d... 42790609us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>>
>>> Do you mind adding a few prints in dwc3_remove_requests to tell us which
>>> endpoint is being processed? Then we'll know for sure which one caused
>>> the crash.
>>>
>> I wouldn't mind but am leaving on a holiday, won't have time until 6 aug.
> not a problem, we'll still be here when you're back :-)

Well, let's go then :-)

To get back in the mood I have retested 5.13.0, 5.14.0-rc1, 5.14.0-rc2 
and 5.14.0-rc5.

I find that 5.13.0 works fine, and the issue starts from 5.14.0-rc1.

With 5.14.0-rc5 the problem seems worse (or different?), and just 
disabling uac2 gadget does not prevent the crash. Even disabling gser 
and mass_storage.usb0 as well there is still a crash.

Now I'm not sure how to proceed. Bisect rc1? Or focus on rc5 (rc6?)?

>> But as I am using configfs (excerpt follows) and just disabling the
>> last 2 line resolves the issue, I'm guessing uac2 is the issue. Or
>> exceeding the available resources.
>>
>> # Create directory structure
>> mkdir "${GADGET_BASE_DIR}"
>> cd "${GADGET_BASE_DIR}"
>> mkdir -p configs/c.1/strings/0x409
>> mkdir -p strings/0x409
>>
>> # Serial device
>> mkdir functions/gser.usb0
>> ln -s functions/gser.usb0 configs/c.1/
>> ###
>>
>> # Ethernet device
>> mkdir functions/eem.usb0
>> echo "${DEV_ETH_ADDR}" > functions/eem.usb0/dev_addr
>> echo "${HOST_ETH_ADDR}" > functions/eem.usb0/host_addr
>> ln -s functions/eem.usb0 configs/c.1/
>>
>> # Mass Storage device
>> mkdir functions/mass_storage.usb0
>> echo 1 > functions/mass_storage.usb0/stall
>> echo 0 > functions/mass_storage.usb0/lun.0/cdrom
>> echo 0 > functions/mass_storage.usb0/lun.0/ro
>> echo 0 > functions/mass_storage.usb0/lun.0/nofua
>> echo "${USBDISK}" > functions/mass_storage.usb0/lun.0/file
>> ln -s functions/mass_storage.usb0 configs/c.1/
>>
>> # UAC2 device
>> mkdir functions/uac2.usb0
>> ln -s functions/uac2.usb0 configs/c.1
> Right, either there's an actual bug in uac2, or we're running out of
> FIFO space.
>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-08-15 20:56                                 ` Ferry Toth
@ 2021-08-16  5:18                                   ` Felipe Balbi
  2021-08-17 22:00                                     ` Ferry Toth
  0 siblings, 1 reply; 73+ messages in thread
From: Felipe Balbi @ 2021-08-16  5:18 UTC (permalink / raw)
  To: Ferry Toth
  Cc: Wesley Cheng, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list, linux-usb, linux-kernel, devicetree, linux-arm-msm,
	jackp, heikki.krogerus, andy.shevchenko


Hi,

Ferry Toth <fntoth@gmail.com> writes:
>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>>>> Hardware name: Intel Corporation Merrifield/BODEGA BAY, BIOS 542
>>>>>>>>>>> 2015.01.21:18.19.48
>>>>>>>>>>> RIP: 0010:0x500000000
>>>>>>>>>>> Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
>>>>>>>>>>> RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
>>>>>>>>>>> RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX: 0000000000000000
>>>>>>>>>>> RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI: ffff8cd546aed200
>>>>>>>>>>> RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09: 0000000000000001
>>>>>>>>>>> R10: ffff8cd541fd28c0 R11: 0000000000000000 R12: ffff8cd547342828
>>>>>>>>>>> R13: ffff8cd546aed248 R14: 0000000000000000 R15: ffff8cd548b1d000
>>>>>>>>>>> FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000) knlGS:0000000000000000
>>>>>>>>>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>>>>>>> CR2: 0000000500000000 CR3: 000000000311e000 CR4: 00000000001006f0
>>>>>>>>>>> Call Trace:
>>>>>>>>>>>        ? dwc3_remove_requests.constprop.0+0x14d/0x170
>>>>>>>>>>>        ? __dwc3_gadget_ep_disable+0x7a/0x160
>>>>>>>>>>>        ? dwc3_gadget_ep_disable+0x3d/0xd0
>>>>>>>>>>>        ? usb_ep_disable+0x1c/0x
>>>>>>>>>>>        ? u_audio_stop_capture+0x79/0x120 [u_audio]
>>>>>>>>>>>        ? afunc_set_alt+0x73/0x80 [usb_f_uac2]
>>>> So this is triggered by a SetInterface request...
>>>>
>>>>>>>>>>>        ? composite_setup+0x224/0x1b90 [libcomposite]
>>>>>>>>>>>        ? __dwc3_gadget_kick_transfer+0x160/0x400
>>>>>>>>>>>        ? dwc3_gadget_ep_queue+0xf3/0x1a0
>>>>>>>>>>>        ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>>>>>        ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>>>>>        ? dwc3_ep0_interrupt+0x459/0xa40
>>>>>>>>>>>        ? dwc3_thread_interrupt+0x8ee/0xf40
>>>>>>>>>>>        ? __schedule+0x235/0x6c0
>>>>>>>>>>>        ? disable_irq_nosync+0x10/0x10
>>>>>>>>>>>        ? irq_thread_fn+0x1b/0x60
>>>>>>>>>>>        ? irq_thread+0xc0/0x160
>>>>>>>>>>>        ? irq_thread_check_affinity+0x70/0x70
>>>>>>>>>>>        ? irq_forced_thread_fn+0x70/0x70
>>>>>>>>>>>        ? kthread+0x122/0x140
>>>>>>>>>>>        ? set_kthread_struct+0x40/0x40
>>>>>>>>>>>        ? ret_from_fork+0x22/0x30
>>>>>>>>>> Do you mind enabling dwc3 traces and collecting them? Trying to figure
>>>>>>>>>> out how we got here.
>>>>>>>>>>
>>>>>>>>> I'll try if I can get the same error by booting with USB in host mode
>>>>>>>>> and then switch to device mode. If so I can enable traces and collect as
>>>>>>>>> you explained me before.
>>>>>>>>>
>>>>>>>>> I'll try before monday, as then I fly for a holiday and will not be
>>>>>>>>> available before rc5.
>>>>>>>> you can enable all of those with kernel cmdline :-)
>>>>>>>>
>>>>>>>> https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
>>>>>>>>
>>>>>>>> you need ftrace_dump_on_oops=1 and also need the correct options on
>>>>>>>> trace_buf_size and trace_event.
>>>>>>>>
>>>>>>> On Edison-Arduino I have a switch to go to device mode, after which
>>>>>>> udev triggers a script configure gadgets through configfs.
>>>>>>>
>>>>>>> I tried to log following these instructions:
>>>>>>>
>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs  <https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs>
>>>>>>>
>>>>>>> Unfortunately the kernel crashes so badly I can not get to the ` cp
>>>>>>> /t/trace /root/trace.txt` line (after a while the watchdog kicks).
>>>>>>>
>>>>>>> What to do next?
>>>>>> Pass ftrace_dump_on_oops to kernel cmdline.
>>>>>>
>>>>> No sure if I did this right, on oops everything is pushed to console
>>>>> (115k2 serial), I hope nothing essential is lost.
>>>>>
>>>>> I copied the screen buffer to file see attached.
>>>> Thank you, I bet it took quite a some time :-) Anyway, looking at
>>>> the logs around Set Interface requests, we can track every endpoint
>>>> that's disabled. I'll take a guess and assume we're failing at the last
>>>> Set Interface, that means we should have something odd with ep6in, but
>>>> everything looks fine in the trace output:
>>>>
>>>> [   75.823107] irq/14-d-596       0d... 42789194us : dwc3_gadget_ep_enable: ep6in: mps 192/346 streams 16 burst 0 ring 0/0 flags E:swbp:<
>>>> [   75.835472] irq/14-d-596       0d... 42789198us : dwc3_alloc_request: ep6in: req 0000000002c71409 length 0/0 zsI ==> 0
>>>> [   75.846416] irq/14-d-596       0d... 42789202us : dwc3_ep_queue: ep6in: req 0000000002c71409 length 0/192 zsI ==> -115
>>>> [   75.857360] irq/14-d-596       0d... 42789204us : dwc3_alloc_request: ep6in: req 00000000a324f5d0 length 0/0 zsI ==> 0
>>>> [   75.868301] irq/14-d-596       0d... 42789206us : dwc3_ep_queue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -115
>>>> [   75.879244] irq/14-d-596       0d... 42789209us : dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0] (Not Active) [Status Phase]
>>>> [   75.891880] irq/14-d-596       0d... 42789211us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>> [   75.989131] irq/14-d-596       0d... 42789224us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>>> [   76.096261] irq/14-d-596       0d... 42789272us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
>>>> [   76.107834] irq/14-d-596       0d... 42789275us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>> [   76.122944] irq/14-d-596       0d... 42789277us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 zsI ==> 0
>>>> [   76.134160] irq/14-d-596       0d... 42789280us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>> [   76.231322] irq/14-d-596       0d... 42789292us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>>> [   76.297418] kworker/-23        0d... 42789670us : dwc3_ep_queue: ep3in: req 0000000029586135 length 0/96 ZsI ==> -115
>>>> [   76.308278] kworker/-23        0d... 42789695us : dwc3_prepare_trb: ep3in: trb 00000000b81213d6 (E1:D0) buf 0000000003b7a800 size 96 ctrl 00000811 (Hlcs:sC:normal)
>>>> [   76.395294] kworker/-23        0d... 42789707us : dwc3_gadget_ep_cmd: ep3in: cmd 'Update Transfer' [60007] params 00000000 00000000 00000000 --> status: Successful
>>>> [   76.471900] irq/14-d-596       0d... 42789842us : dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
>>>> [   76.489308] irq/14-d-596       0d... 42789845us : dwc3_ctrl_req: Set Interface(Intf = 5, Alt.Setting = 0)
>>>> [   76.505650] irq/14-d-596       0d... 42789851us : dwc3_ep_dequeue: ep6in: req 0000000002c71409 length 0/192 zsI ==> -115
>>>> [   76.523315] irq/14-d-596       0d... 42789854us : dwc3_gadget_giveback: ep6in: req 0000000002c71409 length 0/192 zsI ==> -104
>>>> [   76.541427] irq/14-d-596       0d... 42789857us : dwc3_free_request: ep6in: req 0000000002c71409 length 0/192 zsI ==> -104
>>>> [   76.559267] irq/14-d-596       0d... 42789859us : dwc3_ep_dequeue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -115
>>>> [   76.576937] irq/14-d-596       0d... 42789861us : dwc3_gadget_giveback: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -104
>>>> [   76.595046] irq/14-d-596       0d... 42789862us : dwc3_free_request: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -104
>>>> [   76.612892] irq/14-d-596       0d... 42789865us : dwc3_gadget_ep_disable: ep6in: mps 192/346 streams 16 burst 0 ring 0/0 flags E:swbp:<
>>>> [   76.665535] irq/14-d-596       0d... 42789873us : dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0] (Not Active) [Status Phase]
>>>> [   76.684716] irq/14-d-596       0d... 42789875us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>> [   76.819195] irq/14-d-596       0d... 42789886us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>>> [   76.926324] irq/14-d-596       0d... 42789930us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
>>>> [   76.937892] irq/14-d-596       0d... 42789933us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>> [   76.953003] irq/14-d-596       0d... 42789935us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 zsI ==> 0
>>>> [   76.964217] irq/14-d-596       0d... 42789938us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>> [   77.061379] irq/14-d-596       0d... 42789950us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>>> [   77.168595] irq/14-d-596       0d... 42790509us : dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
>>>> [   77.180159] irq/14-d-596       0d... 42790512us : dwc3_ctrl_req: Get String Descriptor(Index = 18, Length = 255)
>>>> [   77.190578] irq/14-d-596       0d... 42790537us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 0000000003b68000 size 36 ctrl 00000c53 (HLcs:SC:data)
>>>> [   77.287648] irq/14-d-596       0d... 42790550us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>>> [   77.333107] irq/14-d-596       0d... 42790557us : dwc3_event: event (000010c2): ep0in: Transfer Not Ready [0] (Not Active) [Data Phase]
>>>> [   77.407223] irq/14-d-596       0d... 42790575us : dwc3_event: event (000090c2): ep0in: Transfer Not Ready [0] (Active) [Data Phase]
>>>> [   77.480985] irq/14-d-596       0d... 42790588us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Data Phase]
>>>> [   77.492376] irq/14-d-596       0d... 42790590us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 0000000003b68000 size 0 ctrl 00000c52 (hLcs:SC:data)
>>>> [   77.507221] irq/14-d-596       0d... 42790595us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 36/36 ZsI ==> 0
>>>> [   77.518609] irq/14-d-596       0d... 42790597us : dwc3_event: event (000020c0): ep0out: Transfer Not Ready [0] (Not Active) [Status Phase]
>>>> [   77.531332] irq/14-d-596       0d... 42790598us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c43 (HLcs:SC:status3)
>>>> [   77.628669] irq/14-d-596       0d... 42790609us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>>>
>>>> Do you mind adding a few prints in dwc3_remove_requests to tell us which
>>>> endpoint is being processed? Then we'll know for sure which one caused
>>>> the crash.
>>>>
>>> I wouldn't mind but am leaving on a holiday, won't have time until 6 aug.
>> not a problem, we'll still be here when you're back :-)
>
> Well, let's go then :-)
>
> To get back in the mood I have retested 5.13.0, 5.14.0-rc1, 5.14.0-rc2
> and 5.14.0-rc5.
>
> I find that 5.13.0 works fine, and the issue starts from 5.14.0-rc1.

That's great finding. We have a bisection point. There are a total of
13764 commits between v5.13 and v5.14-rc1

	$ git rev-list  --count v5.13..v5.14-rc1
	13764

git bisect should find the offending commit in at most 14 tries. That's
not too bad.

> With 5.14.0-rc5 the problem seems worse (or different?), and just
> disabling uac2 gadget does not prevent the crash. Even disabling gser 
> and mass_storage.usb0 as well there is still a crash.
>
> Now I'm not sure how to proceed. Bisect rc1? Or focus on rc5 (rc6?)?

I'd first bisect between 5.13 and v5.14-rc1. Once you find the offending
commit, verify if reverting that on -rc1 works, then verify if reverting
on -rc5 also works :-)

-- 
balbi

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-08-16  5:18                                   ` Felipe Balbi
@ 2021-08-17 22:00                                     ` Ferry Toth
  2021-08-18 19:07                                       ` Ferry Toth
  0 siblings, 1 reply; 73+ messages in thread
From: Ferry Toth @ 2021-08-17 22:00 UTC (permalink / raw)
  To: Felipe Balbi
  Cc: Wesley Cheng, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list, linux-usb, linux-kernel, devicetree, linux-arm-msm,
	jackp, heikki.krogerus, andy.shevchenko, Ruslan Bilovol

Hi,

Op 16-08-2021 om 07:18 schreef Felipe Balbi:
> Hi,
>
> Ferry Toth <fntoth@gmail.com> writes:
>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>>>>> Hardware name: Intel Corporation Merrifield/BODEGA BAY, BIOS 542
>>>>>>>>>>>> 2015.01.21:18.19.48
>>>>>>>>>>>> RIP: 0010:0x500000000
>>>>>>>>>>>> Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
>>>>>>>>>>>> RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
>>>>>>>>>>>> RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX: 0000000000000000
>>>>>>>>>>>> RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI: ffff8cd546aed200
>>>>>>>>>>>> RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09: 0000000000000001
>>>>>>>>>>>> R10: ffff8cd541fd28c0 R11: 0000000000000000 R12: ffff8cd547342828
>>>>>>>>>>>> R13: ffff8cd546aed248 R14: 0000000000000000 R15: ffff8cd548b1d000
>>>>>>>>>>>> FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000) knlGS:0000000000000000
>>>>>>>>>>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>>>>>>>> CR2: 0000000500000000 CR3: 000000000311e000 CR4: 00000000001006f0
>>>>>>>>>>>> Call Trace:
>>>>>>>>>>>>         ? dwc3_remove_requests.constprop.0+0x14d/0x170
>>>>>>>>>>>>         ? __dwc3_gadget_ep_disable+0x7a/0x160
>>>>>>>>>>>>         ? dwc3_gadget_ep_disable+0x3d/0xd0
>>>>>>>>>>>>         ? usb_ep_disable+0x1c/0x
>>>>>>>>>>>>         ? u_audio_stop_capture+0x79/0x120 [u_audio]
>>>>>>>>>>>>         ? afunc_set_alt+0x73/0x80 [usb_f_uac2]
>>>>> So this is triggered by a SetInterface request...
>>>>>
>>>>>>>>>>>>         ? composite_setup+0x224/0x1b90 [libcomposite]
>>>>>>>>>>>>         ? __dwc3_gadget_kick_transfer+0x160/0x400
>>>>>>>>>>>>         ? dwc3_gadget_ep_queue+0xf3/0x1a0
>>>>>>>>>>>>         ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>>>>>>         ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>>>>>>         ? dwc3_ep0_interrupt+0x459/0xa40
>>>>>>>>>>>>         ? dwc3_thread_interrupt+0x8ee/0xf40
>>>>>>>>>>>>         ? __schedule+0x235/0x6c0
>>>>>>>>>>>>         ? disable_irq_nosync+0x10/0x10
>>>>>>>>>>>>         ? irq_thread_fn+0x1b/0x60
>>>>>>>>>>>>         ? irq_thread+0xc0/0x160
>>>>>>>>>>>>         ? irq_thread_check_affinity+0x70/0x70
>>>>>>>>>>>>         ? irq_forced_thread_fn+0x70/0x70
>>>>>>>>>>>>         ? kthread+0x122/0x140
>>>>>>>>>>>>         ? set_kthread_struct+0x40/0x40
>>>>>>>>>>>>         ? ret_from_fork+0x22/0x30
>>>>>>>>>>> Do you mind enabling dwc3 traces and collecting them? Trying to figure
>>>>>>>>>>> out how we got here.
>>>>>>>>>>>
>>>>>>>>>> I'll try if I can get the same error by booting with USB in host mode
>>>>>>>>>> and then switch to device mode. If so I can enable traces and collect as
>>>>>>>>>> you explained me before.
>>>>>>>>>>
>>>>>>>>>> I'll try before monday, as then I fly for a holiday and will not be
>>>>>>>>>> available before rc5.
>>>>>>>>> you can enable all of those with kernel cmdline :-)
>>>>>>>>>
>>>>>>>>> https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
>>>>>>>>>
>>>>>>>>> you need ftrace_dump_on_oops=1 and also need the correct options on
>>>>>>>>> trace_buf_size and trace_event.
>>>>>>>>>
>>>>>>>> On Edison-Arduino I have a switch to go to device mode, after which
>>>>>>>> udev triggers a script configure gadgets through configfs.
>>>>>>>>
>>>>>>>> I tried to log following these instructions:
>>>>>>>>
>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs  <https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs>
>>>>>>>>
>>>>>>>> Unfortunately the kernel crashes so badly I can not get to the ` cp
>>>>>>>> /t/trace /root/trace.txt` line (after a while the watchdog kicks).
>>>>>>>>
>>>>>>>> What to do next?
>>>>>>> Pass ftrace_dump_on_oops to kernel cmdline.
>>>>>>>
>>>>>> No sure if I did this right, on oops everything is pushed to console
>>>>>> (115k2 serial), I hope nothing essential is lost.
>>>>>>
>>>>>> I copied the screen buffer to file see attached.
>>>>> Thank you, I bet it took quite a some time :-) Anyway, looking at
>>>>> the logs around Set Interface requests, we can track every endpoint
>>>>> that's disabled. I'll take a guess and assume we're failing at the last
>>>>> Set Interface, that means we should have something odd with ep6in, but
>>>>> everything looks fine in the trace output:
>>>>>
>>>>> [   75.823107] irq/14-d-596       0d... 42789194us : dwc3_gadget_ep_enable: ep6in: mps 192/346 streams 16 burst 0 ring 0/0 flags E:swbp:<
>>>>> [   75.835472] irq/14-d-596       0d... 42789198us : dwc3_alloc_request: ep6in: req 0000000002c71409 length 0/0 zsI ==> 0
>>>>> [   75.846416] irq/14-d-596       0d... 42789202us : dwc3_ep_queue: ep6in: req 0000000002c71409 length 0/192 zsI ==> -115
>>>>> [   75.857360] irq/14-d-596       0d... 42789204us : dwc3_alloc_request: ep6in: req 00000000a324f5d0 length 0/0 zsI ==> 0
>>>>> [   75.868301] irq/14-d-596       0d... 42789206us : dwc3_ep_queue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -115
>>>>> [   75.879244] irq/14-d-596       0d... 42789209us : dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0] (Not Active) [Status Phase]
>>>>> [   75.891880] irq/14-d-596       0d... 42789211us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>>> [   75.989131] irq/14-d-596       0d... 42789224us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>>>> [   76.096261] irq/14-d-596       0d... 42789272us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
>>>>> [   76.107834] irq/14-d-596       0d... 42789275us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>>> [   76.122944] irq/14-d-596       0d... 42789277us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 zsI ==> 0
>>>>> [   76.134160] irq/14-d-596       0d... 42789280us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>>> [   76.231322] irq/14-d-596       0d... 42789292us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>>>> [   76.297418] kworker/-23        0d... 42789670us : dwc3_ep_queue: ep3in: req 0000000029586135 length 0/96 ZsI ==> -115
>>>>> [   76.308278] kworker/-23        0d... 42789695us : dwc3_prepare_trb: ep3in: trb 00000000b81213d6 (E1:D0) buf 0000000003b7a800 size 96 ctrl 00000811 (Hlcs:sC:normal)
>>>>> [   76.395294] kworker/-23        0d... 42789707us : dwc3_gadget_ep_cmd: ep3in: cmd 'Update Transfer' [60007] params 00000000 00000000 00000000 --> status: Successful
>>>>> [   76.471900] irq/14-d-596       0d... 42789842us : dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
>>>>> [   76.489308] irq/14-d-596       0d... 42789845us : dwc3_ctrl_req: Set Interface(Intf = 5, Alt.Setting = 0)
>>>>> [   76.505650] irq/14-d-596       0d... 42789851us : dwc3_ep_dequeue: ep6in: req 0000000002c71409 length 0/192 zsI ==> -115
>>>>> [   76.523315] irq/14-d-596       0d... 42789854us : dwc3_gadget_giveback: ep6in: req 0000000002c71409 length 0/192 zsI ==> -104
>>>>> [   76.541427] irq/14-d-596       0d... 42789857us : dwc3_free_request: ep6in: req 0000000002c71409 length 0/192 zsI ==> -104
>>>>> [   76.559267] irq/14-d-596       0d... 42789859us : dwc3_ep_dequeue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -115
>>>>> [   76.576937] irq/14-d-596       0d... 42789861us : dwc3_gadget_giveback: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -104
>>>>> [   76.595046] irq/14-d-596       0d... 42789862us : dwc3_free_request: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -104
>>>>> [   76.612892] irq/14-d-596       0d... 42789865us : dwc3_gadget_ep_disable: ep6in: mps 192/346 streams 16 burst 0 ring 0/0 flags E:swbp:<
>>>>> [   76.665535] irq/14-d-596       0d... 42789873us : dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0] (Not Active) [Status Phase]
>>>>> [   76.684716] irq/14-d-596       0d... 42789875us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>>> [   76.819195] irq/14-d-596       0d... 42789886us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>>>> [   76.926324] irq/14-d-596       0d... 42789930us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
>>>>> [   76.937892] irq/14-d-596       0d... 42789933us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>>> [   76.953003] irq/14-d-596       0d... 42789935us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 zsI ==> 0
>>>>> [   76.964217] irq/14-d-596       0d... 42789938us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>>> [   77.061379] irq/14-d-596       0d... 42789950us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>>>> [   77.168595] irq/14-d-596       0d... 42790509us : dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
>>>>> [   77.180159] irq/14-d-596       0d... 42790512us : dwc3_ctrl_req: Get String Descriptor(Index = 18, Length = 255)
>>>>> [   77.190578] irq/14-d-596       0d... 42790537us : dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 0000000003b68000 size 36 ctrl 00000c53 (HLcs:SC:data)
>>>>> [   77.287648] irq/14-d-596       0d... 42790550us : dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>>>> [   77.333107] irq/14-d-596       0d... 42790557us : dwc3_event: event (000010c2): ep0in: Transfer Not Ready [0] (Not Active) [Data Phase]
>>>>> [   77.407223] irq/14-d-596       0d... 42790575us : dwc3_event: event (000090c2): ep0in: Transfer Not Ready [0] (Active) [Data Phase]
>>>>> [   77.480985] irq/14-d-596       0d... 42790588us : dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) [Data Phase]
>>>>> [   77.492376] irq/14-d-596       0d... 42790590us : dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 0000000003b68000 size 0 ctrl 00000c52 (hLcs:SC:data)
>>>>> [   77.507221] irq/14-d-596       0d... 42790595us : dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 36/36 ZsI ==> 0
>>>>> [   77.518609] irq/14-d-596       0d... 42790597us : dwc3_event: event (000020c0): ep0out: Transfer Not Ready [0] (Not Active) [Status Phase]
>>>>> [   77.531332] irq/14-d-596       0d... 42790598us : dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 000000001bded000 size 0 ctrl 00000c43 (HLcs:SC:status3)
>>>>> [   77.628669] irq/14-d-596       0d... 42790609us : dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 00000000 1bded000 00000000 --> status: Successful
>>>>>
>>>>> Do you mind adding a few prints in dwc3_remove_requests to tell us which
>>>>> endpoint is being processed? Then we'll know for sure which one caused
>>>>> the crash.
>>>>>
>>>> I wouldn't mind but am leaving on a holiday, won't have time until 6 aug.
>>> not a problem, we'll still be here when you're back :-)
>> Well, let's go then :-)
>>
>> To get back in the mood I have retested 5.13.0, 5.14.0-rc1, 5.14.0-rc2
>> and 5.14.0-rc5.
>>
>> I find that 5.13.0 works fine, and the issue starts from 5.14.0-rc1.
> That's great finding. We have a bisection point. There are a total of
> 13764 commits between v5.13 and v5.14-rc1
>
> 	$ git rev-list  --count v5.13..v5.14-rc1
> 	13764
>
> git bisect should find the offending commit in at most 14 tries. That's
> not too bad.
I correctly guesstimated that the problem got introduced by the usb 
merge 79160a60

"Merge tag 'usb-5.14-rc1' of 
git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb"

116 commits(7 bisects).

24f779dac8f3efb9629adc0e486914d93dc45517 is the first bad commit

"usb: gadget: f_uac2/u_audio: add feedback endpoint support"

Ruslan's 3 patches are related to each other so I reverted all three 
24f779da...e89bb428 and applied the reverts to rc1.

I can confirm this indeed resolves the problem in rc1.

Is late now, tomorrow evening I will apply the reverts to rc6.

>> With 5.14.0-rc5 the problem seems worse (or different?), and just
>> disabling uac2 gadget does not prevent the crash. Even disabling gser
>> and mass_storage.usb0 as well there is still a crash.
>>
>> Now I'm not sure how to proceed. Bisect rc1? Or focus on rc5 (rc6?)?
> I'd first bisect between 5.13 and v5.14-rc1. Once you find the offending
> commit, verify if reverting that on -rc1 works, then verify if reverting
> on -rc5 also works :-)
>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-08-17 22:00                                     ` Ferry Toth
@ 2021-08-18 19:07                                       ` Ferry Toth
  2021-08-19  7:51                                         ` Pavel Hofman
  2021-08-19  7:55                                         ` Andy Shevchenko
  0 siblings, 2 replies; 73+ messages in thread
From: Ferry Toth @ 2021-08-18 19:07 UTC (permalink / raw)
  To: Felipe Balbi
  Cc: Wesley Cheng, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list, linux-usb, linux-kernel, devicetree, linux-arm-msm,
	jackp, heikki.krogerus, andy.shevchenko, Ruslan Bilovol

Hi,

Op 18-08-2021 om 00:00 schreef Ferry Toth:
> Hi,
>
> Op 16-08-2021 om 07:18 schreef Felipe Balbi:
>> Hi,
>>
>> Ferry Toth <fntoth@gmail.com> writes:
>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>>>>>> Hardware name: Intel Corporation Merrifield/BODEGA BAY, 
>>>>>>>>>>>>> BIOS 542
>>>>>>>>>>>>> 2015.01.21:18.19.48
>>>>>>>>>>>>> RIP: 0010:0x500000000
>>>>>>>>>>>>> Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
>>>>>>>>>>>>> RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
>>>>>>>>>>>>> RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX: 
>>>>>>>>>>>>> 0000000000000000
>>>>>>>>>>>>> RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI: 
>>>>>>>>>>>>> ffff8cd546aed200
>>>>>>>>>>>>> RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09: 
>>>>>>>>>>>>> 0000000000000001
>>>>>>>>>>>>> R10: ffff8cd541fd28c0 R11: 0000000000000000 R12: 
>>>>>>>>>>>>> ffff8cd547342828
>>>>>>>>>>>>> R13: ffff8cd546aed248 R14: 0000000000000000 R15: 
>>>>>>>>>>>>> ffff8cd548b1d000
>>>>>>>>>>>>> FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000) 
>>>>>>>>>>>>> knlGS:0000000000000000
>>>>>>>>>>>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>>>>>>>>> CR2: 0000000500000000 CR3: 000000000311e000 CR4: 
>>>>>>>>>>>>> 00000000001006f0
>>>>>>>>>>>>> Call Trace:
>>>>>>>>>>>>>         ? dwc3_remove_requests.constprop.0+0x14d/0x170
>>>>>>>>>>>>>         ? __dwc3_gadget_ep_disable+0x7a/0x160
>>>>>>>>>>>>>         ? dwc3_gadget_ep_disable+0x3d/0xd0
>>>>>>>>>>>>>         ? usb_ep_disable+0x1c/0x
>>>>>>>>>>>>>         ? u_audio_stop_capture+0x79/0x120 [u_audio]
>>>>>>>>>>>>>         ? afunc_set_alt+0x73/0x80 [usb_f_uac2]
>>>>>> So this is triggered by a SetInterface request...
>>>>>>
>>>>>>>>>>>>>         ? composite_setup+0x224/0x1b90 [libcomposite]
>>>>>>>>>>>>>         ? __dwc3_gadget_kick_transfer+0x160/0x400
>>>>>>>>>>>>>         ? dwc3_gadget_ep_queue+0xf3/0x1a0
>>>>>>>>>>>>>         ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>>>>>>>         ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>>>>>>>         ? dwc3_ep0_interrupt+0x459/0xa40
>>>>>>>>>>>>>         ? dwc3_thread_interrupt+0x8ee/0xf40
>>>>>>>>>>>>>         ? __schedule+0x235/0x6c0
>>>>>>>>>>>>>         ? disable_irq_nosync+0x10/0x10
>>>>>>>>>>>>>         ? irq_thread_fn+0x1b/0x60
>>>>>>>>>>>>>         ? irq_thread+0xc0/0x160
>>>>>>>>>>>>>         ? irq_thread_check_affinity+0x70/0x70
>>>>>>>>>>>>>         ? irq_forced_thread_fn+0x70/0x70
>>>>>>>>>>>>>         ? kthread+0x122/0x140
>>>>>>>>>>>>>         ? set_kthread_struct+0x40/0x40
>>>>>>>>>>>>>         ? ret_from_fork+0x22/0x30
>>>>>>>>>>>> Do you mind enabling dwc3 traces and collecting them? 
>>>>>>>>>>>> Trying to figure
>>>>>>>>>>>> out how we got here.
>>>>>>>>>>>>
>>>>>>>>>>> I'll try if I can get the same error by booting with USB in 
>>>>>>>>>>> host mode
>>>>>>>>>>> and then switch to device mode. If so I can enable traces 
>>>>>>>>>>> and collect as
>>>>>>>>>>> you explained me before.
>>>>>>>>>>>
>>>>>>>>>>> I'll try before monday, as then I fly for a holiday and will 
>>>>>>>>>>> not be
>>>>>>>>>>> available before rc5.
>>>>>>>>>> you can enable all of those with kernel cmdline :-)
>>>>>>>>>>
>>>>>>>>>> https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html 
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> you need ftrace_dump_on_oops=1 and also need the correct 
>>>>>>>>>> options on
>>>>>>>>>> trace_buf_size and trace_event.
>>>>>>>>>>
>>>>>>>>> On Edison-Arduino I have a switch to go to device mode, after 
>>>>>>>>> which
>>>>>>>>> udev triggers a script configure gadgets through configfs.
>>>>>>>>>
>>>>>>>>> I tried to log following these instructions:
>>>>>>>>>
>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs 
>>>>>>>>> <https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs> 
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Unfortunately the kernel crashes so badly I can not get to the 
>>>>>>>>> ` cp
>>>>>>>>> /t/trace /root/trace.txt` line (after a while the watchdog 
>>>>>>>>> kicks).
>>>>>>>>>
>>>>>>>>> What to do next?
>>>>>>>> Pass ftrace_dump_on_oops to kernel cmdline.
>>>>>>>>
>>>>>>> No sure if I did this right, on oops everything is pushed to 
>>>>>>> console
>>>>>>> (115k2 serial), I hope nothing essential is lost.
>>>>>>>
>>>>>>> I copied the screen buffer to file see attached.
>>>>>> Thank you, I bet it took quite a some time :-) Anyway, looking at
>>>>>> the logs around Set Interface requests, we can track every endpoint
>>>>>> that's disabled. I'll take a guess and assume we're failing at 
>>>>>> the last
>>>>>> Set Interface, that means we should have something odd with 
>>>>>> ep6in, but
>>>>>> everything looks fine in the trace output:
>>>>>>
>>>>>> [   75.823107] irq/14-d-596       0d... 42789194us : 
>>>>>> dwc3_gadget_ep_enable: ep6in: mps 192/346 streams 16 burst 0 ring 
>>>>>> 0/0 flags E:swbp:<
>>>>>> [   75.835472] irq/14-d-596       0d... 42789198us : 
>>>>>> dwc3_alloc_request: ep6in: req 0000000002c71409 length 0/0 zsI ==> 0
>>>>>> [   75.846416] irq/14-d-596       0d... 42789202us : 
>>>>>> dwc3_ep_queue: ep6in: req 0000000002c71409 length 0/192 zsI ==> -115
>>>>>> [   75.857360] irq/14-d-596       0d... 42789204us : 
>>>>>> dwc3_alloc_request: ep6in: req 00000000a324f5d0 length 0/0 zsI ==> 0
>>>>>> [   75.868301] irq/14-d-596       0d... 42789206us : 
>>>>>> dwc3_ep_queue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -115
>>>>>> [   75.879244] irq/14-d-596       0d... 42789209us : dwc3_event: 
>>>>>> event (000020c2): ep0in: Transfer Not Ready [0] (Not Active) 
>>>>>> [Status Phase]
>>>>>> [   75.891880] irq/14-d-596       0d... 42789211us : 
>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 
>>>>>> 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>>>> [   75.989131] irq/14-d-596       0d... 42789224us : 
>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 
>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>> [   76.096261] irq/14-d-596       0d... 42789272us : dwc3_event: 
>>>>>> event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
>>>>>> [   76.107834] irq/14-d-596       0d... 42789275us : 
>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>> 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>>>> [   76.122944] irq/14-d-596       0d... 42789277us : 
>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 zsI 
>>>>>> ==> 0
>>>>>> [   76.134160] irq/14-d-596       0d... 42789280us : 
>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>> 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>>>> [   76.231322] irq/14-d-596       0d... 42789292us : 
>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 
>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>> [   76.297418] kworker/-23        0d... 42789670us : 
>>>>>> dwc3_ep_queue: ep3in: req 0000000029586135 length 0/96 ZsI ==> -115
>>>>>> [   76.308278] kworker/-23        0d... 42789695us : 
>>>>>> dwc3_prepare_trb: ep3in: trb 00000000b81213d6 (E1:D0) buf 
>>>>>> 0000000003b7a800 size 96 ctrl 00000811 (Hlcs:sC:normal)
>>>>>> [   76.395294] kworker/-23        0d... 42789707us : 
>>>>>> dwc3_gadget_ep_cmd: ep3in: cmd 'Update Transfer' [60007] params 
>>>>>> 00000000 00000000 00000000 --> status: Successful
>>>>>> [   76.471900] irq/14-d-596       0d... 42789842us : dwc3_event: 
>>>>>> event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
>>>>>> [   76.489308] irq/14-d-596       0d... 42789845us : 
>>>>>> dwc3_ctrl_req: Set Interface(Intf = 5, Alt.Setting = 0)
>>>>>> [   76.505650] irq/14-d-596       0d... 42789851us : 
>>>>>> dwc3_ep_dequeue: ep6in: req 0000000002c71409 length 0/192 zsI ==> 
>>>>>> -115
>>>>>> [   76.523315] irq/14-d-596       0d... 42789854us : 
>>>>>> dwc3_gadget_giveback: ep6in: req 0000000002c71409 length 0/192 
>>>>>> zsI ==> -104
>>>>>> [   76.541427] irq/14-d-596       0d... 42789857us : 
>>>>>> dwc3_free_request: ep6in: req 0000000002c71409 length 0/192 zsI 
>>>>>> ==> -104
>>>>>> [   76.559267] irq/14-d-596       0d... 42789859us : 
>>>>>> dwc3_ep_dequeue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> 
>>>>>> -115
>>>>>> [   76.576937] irq/14-d-596       0d... 42789861us : 
>>>>>> dwc3_gadget_giveback: ep6in: req 00000000a324f5d0 length 0/192 
>>>>>> zsI ==> -104
>>>>>> [   76.595046] irq/14-d-596       0d... 42789862us : 
>>>>>> dwc3_free_request: ep6in: req 00000000a324f5d0 length 0/192 zsI 
>>>>>> ==> -104
>>>>>> [   76.612892] irq/14-d-596       0d... 42789865us : 
>>>>>> dwc3_gadget_ep_disable: ep6in: mps 192/346 streams 16 burst 0 
>>>>>> ring 0/0 flags E:swbp:<
>>>>>> [   76.665535] irq/14-d-596       0d... 42789873us : dwc3_event: 
>>>>>> event (000020c2): ep0in: Transfer Not Ready [0] (Not Active) 
>>>>>> [Status Phase]
>>>>>> [   76.684716] irq/14-d-596       0d... 42789875us : 
>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 
>>>>>> 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>>>> [   76.819195] irq/14-d-596       0d... 42789886us : 
>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 
>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>> [   76.926324] irq/14-d-596       0d... 42789930us : dwc3_event: 
>>>>>> event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
>>>>>> [   76.937892] irq/14-d-596       0d... 42789933us : 
>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>> 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>>>> [   76.953003] irq/14-d-596       0d... 42789935us : 
>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 zsI 
>>>>>> ==> 0
>>>>>> [   76.964217] irq/14-d-596       0d... 42789938us : 
>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>> 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>>>> [   77.061379] irq/14-d-596       0d... 42789950us : 
>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 
>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>> [   77.168595] irq/14-d-596       0d... 42790509us : dwc3_event: 
>>>>>> event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
>>>>>> [   77.180159] irq/14-d-596       0d... 42790512us : 
>>>>>> dwc3_ctrl_req: Get String Descriptor(Index = 18, Length = 255)
>>>>>> [   77.190578] irq/14-d-596       0d... 42790537us : 
>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 
>>>>>> 0000000003b68000 size 36 ctrl 00000c53 (HLcs:SC:data)
>>>>>> [   77.287648] irq/14-d-596       0d... 42790550us : 
>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 
>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>> [   77.333107] irq/14-d-596       0d... 42790557us : dwc3_event: 
>>>>>> event (000010c2): ep0in: Transfer Not Ready [0] (Not Active) 
>>>>>> [Data Phase]
>>>>>> [   77.407223] irq/14-d-596       0d... 42790575us : dwc3_event: 
>>>>>> event (000090c2): ep0in: Transfer Not Ready [0] (Active) [Data 
>>>>>> Phase]
>>>>>> [   77.480985] irq/14-d-596       0d... 42790588us : dwc3_event: 
>>>>>> event (0000c042): ep0in: Transfer Complete (sIL) [Data Phase]
>>>>>> [   77.492376] irq/14-d-596       0d... 42790590us : 
>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>> 0000000003b68000 size 0 ctrl 00000c52 (hLcs:SC:data)
>>>>>> [   77.507221] irq/14-d-596       0d... 42790595us : 
>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 36/36 
>>>>>> ZsI ==> 0
>>>>>> [   77.518609] irq/14-d-596       0d... 42790597us : dwc3_event: 
>>>>>> event (000020c0): ep0out: Transfer Not Ready [0] (Not Active) 
>>>>>> [Status Phase]
>>>>>> [   77.531332] irq/14-d-596       0d... 42790598us : 
>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>> 000000001bded000 size 0 ctrl 00000c43 (HLcs:SC:status3)
>>>>>> [   77.628669] irq/14-d-596       0d... 42790609us : 
>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 
>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>
>>>>>> Do you mind adding a few prints in dwc3_remove_requests to tell 
>>>>>> us which
>>>>>> endpoint is being processed? Then we'll know for sure which one 
>>>>>> caused
>>>>>> the crash.
>>>>>>
>>>>> I wouldn't mind but am leaving on a holiday, won't have time until 
>>>>> 6 aug.
>>>> not a problem, we'll still be here when you're back :-)
>>> Well, let's go then :-)
>>>
>>> To get back in the mood I have retested 5.13.0, 5.14.0-rc1, 5.14.0-rc2
>>> and 5.14.0-rc5.
>>>
>>> I find that 5.13.0 works fine, and the issue starts from 5.14.0-rc1.
>> That's great finding. We have a bisection point. There are a total of
>> 13764 commits between v5.13 and v5.14-rc1
>>
>>     $ git rev-list  --count v5.13..v5.14-rc1
>>     13764
>>
>> git bisect should find the offending commit in at most 14 tries. That's
>> not too bad.
> I correctly guesstimated that the problem got introduced by the usb 
> merge 79160a60
>
> "Merge tag 'usb-5.14-rc1' of 
> git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb"
>
> 116 commits(7 bisects).
>
> 24f779dac8f3efb9629adc0e486914d93dc45517 is the first bad commit
>
> "usb: gadget: f_uac2/u_audio: add feedback endpoint support"
>
> Ruslan's 3 patches are related to each other so I reverted all three 
> 24f779da...e89bb428 and applied the reverts to rc1.
>
> I can confirm this indeed resolves the problem in rc1.
>
> Is late now, tomorrow evening I will apply the reverts to rc6.

With these reverts rc6 works fine as well.

So, where do we go from here?

>
>>> With 5.14.0-rc5 the problem seems worse (or different?), and just
>>> disabling uac2 gadget does not prevent the crash. Even disabling gser
>>> and mass_storage.usb0 as well there is still a crash.
>>>
>>> Now I'm not sure how to proceed. Bisect rc1? Or focus on rc5 (rc6?)?
>> I'd first bisect between 5.13 and v5.14-rc1. Once you find the offending
>> commit, verify if reverting that on -rc1 works, then verify if reverting
>> on -rc5 also works :-)
>>

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-08-18 19:07                                       ` Ferry Toth
@ 2021-08-19  7:51                                         ` Pavel Hofman
  2021-08-19 20:10                                           ` Ferry Toth
  2021-08-19  7:55                                         ` Andy Shevchenko
  1 sibling, 1 reply; 73+ messages in thread
From: Pavel Hofman @ 2021-08-19  7:51 UTC (permalink / raw)
  To: Ferry Toth, Felipe Balbi
  Cc: Wesley Cheng, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list, linux-usb, linux-kernel, devicetree, linux-arm-msm,
	jackp, heikki.krogerus, andy.shevchenko, Ruslan Bilovol

Hi,

Dne 18. 08. 21 v 21:07 Ferry Toth napsal(a):
> Hi,
> 
> Op 18-08-2021 om 00:00 schreef Ferry Toth:
>> Hi,
>>
>> Op 16-08-2021 om 07:18 schreef Felipe Balbi:
>>> Hi,
>>>
>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>>>>>>> Hardware name: Intel Corporation Merrifield/BODEGA BAY, 
>>>>>>>>>>>>>> BIOS 542
>>>>>>>>>>>>>> 2015.01.21:18.19.48
>>>>>>>>>>>>>> RIP: 0010:0x500000000
>>>>>>>>>>>>>> Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
>>>>>>>>>>>>>> RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
>>>>>>>>>>>>>> RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX: 
>>>>>>>>>>>>>> 0000000000000000
>>>>>>>>>>>>>> RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI: 
>>>>>>>>>>>>>> ffff8cd546aed200
>>>>>>>>>>>>>> RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09: 
>>>>>>>>>>>>>> 0000000000000001
>>>>>>>>>>>>>> R10: ffff8cd541fd28c0 R11: 0000000000000000 R12: 
>>>>>>>>>>>>>> ffff8cd547342828
>>>>>>>>>>>>>> R13: ffff8cd546aed248 R14: 0000000000000000 R15: 
>>>>>>>>>>>>>> ffff8cd548b1d000
>>>>>>>>>>>>>> FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000) 
>>>>>>>>>>>>>> knlGS:0000000000000000
>>>>>>>>>>>>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>>>>>>>>>> CR2: 0000000500000000 CR3: 000000000311e000 CR4: 
>>>>>>>>>>>>>> 00000000001006f0
>>>>>>>>>>>>>> Call Trace:
>>>>>>>>>>>>>>         ? dwc3_remove_requests.constprop.0+0x14d/0x170
>>>>>>>>>>>>>>         ? __dwc3_gadget_ep_disable+0x7a/0x160
>>>>>>>>>>>>>>         ? dwc3_gadget_ep_disable+0x3d/0xd0
>>>>>>>>>>>>>>         ? usb_ep_disable+0x1c/0x
>>>>>>>>>>>>>>         ? u_audio_stop_capture+0x79/0x120 [u_audio]
>>>>>>>>>>>>>>         ? afunc_set_alt+0x73/0x80 [usb_f_uac2]
>>>>>>> So this is triggered by a SetInterface request...
>>>>>>>
>>>>>>>>>>>>>>         ? composite_setup+0x224/0x1b90 [libcomposite]
>>>>>>>>>>>>>>         ? __dwc3_gadget_kick_transfer+0x160/0x400
>>>>>>>>>>>>>>         ? dwc3_gadget_ep_queue+0xf3/0x1a0
>>>>>>>>>>>>>>         ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>>>>>>>>         ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>>>>>>>>         ? dwc3_ep0_interrupt+0x459/0xa40
>>>>>>>>>>>>>>         ? dwc3_thread_interrupt+0x8ee/0xf40
>>>>>>>>>>>>>>         ? __schedule+0x235/0x6c0
>>>>>>>>>>>>>>         ? disable_irq_nosync+0x10/0x10
>>>>>>>>>>>>>>         ? irq_thread_fn+0x1b/0x60
>>>>>>>>>>>>>>         ? irq_thread+0xc0/0x160
>>>>>>>>>>>>>>         ? irq_thread_check_affinity+0x70/0x70
>>>>>>>>>>>>>>         ? irq_forced_thread_fn+0x70/0x70
>>>>>>>>>>>>>>         ? kthread+0x122/0x140
>>>>>>>>>>>>>>         ? set_kthread_struct+0x40/0x40
>>>>>>>>>>>>>>         ? ret_from_fork+0x22/0x30
>>>>>>>>>>>>> Do you mind enabling dwc3 traces and collecting them? 
>>>>>>>>>>>>> Trying to figure
>>>>>>>>>>>>> out how we got here.
>>>>>>>>>>>>>
>>>>>>>>>>>> I'll try if I can get the same error by booting with USB in 
>>>>>>>>>>>> host mode
>>>>>>>>>>>> and then switch to device mode. If so I can enable traces 
>>>>>>>>>>>> and collect as
>>>>>>>>>>>> you explained me before.
>>>>>>>>>>>>
>>>>>>>>>>>> I'll try before monday, as then I fly for a holiday and will 
>>>>>>>>>>>> not be
>>>>>>>>>>>> available before rc5.
>>>>>>>>>>> you can enable all of those with kernel cmdline :-)
>>>>>>>>>>>
>>>>>>>>>>> https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html 
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> you need ftrace_dump_on_oops=1 and also need the correct 
>>>>>>>>>>> options on
>>>>>>>>>>> trace_buf_size and trace_event.
>>>>>>>>>>>
>>>>>>>>>> On Edison-Arduino I have a switch to go to device mode, after 
>>>>>>>>>> which
>>>>>>>>>> udev triggers a script configure gadgets through configfs.
>>>>>>>>>>
>>>>>>>>>> I tried to log following these instructions:
>>>>>>>>>>
>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs 
>>>>>>>>>> <https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs> 
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Unfortunately the kernel crashes so badly I can not get to the 
>>>>>>>>>> ` cp
>>>>>>>>>> /t/trace /root/trace.txt` line (after a while the watchdog 
>>>>>>>>>> kicks).
>>>>>>>>>>
>>>>>>>>>> What to do next?
>>>>>>>>> Pass ftrace_dump_on_oops to kernel cmdline.
>>>>>>>>>
>>>>>>>> No sure if I did this right, on oops everything is pushed to 
>>>>>>>> console
>>>>>>>> (115k2 serial), I hope nothing essential is lost.
>>>>>>>>
>>>>>>>> I copied the screen buffer to file see attached.
>>>>>>> Thank you, I bet it took quite a some time :-) Anyway, looking at
>>>>>>> the logs around Set Interface requests, we can track every endpoint
>>>>>>> that's disabled. I'll take a guess and assume we're failing at 
>>>>>>> the last
>>>>>>> Set Interface, that means we should have something odd with 
>>>>>>> ep6in, but
>>>>>>> everything looks fine in the trace output:
>>>>>>>
>>>>>>> [   75.823107] irq/14-d-596       0d... 42789194us : 
>>>>>>> dwc3_gadget_ep_enable: ep6in: mps 192/346 streams 16 burst 0 ring 
>>>>>>> 0/0 flags E:swbp:<
>>>>>>> [   75.835472] irq/14-d-596       0d... 42789198us : 
>>>>>>> dwc3_alloc_request: ep6in: req 0000000002c71409 length 0/0 zsI ==> 0
>>>>>>> [   75.846416] irq/14-d-596       0d... 42789202us : 
>>>>>>> dwc3_ep_queue: ep6in: req 0000000002c71409 length 0/192 zsI ==> -115
>>>>>>> [   75.857360] irq/14-d-596       0d... 42789204us : 
>>>>>>> dwc3_alloc_request: ep6in: req 00000000a324f5d0 length 0/0 zsI ==> 0
>>>>>>> [   75.868301] irq/14-d-596       0d... 42789206us : 
>>>>>>> dwc3_ep_queue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -115
>>>>>>> [   75.879244] irq/14-d-596       0d... 42789209us : dwc3_event: 
>>>>>>> event (000020c2): ep0in: Transfer Not Ready [0] (Not Active) 
>>>>>>> [Status Phase]
>>>>>>> [   75.891880] irq/14-d-596       0d... 42789211us : 
>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>> 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>>>>> [   75.989131] irq/14-d-596       0d... 42789224us : 
>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 
>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>> [   76.096261] irq/14-d-596       0d... 42789272us : dwc3_event: 
>>>>>>> event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
>>>>>>> [   76.107834] irq/14-d-596       0d... 42789275us : 
>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>> 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>>>>> [   76.122944] irq/14-d-596       0d... 42789277us : 
>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 zsI 
>>>>>>> ==> 0
>>>>>>> [   76.134160] irq/14-d-596       0d... 42789280us : 
>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>> 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>>>>> [   76.231322] irq/14-d-596       0d... 42789292us : 
>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 
>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>> [   76.297418] kworker/-23        0d... 42789670us : 
>>>>>>> dwc3_ep_queue: ep3in: req 0000000029586135 length 0/96 ZsI ==> -115
>>>>>>> [   76.308278] kworker/-23        0d... 42789695us : 
>>>>>>> dwc3_prepare_trb: ep3in: trb 00000000b81213d6 (E1:D0) buf 
>>>>>>> 0000000003b7a800 size 96 ctrl 00000811 (Hlcs:sC:normal)
>>>>>>> [   76.395294] kworker/-23        0d... 42789707us : 
>>>>>>> dwc3_gadget_ep_cmd: ep3in: cmd 'Update Transfer' [60007] params 
>>>>>>> 00000000 00000000 00000000 --> status: Successful
>>>>>>> [   76.471900] irq/14-d-596       0d... 42789842us : dwc3_event: 
>>>>>>> event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
>>>>>>> [   76.489308] irq/14-d-596       0d... 42789845us : 
>>>>>>> dwc3_ctrl_req: Set Interface(Intf = 5, Alt.Setting = 0)
>>>>>>> [   76.505650] irq/14-d-596       0d... 42789851us : 
>>>>>>> dwc3_ep_dequeue: ep6in: req 0000000002c71409 length 0/192 zsI ==> 
>>>>>>> -115
>>>>>>> [   76.523315] irq/14-d-596       0d... 42789854us : 
>>>>>>> dwc3_gadget_giveback: ep6in: req 0000000002c71409 length 0/192 
>>>>>>> zsI ==> -104
>>>>>>> [   76.541427] irq/14-d-596       0d... 42789857us : 
>>>>>>> dwc3_free_request: ep6in: req 0000000002c71409 length 0/192 zsI 
>>>>>>> ==> -104
>>>>>>> [   76.559267] irq/14-d-596       0d... 42789859us : 
>>>>>>> dwc3_ep_dequeue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> 
>>>>>>> -115
>>>>>>> [   76.576937] irq/14-d-596       0d... 42789861us : 
>>>>>>> dwc3_gadget_giveback: ep6in: req 00000000a324f5d0 length 0/192 
>>>>>>> zsI ==> -104
>>>>>>> [   76.595046] irq/14-d-596       0d... 42789862us : 
>>>>>>> dwc3_free_request: ep6in: req 00000000a324f5d0 length 0/192 zsI 
>>>>>>> ==> -104
>>>>>>> [   76.612892] irq/14-d-596       0d... 42789865us : 
>>>>>>> dwc3_gadget_ep_disable: ep6in: mps 192/346 streams 16 burst 0 
>>>>>>> ring 0/0 flags E:swbp:<
>>>>>>> [   76.665535] irq/14-d-596       0d... 42789873us : dwc3_event: 
>>>>>>> event (000020c2): ep0in: Transfer Not Ready [0] (Not Active) 
>>>>>>> [Status Phase]
>>>>>>> [   76.684716] irq/14-d-596       0d... 42789875us : 
>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>> 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>>>>> [   76.819195] irq/14-d-596       0d... 42789886us : 
>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 
>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>> [   76.926324] irq/14-d-596       0d... 42789930us : dwc3_event: 
>>>>>>> event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
>>>>>>> [   76.937892] irq/14-d-596       0d... 42789933us : 
>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>> 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>>>>> [   76.953003] irq/14-d-596       0d... 42789935us : 
>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 zsI 
>>>>>>> ==> 0
>>>>>>> [   76.964217] irq/14-d-596       0d... 42789938us : 
>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>> 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>>>>> [   77.061379] irq/14-d-596       0d... 42789950us : 
>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 
>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>> [   77.168595] irq/14-d-596       0d... 42790509us : dwc3_event: 
>>>>>>> event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
>>>>>>> [   77.180159] irq/14-d-596       0d... 42790512us : 
>>>>>>> dwc3_ctrl_req: Get String Descriptor(Index = 18, Length = 255)
>>>>>>> [   77.190578] irq/14-d-596       0d... 42790537us : 
>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>> 0000000003b68000 size 36 ctrl 00000c53 (HLcs:SC:data)
>>>>>>> [   77.287648] irq/14-d-596       0d... 42790550us : 
>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 
>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>> [   77.333107] irq/14-d-596       0d... 42790557us : dwc3_event: 
>>>>>>> event (000010c2): ep0in: Transfer Not Ready [0] (Not Active) 
>>>>>>> [Data Phase]
>>>>>>> [   77.407223] irq/14-d-596       0d... 42790575us : dwc3_event: 
>>>>>>> event (000090c2): ep0in: Transfer Not Ready [0] (Active) [Data 
>>>>>>> Phase]
>>>>>>> [   77.480985] irq/14-d-596       0d... 42790588us : dwc3_event: 
>>>>>>> event (0000c042): ep0in: Transfer Complete (sIL) [Data Phase]
>>>>>>> [   77.492376] irq/14-d-596       0d... 42790590us : 
>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>> 0000000003b68000 size 0 ctrl 00000c52 (hLcs:SC:data)
>>>>>>> [   77.507221] irq/14-d-596       0d... 42790595us : 
>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 36/36 
>>>>>>> ZsI ==> 0
>>>>>>> [   77.518609] irq/14-d-596       0d... 42790597us : dwc3_event: 
>>>>>>> event (000020c0): ep0out: Transfer Not Ready [0] (Not Active) 
>>>>>>> [Status Phase]
>>>>>>> [   77.531332] irq/14-d-596       0d... 42790598us : 
>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>> 000000001bded000 size 0 ctrl 00000c43 (HLcs:SC:status3)
>>>>>>> [   77.628669] irq/14-d-596       0d... 42790609us : 
>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 
>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>
>>>>>>> Do you mind adding a few prints in dwc3_remove_requests to tell 
>>>>>>> us which
>>>>>>> endpoint is being processed? Then we'll know for sure which one 
>>>>>>> caused
>>>>>>> the crash.
>>>>>>>
>>>>>> I wouldn't mind but am leaving on a holiday, won't have time until 
>>>>>> 6 aug.
>>>>> not a problem, we'll still be here when you're back :-)
>>>> Well, let's go then :-)
>>>>
>>>> To get back in the mood I have retested 5.13.0, 5.14.0-rc1, 5.14.0-rc2
>>>> and 5.14.0-rc5.
>>>>
>>>> I find that 5.13.0 works fine, and the issue starts from 5.14.0-rc1.
>>> That's great finding. We have a bisection point. There are a total of
>>> 13764 commits between v5.13 and v5.14-rc1
>>>
>>>     $ git rev-list  --count v5.13..v5.14-rc1
>>>     13764
>>>
>>> git bisect should find the offending commit in at most 14 tries. That's
>>> not too bad.
>> I correctly guesstimated that the problem got introduced by the usb 
>> merge 79160a60
>>
>> "Merge tag 'usb-5.14-rc1' of 
>> git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb"
>>
>> 116 commits(7 bisects).
>>
>> 24f779dac8f3efb9629adc0e486914d93dc45517 is the first bad commit
>>
>> "usb: gadget: f_uac2/u_audio: add feedback endpoint support"
>>
>> Ruslan's 3 patches are related to each other so I reverted all three 
>> 24f779da...e89bb428 and applied the reverts to rc1.
>>
>> I can confirm this indeed resolves the problem in rc1.
>>
>> Is late now, tomorrow evening I will apply the reverts to rc6.
> 
> With these reverts rc6 works fine as well.
> 
> So, where do we go from here?
> 

I know the patches have been tested on dwc2 (by me and others).  I do 
not know if Ruslan or Jerome tested them on dwc3 but probably not. 
Ruslan has talked about RPi (my case too) and BeagleboneBlack, both with 
dwc2. Perhaps the dwc2 behaves a bit differently than dwc3?

The patches add a new EP-IN for async feedback. I am sorry I have not 
followed your long thread (it started as unrelated to uac). Does the 
problem appear with f_uac1 or f_uac2? Please how have you reached the 
above problem?

u_audio_stop_capture calls free_ep_fback 
https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/u_audio.c#L414 
which clears the feedback request prm->req_fback.

Then (reading from your the call trace), usb_ep_disable calls 
dwc3_gadget_ep_disable which for every request in the EP calls 
dwc3_remove_requests which gives every found request back to the gadget 
function, IIUC by calling the complete method of the request 
https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/udc/core.c#L912 
. Perhaps the problem is somewhere here?

The dwc2 gadget has its endpoint_disable method a bit different 
https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/dwc2/gadget.c#L4200 
  -> 
https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/dwc2/gadget.c#L3251 
-> 
https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/dwc2/gadget.c#L2090 
which also calls usb_gadget_giveback_request 
https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/dwc2/gadget.c#L2126 
. Perhaps there is some minor but important difference in processing the 
requests when disabling the endpoint between dwc2 and dwc3?

Thanks,

Pavel.

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-08-18 19:07                                       ` Ferry Toth
  2021-08-19  7:51                                         ` Pavel Hofman
@ 2021-08-19  7:55                                         ` Andy Shevchenko
  1 sibling, 0 replies; 73+ messages in thread
From: Andy Shevchenko @ 2021-08-19  7:55 UTC (permalink / raw)
  To: Ferry Toth
  Cc: Felipe Balbi, Wesley Cheng, Greg Kroah-Hartman, Rob Herring,
	Andy Gross, Bjorn Andersson, Frank Rowand, USB,
	Linux Kernel Mailing List, devicetree, linux-arm-msm, Jack Pham,
	Krogerus, Heikki, Ruslan Bilovol

On Wed, Aug 18, 2021 at 10:07 PM Ferry Toth <fntoth@gmail.com> wrote:
> Op 18-08-2021 om 00:00 schreef Ferry Toth:
> > Op 16-08-2021 om 07:18 schreef Felipe Balbi:
> >> Ferry Toth <fntoth@gmail.com> writes:
> >>>> Ferry Toth <fntoth@gmail.com> writes:
> >>>>>>>> Ferry Toth <fntoth@gmail.com> writes:
> >>>>>>>>>>>>> Hardware name: Intel Corporation Merrifield/BODEGA BAY,
> >>>>>>>>>>>>> BIOS 542
> >>>>>>>>>>>>> 2015.01.21:18.19.48
> >>>>>>>>>>>>> RIP: 0010:0x500000000
> >>>>>>>>>>>>> Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
> >>>>>>>>>>>>> RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
> >>>>>>>>>>>>> RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX:
> >>>>>>>>>>>>> 0000000000000000
> >>>>>>>>>>>>> RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI:
> >>>>>>>>>>>>> ffff8cd546aed200
> >>>>>>>>>>>>> RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09:
> >>>>>>>>>>>>> 0000000000000001
> >>>>>>>>>>>>> R10: ffff8cd541fd28c0 R11: 0000000000000000 R12:
> >>>>>>>>>>>>> ffff8cd547342828
> >>>>>>>>>>>>> R13: ffff8cd546aed248 R14: 0000000000000000 R15:
> >>>>>>>>>>>>> ffff8cd548b1d000
> >>>>>>>>>>>>> FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000)
> >>>>>>>>>>>>> knlGS:0000000000000000
> >>>>>>>>>>>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >>>>>>>>>>>>> CR2: 0000000500000000 CR3: 000000000311e000 CR4:
> >>>>>>>>>>>>> 00000000001006f0
> >>>>>>>>>>>>> Call Trace:
> >>>>>>>>>>>>>         ? dwc3_remove_requests.constprop.0+0x14d/0x170
> >>>>>>>>>>>>>         ? __dwc3_gadget_ep_disable+0x7a/0x160
> >>>>>>>>>>>>>         ? dwc3_gadget_ep_disable+0x3d/0xd0
> >>>>>>>>>>>>>         ? usb_ep_disable+0x1c/0x
> >>>>>>>>>>>>>         ? u_audio_stop_capture+0x79/0x120 [u_audio]
> >>>>>>>>>>>>>         ? afunc_set_alt+0x73/0x80 [usb_f_uac2]
> >>>>>> So this is triggered by a SetInterface request...
> >>>>>>
> >>>>>>>>>>>>>         ? composite_setup+0x224/0x1b90 [libcomposite]
> >>>>>>>>>>>>>         ? __dwc3_gadget_kick_transfer+0x160/0x400
> >>>>>>>>>>>>>         ? dwc3_gadget_ep_queue+0xf3/0x1a0
> >>>>>>>>>>>>>         ? configfs_composite_setup+0x6b/0x90 [libcomposite]
> >>>>>>>>>>>>>         ? configfs_composite_setup+0x6b/0x90 [libcomposite]
> >>>>>>>>>>>>>         ? dwc3_ep0_interrupt+0x459/0xa40
> >>>>>>>>>>>>>         ? dwc3_thread_interrupt+0x8ee/0xf40
> >>>>>>>>>>>>>         ? __schedule+0x235/0x6c0
> >>>>>>>>>>>>>         ? disable_irq_nosync+0x10/0x10
> >>>>>>>>>>>>>         ? irq_thread_fn+0x1b/0x60
> >>>>>>>>>>>>>         ? irq_thread+0xc0/0x160
> >>>>>>>>>>>>>         ? irq_thread_check_affinity+0x70/0x70
> >>>>>>>>>>>>>         ? irq_forced_thread_fn+0x70/0x70
> >>>>>>>>>>>>>         ? kthread+0x122/0x140
> >>>>>>>>>>>>>         ? set_kthread_struct+0x40/0x40
> >>>>>>>>>>>>>         ? ret_from_fork+0x22/0x30
> >>>>>>>>>>>> Do you mind enabling dwc3 traces and collecting them?
> >>>>>>>>>>>> Trying to figure
> >>>>>>>>>>>> out how we got here.
> >>>>>>>>>>>>
> >>>>>>>>>>> I'll try if I can get the same error by booting with USB in
> >>>>>>>>>>> host mode
> >>>>>>>>>>> and then switch to device mode. If so I can enable traces
> >>>>>>>>>>> and collect as
> >>>>>>>>>>> you explained me before.
> >>>>>>>>>>>
> >>>>>>>>>>> I'll try before monday, as then I fly for a holiday and will
> >>>>>>>>>>> not be
> >>>>>>>>>>> available before rc5.
> >>>>>>>>>> you can enable all of those with kernel cmdline :-)
> >>>>>>>>>>
> >>>>>>>>>> https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> you need ftrace_dump_on_oops=1 and also need the correct
> >>>>>>>>>> options on
> >>>>>>>>>> trace_buf_size and trace_event.
> >>>>>>>>>>
> >>>>>>>>> On Edison-Arduino I have a switch to go to device mode, after
> >>>>>>>>> which
> >>>>>>>>> udev triggers a script configure gadgets through configfs.
> >>>>>>>>>
> >>>>>>>>> I tried to log following these instructions:
> >>>>>>>>>
> >>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs
> >>>>>>>>> <https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Unfortunately the kernel crashes so badly I can not get to the
> >>>>>>>>> ` cp
> >>>>>>>>> /t/trace /root/trace.txt` line (after a while the watchdog
> >>>>>>>>> kicks).
> >>>>>>>>>
> >>>>>>>>> What to do next?
> >>>>>>>> Pass ftrace_dump_on_oops to kernel cmdline.
> >>>>>>>>
> >>>>>>> No sure if I did this right, on oops everything is pushed to
> >>>>>>> console
> >>>>>>> (115k2 serial), I hope nothing essential is lost.
> >>>>>>>
> >>>>>>> I copied the screen buffer to file see attached.
> >>>>>> Thank you, I bet it took quite a some time :-) Anyway, looking at
> >>>>>> the logs around Set Interface requests, we can track every endpoint
> >>>>>> that's disabled. I'll take a guess and assume we're failing at
> >>>>>> the last
> >>>>>> Set Interface, that means we should have something odd with
> >>>>>> ep6in, but
> >>>>>> everything looks fine in the trace output:
> >>>>>>
> >>>>>> [   75.823107] irq/14-d-596       0d... 42789194us :
> >>>>>> dwc3_gadget_ep_enable: ep6in: mps 192/346 streams 16 burst 0 ring
> >>>>>> 0/0 flags E:swbp:<
> >>>>>> [   75.835472] irq/14-d-596       0d... 42789198us :
> >>>>>> dwc3_alloc_request: ep6in: req 0000000002c71409 length 0/0 zsI ==> 0
> >>>>>> [   75.846416] irq/14-d-596       0d... 42789202us :
> >>>>>> dwc3_ep_queue: ep6in: req 0000000002c71409 length 0/192 zsI ==> -115
> >>>>>> [   75.857360] irq/14-d-596       0d... 42789204us :
> >>>>>> dwc3_alloc_request: ep6in: req 00000000a324f5d0 length 0/0 zsI ==> 0
> >>>>>> [   75.868301] irq/14-d-596       0d... 42789206us :
> >>>>>> dwc3_ep_queue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> -115
> >>>>>> [   75.879244] irq/14-d-596       0d... 42789209us : dwc3_event:
> >>>>>> event (000020c2): ep0in: Transfer Not Ready [0] (Not Active)
> >>>>>> [Status Phase]
> >>>>>> [   75.891880] irq/14-d-596       0d... 42789211us :
> >>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf
> >>>>>> 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
> >>>>>> [   75.989131] irq/14-d-596       0d... 42789224us :
> >>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params
> >>>>>> 00000000 1bded000 00000000 --> status: Successful
> >>>>>> [   76.096261] irq/14-d-596       0d... 42789272us : dwc3_event:
> >>>>>> event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
> >>>>>> [   76.107834] irq/14-d-596       0d... 42789275us :
> >>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
> >>>>>> 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
> >>>>>> [   76.122944] irq/14-d-596       0d... 42789277us :
> >>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 zsI
> >>>>>> ==> 0
> >>>>>> [   76.134160] irq/14-d-596       0d... 42789280us :
> >>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
> >>>>>> 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
> >>>>>> [   76.231322] irq/14-d-596       0d... 42789292us :
> >>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params
> >>>>>> 00000000 1bded000 00000000 --> status: Successful
> >>>>>> [   76.297418] kworker/-23        0d... 42789670us :
> >>>>>> dwc3_ep_queue: ep3in: req 0000000029586135 length 0/96 ZsI ==> -115
> >>>>>> [   76.308278] kworker/-23        0d... 42789695us :
> >>>>>> dwc3_prepare_trb: ep3in: trb 00000000b81213d6 (E1:D0) buf
> >>>>>> 0000000003b7a800 size 96 ctrl 00000811 (Hlcs:sC:normal)
> >>>>>> [   76.395294] kworker/-23        0d... 42789707us :
> >>>>>> dwc3_gadget_ep_cmd: ep3in: cmd 'Update Transfer' [60007] params
> >>>>>> 00000000 00000000 00000000 --> status: Successful
> >>>>>> [   76.471900] irq/14-d-596       0d... 42789842us : dwc3_event:
> >>>>>> event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
> >>>>>> [   76.489308] irq/14-d-596       0d... 42789845us :
> >>>>>> dwc3_ctrl_req: Set Interface(Intf = 5, Alt.Setting = 0)
> >>>>>> [   76.505650] irq/14-d-596       0d... 42789851us :
> >>>>>> dwc3_ep_dequeue: ep6in: req 0000000002c71409 length 0/192 zsI ==>
> >>>>>> -115
> >>>>>> [   76.523315] irq/14-d-596       0d... 42789854us :
> >>>>>> dwc3_gadget_giveback: ep6in: req 0000000002c71409 length 0/192
> >>>>>> zsI ==> -104
> >>>>>> [   76.541427] irq/14-d-596       0d... 42789857us :
> >>>>>> dwc3_free_request: ep6in: req 0000000002c71409 length 0/192 zsI
> >>>>>> ==> -104
> >>>>>> [   76.559267] irq/14-d-596       0d... 42789859us :
> >>>>>> dwc3_ep_dequeue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==>
> >>>>>> -115
> >>>>>> [   76.576937] irq/14-d-596       0d... 42789861us :
> >>>>>> dwc3_gadget_giveback: ep6in: req 00000000a324f5d0 length 0/192
> >>>>>> zsI ==> -104
> >>>>>> [   76.595046] irq/14-d-596       0d... 42789862us :
> >>>>>> dwc3_free_request: ep6in: req 00000000a324f5d0 length 0/192 zsI
> >>>>>> ==> -104
> >>>>>> [   76.612892] irq/14-d-596       0d... 42789865us :
> >>>>>> dwc3_gadget_ep_disable: ep6in: mps 192/346 streams 16 burst 0
> >>>>>> ring 0/0 flags E:swbp:<
> >>>>>> [   76.665535] irq/14-d-596       0d... 42789873us : dwc3_event:
> >>>>>> event (000020c2): ep0in: Transfer Not Ready [0] (Not Active)
> >>>>>> [Status Phase]
> >>>>>> [   76.684716] irq/14-d-596       0d... 42789875us :
> >>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf
> >>>>>> 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
> >>>>>> [   76.819195] irq/14-d-596       0d... 42789886us :
> >>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params
> >>>>>> 00000000 1bded000 00000000 --> status: Successful
> >>>>>> [   76.926324] irq/14-d-596       0d... 42789930us : dwc3_event:
> >>>>>> event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
> >>>>>> [   76.937892] irq/14-d-596       0d... 42789933us :
> >>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
> >>>>>> 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
> >>>>>> [   76.953003] irq/14-d-596       0d... 42789935us :
> >>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 zsI
> >>>>>> ==> 0
> >>>>>> [   76.964217] irq/14-d-596       0d... 42789938us :
> >>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
> >>>>>> 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
> >>>>>> [   77.061379] irq/14-d-596       0d... 42789950us :
> >>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params
> >>>>>> 00000000 1bded000 00000000 --> status: Successful
> >>>>>> [   77.168595] irq/14-d-596       0d... 42790509us : dwc3_event:
> >>>>>> event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
> >>>>>> [   77.180159] irq/14-d-596       0d... 42790512us :
> >>>>>> dwc3_ctrl_req: Get String Descriptor(Index = 18, Length = 255)
> >>>>>> [   77.190578] irq/14-d-596       0d... 42790537us :
> >>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf
> >>>>>> 0000000003b68000 size 36 ctrl 00000c53 (HLcs:SC:data)
> >>>>>> [   77.287648] irq/14-d-596       0d... 42790550us :
> >>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params
> >>>>>> 00000000 1bded000 00000000 --> status: Successful
> >>>>>> [   77.333107] irq/14-d-596       0d... 42790557us : dwc3_event:
> >>>>>> event (000010c2): ep0in: Transfer Not Ready [0] (Not Active)
> >>>>>> [Data Phase]
> >>>>>> [   77.407223] irq/14-d-596       0d... 42790575us : dwc3_event:
> >>>>>> event (000090c2): ep0in: Transfer Not Ready [0] (Active) [Data
> >>>>>> Phase]
> >>>>>> [   77.480985] irq/14-d-596       0d... 42790588us : dwc3_event:
> >>>>>> event (0000c042): ep0in: Transfer Complete (sIL) [Data Phase]
> >>>>>> [   77.492376] irq/14-d-596       0d... 42790590us :
> >>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
> >>>>>> 0000000003b68000 size 0 ctrl 00000c52 (hLcs:SC:data)
> >>>>>> [   77.507221] irq/14-d-596       0d... 42790595us :
> >>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 36/36
> >>>>>> ZsI ==> 0
> >>>>>> [   77.518609] irq/14-d-596       0d... 42790597us : dwc3_event:
> >>>>>> event (000020c0): ep0out: Transfer Not Ready [0] (Not Active)
> >>>>>> [Status Phase]
> >>>>>> [   77.531332] irq/14-d-596       0d... 42790598us :
> >>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
> >>>>>> 000000001bded000 size 0 ctrl 00000c43 (HLcs:SC:status3)
> >>>>>> [   77.628669] irq/14-d-596       0d... 42790609us :
> >>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params
> >>>>>> 00000000 1bded000 00000000 --> status: Successful
> >>>>>>
> >>>>>> Do you mind adding a few prints in dwc3_remove_requests to tell
> >>>>>> us which
> >>>>>> endpoint is being processed? Then we'll know for sure which one
> >>>>>> caused
> >>>>>> the crash.
> >>>>>>
> >>>>> I wouldn't mind but am leaving on a holiday, won't have time until
> >>>>> 6 aug.
> >>>> not a problem, we'll still be here when you're back :-)
> >>> Well, let's go then :-)
> >>>
> >>> To get back in the mood I have retested 5.13.0, 5.14.0-rc1, 5.14.0-rc2
> >>> and 5.14.0-rc5.
> >>>
> >>> I find that 5.13.0 works fine, and the issue starts from 5.14.0-rc1.
> >> That's great finding. We have a bisection point. There are a total of
> >> 13764 commits between v5.13 and v5.14-rc1
> >>
> >>     $ git rev-list  --count v5.13..v5.14-rc1
> >>     13764
> >>
> >> git bisect should find the offending commit in at most 14 tries. That's
> >> not too bad.
> > I correctly guesstimated that the problem got introduced by the usb
> > merge 79160a60
> >
> > "Merge tag 'usb-5.14-rc1' of
> > git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb"
> >
> > 116 commits(7 bisects).
> >
> > 24f779dac8f3efb9629adc0e486914d93dc45517 is the first bad commit
> >
> > "usb: gadget: f_uac2/u_audio: add feedback endpoint support"
> >
> > Ruslan's 3 patches are related to each other so I reverted all three
> > 24f779da...e89bb428 and applied the reverts to rc1.
> >
> > I can confirm this indeed resolves the problem in rc1.
> >
> > Is late now, tomorrow evening I will apply the reverts to rc6.
>
> With these reverts rc6 works fine as well.
>
> So, where do we go from here?

Since it's rc6 to rc7 I think the best is to send a revert series to
handle the regression.

> >>> With 5.14.0-rc5 the problem seems worse (or different?), and just
> >>> disabling uac2 gadget does not prevent the crash. Even disabling gser
> >>> and mass_storage.usb0 as well there is still a crash.
> >>>
> >>> Now I'm not sure how to proceed. Bisect rc1? Or focus on rc5 (rc6?)?
> >> I'd first bisect between 5.13 and v5.14-rc1. Once you find the offending
> >> commit, verify if reverting that on -rc1 works, then verify if reverting
> >> on -rc5 also works :-)

-- 
With Best Regards,
Andy Shevchenko

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-08-19  7:51                                         ` Pavel Hofman
@ 2021-08-19 20:10                                           ` Ferry Toth
  2021-08-19 21:04                                             ` Pavel Hofman
  0 siblings, 1 reply; 73+ messages in thread
From: Ferry Toth @ 2021-08-19 20:10 UTC (permalink / raw)
  To: Pavel Hofman, Felipe Balbi
  Cc: Wesley Cheng, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list, linux-usb, linux-kernel, devicetree, linux-arm-msm,
	jackp, heikki.krogerus, andy.shevchenko, Ruslan Bilovol

Hi

Op 19-08-2021 om 09:51 schreef Pavel Hofman:
> Hi,
> 
> Dne 18. 08. 21 v 21:07 Ferry Toth napsal(a):
>> Hi,
>>
>> Op 18-08-2021 om 00:00 schreef Ferry Toth:
>>> Hi,
>>>
>>> Op 16-08-2021 om 07:18 schreef Felipe Balbi:
>>>> Hi,
>>>>
>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>>>>>>>> Hardware name: Intel Corporation Merrifield/BODEGA BAY, 
>>>>>>>>>>>>>>> BIOS 542
>>>>>>>>>>>>>>> 2015.01.21:18.19.48
>>>>>>>>>>>>>>> RIP: 0010:0x500000000
>>>>>>>>>>>>>>> Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
>>>>>>>>>>>>>>> RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
>>>>>>>>>>>>>>> RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX: 
>>>>>>>>>>>>>>> 0000000000000000
>>>>>>>>>>>>>>> RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI: 
>>>>>>>>>>>>>>> ffff8cd546aed200
>>>>>>>>>>>>>>> RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09: 
>>>>>>>>>>>>>>> 0000000000000001
>>>>>>>>>>>>>>> R10: ffff8cd541fd28c0 R11: 0000000000000000 R12: 
>>>>>>>>>>>>>>> ffff8cd547342828
>>>>>>>>>>>>>>> R13: ffff8cd546aed248 R14: 0000000000000000 R15: 
>>>>>>>>>>>>>>> ffff8cd548b1d000
>>>>>>>>>>>>>>> FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000) 
>>>>>>>>>>>>>>> knlGS:0000000000000000
>>>>>>>>>>>>>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>>>>>>>>>>> CR2: 0000000500000000 CR3: 000000000311e000 CR4: 
>>>>>>>>>>>>>>> 00000000001006f0
>>>>>>>>>>>>>>> Call Trace:
>>>>>>>>>>>>>>>         ? dwc3_remove_requests.constprop.0+0x14d/0x170
>>>>>>>>>>>>>>>         ? __dwc3_gadget_ep_disable+0x7a/0x160
>>>>>>>>>>>>>>>         ? dwc3_gadget_ep_disable+0x3d/0xd0
>>>>>>>>>>>>>>>         ? usb_ep_disable+0x1c/0x
>>>>>>>>>>>>>>>         ? u_audio_stop_capture+0x79/0x120 [u_audio]
>>>>>>>>>>>>>>>         ? afunc_set_alt+0x73/0x80 [usb_f_uac2]
>>>>>>>> So this is triggered by a SetInterface request...
>>>>>>>>
>>>>>>>>>>>>>>>         ? composite_setup+0x224/0x1b90 [libcomposite]
>>>>>>>>>>>>>>>         ? __dwc3_gadget_kick_transfer+0x160/0x400
>>>>>>>>>>>>>>>         ? dwc3_gadget_ep_queue+0xf3/0x1a0
>>>>>>>>>>>>>>>         ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>>>>>>>>>         ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>>>>>>>>>         ? dwc3_ep0_interrupt+0x459/0xa40
>>>>>>>>>>>>>>>         ? dwc3_thread_interrupt+0x8ee/0xf40
>>>>>>>>>>>>>>>         ? __schedule+0x235/0x6c0
>>>>>>>>>>>>>>>         ? disable_irq_nosync+0x10/0x10
>>>>>>>>>>>>>>>         ? irq_thread_fn+0x1b/0x60
>>>>>>>>>>>>>>>         ? irq_thread+0xc0/0x160
>>>>>>>>>>>>>>>         ? irq_thread_check_affinity+0x70/0x70
>>>>>>>>>>>>>>>         ? irq_forced_thread_fn+0x70/0x70
>>>>>>>>>>>>>>>         ? kthread+0x122/0x140
>>>>>>>>>>>>>>>         ? set_kthread_struct+0x40/0x40
>>>>>>>>>>>>>>>         ? ret_from_fork+0x22/0x30
>>>>>>>>>>>>>> Do you mind enabling dwc3 traces and collecting them? 
>>>>>>>>>>>>>> Trying to figure
>>>>>>>>>>>>>> out how we got here.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> I'll try if I can get the same error by booting with USB in 
>>>>>>>>>>>>> host mode
>>>>>>>>>>>>> and then switch to device mode. If so I can enable traces 
>>>>>>>>>>>>> and collect as
>>>>>>>>>>>>> you explained me before.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'll try before monday, as then I fly for a holiday and 
>>>>>>>>>>>>> will not be
>>>>>>>>>>>>> available before rc5.
>>>>>>>>>>>> you can enable all of those with kernel cmdline :-)
>>>>>>>>>>>>
>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html 
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> you need ftrace_dump_on_oops=1 and also need the correct 
>>>>>>>>>>>> options on
>>>>>>>>>>>> trace_buf_size and trace_event.
>>>>>>>>>>>>
>>>>>>>>>>> On Edison-Arduino I have a switch to go to device mode, after 
>>>>>>>>>>> which
>>>>>>>>>>> udev triggers a script configure gadgets through configfs.
>>>>>>>>>>>
>>>>>>>>>>> I tried to log following these instructions:
>>>>>>>>>>>
>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs 
>>>>>>>>>>> <https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs> 
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Unfortunately the kernel crashes so badly I can not get to 
>>>>>>>>>>> the ` cp
>>>>>>>>>>> /t/trace /root/trace.txt` line (after a while the watchdog 
>>>>>>>>>>> kicks).
>>>>>>>>>>>
>>>>>>>>>>> What to do next?
>>>>>>>>>> Pass ftrace_dump_on_oops to kernel cmdline.
>>>>>>>>>>
>>>>>>>>> No sure if I did this right, on oops everything is pushed to 
>>>>>>>>> console
>>>>>>>>> (115k2 serial), I hope nothing essential is lost.
>>>>>>>>>
>>>>>>>>> I copied the screen buffer to file see attached.
>>>>>>>> Thank you, I bet it took quite a some time :-) Anyway, looking at
>>>>>>>> the logs around Set Interface requests, we can track every endpoint
>>>>>>>> that's disabled. I'll take a guess and assume we're failing at 
>>>>>>>> the last
>>>>>>>> Set Interface, that means we should have something odd with 
>>>>>>>> ep6in, but
>>>>>>>> everything looks fine in the trace output:
>>>>>>>>
>>>>>>>> [   75.823107] irq/14-d-596       0d... 42789194us : 
>>>>>>>> dwc3_gadget_ep_enable: ep6in: mps 192/346 streams 16 burst 0 
>>>>>>>> ring 0/0 flags E:swbp:<
>>>>>>>> [   75.835472] irq/14-d-596       0d... 42789198us : 
>>>>>>>> dwc3_alloc_request: ep6in: req 0000000002c71409 length 0/0 zsI 
>>>>>>>> ==> 0
>>>>>>>> [   75.846416] irq/14-d-596       0d... 42789202us : 
>>>>>>>> dwc3_ep_queue: ep6in: req 0000000002c71409 length 0/192 zsI ==> 
>>>>>>>> -115
>>>>>>>> [   75.857360] irq/14-d-596       0d... 42789204us : 
>>>>>>>> dwc3_alloc_request: ep6in: req 00000000a324f5d0 length 0/0 zsI 
>>>>>>>> ==> 0
>>>>>>>> [   75.868301] irq/14-d-596       0d... 42789206us : 
>>>>>>>> dwc3_ep_queue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> 
>>>>>>>> -115
>>>>>>>> [   75.879244] irq/14-d-596       0d... 42789209us : dwc3_event: 
>>>>>>>> event (000020c2): ep0in: Transfer Not Ready [0] (Not Active) 
>>>>>>>> [Status Phase]
>>>>>>>> [   75.891880] irq/14-d-596       0d... 42789211us : 
>>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>> 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>>>>>> [   75.989131] irq/14-d-596       0d... 42789224us : 
>>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 
>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>> [   76.096261] irq/14-d-596       0d... 42789272us : dwc3_event: 
>>>>>>>> event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
>>>>>>>> [   76.107834] irq/14-d-596       0d... 42789275us : 
>>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>> 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>>>>>> [   76.122944] irq/14-d-596       0d... 42789277us : 
>>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 
>>>>>>>> zsI ==> 0
>>>>>>>> [   76.134160] irq/14-d-596       0d... 42789280us : 
>>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>> 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>>>>>> [   76.231322] irq/14-d-596       0d... 42789292us : 
>>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 
>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>> [   76.297418] kworker/-23        0d... 42789670us : 
>>>>>>>> dwc3_ep_queue: ep3in: req 0000000029586135 length 0/96 ZsI ==> -115
>>>>>>>> [   76.308278] kworker/-23        0d... 42789695us : 
>>>>>>>> dwc3_prepare_trb: ep3in: trb 00000000b81213d6 (E1:D0) buf 
>>>>>>>> 0000000003b7a800 size 96 ctrl 00000811 (Hlcs:sC:normal)
>>>>>>>> [   76.395294] kworker/-23        0d... 42789707us : 
>>>>>>>> dwc3_gadget_ep_cmd: ep3in: cmd 'Update Transfer' [60007] params 
>>>>>>>> 00000000 00000000 00000000 --> status: Successful
>>>>>>>> [   76.471900] irq/14-d-596       0d... 42789842us : dwc3_event: 
>>>>>>>> event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
>>>>>>>> [   76.489308] irq/14-d-596       0d... 42789845us : 
>>>>>>>> dwc3_ctrl_req: Set Interface(Intf = 5, Alt.Setting = 0)
>>>>>>>> [   76.505650] irq/14-d-596       0d... 42789851us : 
>>>>>>>> dwc3_ep_dequeue: ep6in: req 0000000002c71409 length 0/192 zsI 
>>>>>>>> ==> -115
>>>>>>>> [   76.523315] irq/14-d-596       0d... 42789854us : 
>>>>>>>> dwc3_gadget_giveback: ep6in: req 0000000002c71409 length 0/192 
>>>>>>>> zsI ==> -104
>>>>>>>> [   76.541427] irq/14-d-596       0d... 42789857us : 
>>>>>>>> dwc3_free_request: ep6in: req 0000000002c71409 length 0/192 zsI 
>>>>>>>> ==> -104
>>>>>>>> [   76.559267] irq/14-d-596       0d... 42789859us : 
>>>>>>>> dwc3_ep_dequeue: ep6in: req 00000000a324f5d0 length 0/192 zsI 
>>>>>>>> ==> -115
>>>>>>>> [   76.576937] irq/14-d-596       0d... 42789861us : 
>>>>>>>> dwc3_gadget_giveback: ep6in: req 00000000a324f5d0 length 0/192 
>>>>>>>> zsI ==> -104
>>>>>>>> [   76.595046] irq/14-d-596       0d... 42789862us : 
>>>>>>>> dwc3_free_request: ep6in: req 00000000a324f5d0 length 0/192 zsI 
>>>>>>>> ==> -104
>>>>>>>> [   76.612892] irq/14-d-596       0d... 42789865us : 
>>>>>>>> dwc3_gadget_ep_disable: ep6in: mps 192/346 streams 16 burst 0 
>>>>>>>> ring 0/0 flags E:swbp:<
>>>>>>>> [   76.665535] irq/14-d-596       0d... 42789873us : dwc3_event: 
>>>>>>>> event (000020c2): ep0in: Transfer Not Ready [0] (Not Active) 
>>>>>>>> [Status Phase]
>>>>>>>> [   76.684716] irq/14-d-596       0d... 42789875us : 
>>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>> 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>>>>>> [   76.819195] irq/14-d-596       0d... 42789886us : 
>>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 
>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>> [   76.926324] irq/14-d-596       0d... 42789930us : dwc3_event: 
>>>>>>>> event (0000c042): ep0in: Transfer Complete (sIL) [Status Phase]
>>>>>>>> [   76.937892] irq/14-d-596       0d... 42789933us : 
>>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>> 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>>>>>> [   76.953003] irq/14-d-596       0d... 42789935us : 
>>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 
>>>>>>>> zsI ==> 0
>>>>>>>> [   76.964217] irq/14-d-596       0d... 42789938us : 
>>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>> 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>>>>>> [   77.061379] irq/14-d-596       0d... 42789950us : 
>>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 
>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>> [   77.168595] irq/14-d-596       0d... 42790509us : dwc3_event: 
>>>>>>>> event (0000c040): ep0out: Transfer Complete (sIL) [Setup Phase]
>>>>>>>> [   77.180159] irq/14-d-596       0d... 42790512us : 
>>>>>>>> dwc3_ctrl_req: Get String Descriptor(Index = 18, Length = 255)
>>>>>>>> [   77.190578] irq/14-d-596       0d... 42790537us : 
>>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>> 0000000003b68000 size 36 ctrl 00000c53 (HLcs:SC:data)
>>>>>>>> [   77.287648] irq/14-d-596       0d... 42790550us : 
>>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 
>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>> [   77.333107] irq/14-d-596       0d... 42790557us : dwc3_event: 
>>>>>>>> event (000010c2): ep0in: Transfer Not Ready [0] (Not Active) 
>>>>>>>> [Data Phase]
>>>>>>>> [   77.407223] irq/14-d-596       0d... 42790575us : dwc3_event: 
>>>>>>>> event (000090c2): ep0in: Transfer Not Ready [0] (Active) [Data 
>>>>>>>> Phase]
>>>>>>>> [   77.480985] irq/14-d-596       0d... 42790588us : dwc3_event: 
>>>>>>>> event (0000c042): ep0in: Transfer Complete (sIL) [Data Phase]
>>>>>>>> [   77.492376] irq/14-d-596       0d... 42790590us : 
>>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>> 0000000003b68000 size 0 ctrl 00000c52 (hLcs:SC:data)
>>>>>>>> [   77.507221] irq/14-d-596       0d... 42790595us : 
>>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 36/36 
>>>>>>>> ZsI ==> 0
>>>>>>>> [   77.518609] irq/14-d-596       0d... 42790597us : dwc3_event: 
>>>>>>>> event (000020c0): ep0out: Transfer Not Ready [0] (Not Active) 
>>>>>>>> [Status Phase]
>>>>>>>> [   77.531332] irq/14-d-596       0d... 42790598us : 
>>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>> 000000001bded000 size 0 ctrl 00000c43 (HLcs:SC:status3)
>>>>>>>> [   77.628669] irq/14-d-596       0d... 42790609us : 
>>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 
>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>
>>>>>>>> Do you mind adding a few prints in dwc3_remove_requests to tell 
>>>>>>>> us which
>>>>>>>> endpoint is being processed? Then we'll know for sure which one 
>>>>>>>> caused
>>>>>>>> the crash.
>>>>>>>>
>>>>>>> I wouldn't mind but am leaving on a holiday, won't have time 
>>>>>>> until 6 aug.
>>>>>> not a problem, we'll still be here when you're back :-)
>>>>> Well, let's go then :-)
>>>>>
>>>>> To get back in the mood I have retested 5.13.0, 5.14.0-rc1, 5.14.0-rc2
>>>>> and 5.14.0-rc5.
>>>>>
>>>>> I find that 5.13.0 works fine, and the issue starts from 5.14.0-rc1.
>>>> That's great finding. We have a bisection point. There are a total of
>>>> 13764 commits between v5.13 and v5.14-rc1
>>>>
>>>>     $ git rev-list  --count v5.13..v5.14-rc1
>>>>     13764
>>>>
>>>> git bisect should find the offending commit in at most 14 tries. That's
>>>> not too bad.
>>> I correctly guesstimated that the problem got introduced by the usb 
>>> merge 79160a60
>>>
>>> "Merge tag 'usb-5.14-rc1' of 
>>> git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb"
>>>
>>> 116 commits(7 bisects).
>>>
>>> 24f779dac8f3efb9629adc0e486914d93dc45517 is the first bad commit
>>>
>>> "usb: gadget: f_uac2/u_audio: add feedback endpoint support"
>>>
>>> Ruslan's 3 patches are related to each other so I reverted all three 
>>> 24f779da...e89bb428 and applied the reverts to rc1.
>>>
>>> I can confirm this indeed resolves the problem in rc1.
>>>
>>> Is late now, tomorrow evening I will apply the reverts to rc6.
>>
>> With these reverts rc6 works fine as well.
>>
>> So, where do we go from here?
>>
> 
> I know the patches have been tested on dwc2 (by me and others).  I do 
> not know if Ruslan or Jerome tested them on dwc3 but probably not. 
> Ruslan has talked about RPi (my case too) and BeagleboneBlack, both with 
> dwc2. Perhaps the dwc2 behaves a bit differently than dwc3?
> 
> The patches add a new EP-IN for async feedback. I am sorry I have not 
> followed your long thread (it started as unrelated to uac). Does the 
> problem appear with f_uac1 or f_uac2? Please how have you reached the 
> above problem?

I'm sorry too. I first believed the issue was related to the patch 
mentioned in the subject line.

The problem appaers with f_uac2. I bost Edison_Arduino board in host 
mode (there is a switch allowing to select host/device mode). When 
flipping the switch to device mode udev run a script:
But as I am using configfs (excerpt follows) and just disabling the last 
2 line resolves the issue, I'm guessing uac2 is the issue. Or exceeding 
the available resources.

# Create directory structure
mkdir "${GADGET_BASE_DIR}"
cd "${GADGET_BASE_DIR}"
mkdir -p configs/c.1/strings/0x409
mkdir -p strings/0x409

# Serial device
mkdir functions/gser.usb0
ln -s functions/gser.usb0 configs/c.1/
###

# Ethernet device
mkdir functions/eem.usb0
echo "${DEV_ETH_ADDR}" > functions/eem.usb0/dev_addr
echo "${HOST_ETH_ADDR}" > functions/eem.usb0/host_addr
ln -s functions/eem.usb0 configs/c.1/

# Mass Storage device
mkdir functions/mass_storage.usb0
echo 1 > functions/mass_storage.usb0/stall
echo 0 > functions/mass_storage.usb0/lun.0/cdrom
echo 0 > functions/mass_storage.usb0/lun.0/ro
echo 0 > functions/mass_storage.usb0/lun.0/nofua
echo "${USBDISK}" > functions/mass_storage.usb0/lun.0/file
ln -s functions/mass_storage.usb0 configs/c.1/

# UAC2 device
mkdir functions/uac2.usb0
ln -s functions/uac2.usb0 configs/c.1
....


> u_audio_stop_capture calls free_ep_fback 
> https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/u_audio.c#L414 
> which clears the feedback request prm->req_fback.
> 
> Then (reading from your the call trace), usb_ep_disable calls 
> dwc3_gadget_ep_disable which for every request in the EP calls 
> dwc3_remove_requests which gives every found request back to the gadget 
> function, IIUC by calling the complete method of the request 
> https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/udc/core.c#L912 
> . Perhaps the problem is somewhere here?
> 
> The dwc2 gadget has its endpoint_disable method a bit different 
> https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/dwc2/gadget.c#L4200 
>   -> 
> https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/dwc2/gadget.c#L3251 
> -> 
> https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/dwc2/gadget.c#L2090 
> which also calls usb_gadget_giveback_request 
> https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/dwc2/gadget.c#L2126 
> . Perhaps there is some minor but important difference in processing the 
> requests when disabling the endpoint between dwc2 and dwc3?
> 
> Thanks,
> 
> Pavel.

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-08-19 20:10                                           ` Ferry Toth
@ 2021-08-19 21:04                                             ` Pavel Hofman
  2021-08-21  2:57                                               ` Thinh Nguyen
  2021-08-22 19:43                                               ` Ferry Toth
  0 siblings, 2 replies; 73+ messages in thread
From: Pavel Hofman @ 2021-08-19 21:04 UTC (permalink / raw)
  To: Ferry Toth, Felipe Balbi
  Cc: Wesley Cheng, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list, linux-usb, linux-kernel, devicetree, linux-arm-msm,
	jackp, heikki.krogerus, andy.shevchenko, Ruslan Bilovol


Dne 19. 08. 21 v 22:10 Ferry Toth napsal(a):
> Hi
> 
> Op 19-08-2021 om 09:51 schreef Pavel Hofman:
>> Hi,
>>
>> Dne 18. 08. 21 v 21:07 Ferry Toth napsal(a):
>>> Hi,
>>>
>>> Op 18-08-2021 om 00:00 schreef Ferry Toth:
>>>> Hi,
>>>>
>>>> Op 16-08-2021 om 07:18 schreef Felipe Balbi:
>>>>> Hi,
>>>>>
>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>>>>>>>>> Hardware name: Intel Corporation Merrifield/BODEGA BAY, 
>>>>>>>>>>>>>>>> BIOS 542
>>>>>>>>>>>>>>>> 2015.01.21:18.19.48
>>>>>>>>>>>>>>>> RIP: 0010:0x500000000
>>>>>>>>>>>>>>>> Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
>>>>>>>>>>>>>>>> RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
>>>>>>>>>>>>>>>> RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX: 
>>>>>>>>>>>>>>>> 0000000000000000
>>>>>>>>>>>>>>>> RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI: 
>>>>>>>>>>>>>>>> ffff8cd546aed200
>>>>>>>>>>>>>>>> RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09: 
>>>>>>>>>>>>>>>> 0000000000000001
>>>>>>>>>>>>>>>> R10: ffff8cd541fd28c0 R11: 0000000000000000 R12: 
>>>>>>>>>>>>>>>> ffff8cd547342828
>>>>>>>>>>>>>>>> R13: ffff8cd546aed248 R14: 0000000000000000 R15: 
>>>>>>>>>>>>>>>> ffff8cd548b1d000
>>>>>>>>>>>>>>>> FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000) 
>>>>>>>>>>>>>>>> knlGS:0000000000000000
>>>>>>>>>>>>>>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>>>>>>>>>>>> CR2: 0000000500000000 CR3: 000000000311e000 CR4: 
>>>>>>>>>>>>>>>> 00000000001006f0
>>>>>>>>>>>>>>>> Call Trace:
>>>>>>>>>>>>>>>>         ? dwc3_remove_requests.constprop.0+0x14d/0x170
>>>>>>>>>>>>>>>>         ? __dwc3_gadget_ep_disable+0x7a/0x160
>>>>>>>>>>>>>>>>         ? dwc3_gadget_ep_disable+0x3d/0xd0
>>>>>>>>>>>>>>>>         ? usb_ep_disable+0x1c/0x
>>>>>>>>>>>>>>>>         ? u_audio_stop_capture+0x79/0x120 [u_audio]
>>>>>>>>>>>>>>>>         ? afunc_set_alt+0x73/0x80 [usb_f_uac2]
>>>>>>>>> So this is triggered by a SetInterface request...
>>>>>>>>>
>>>>>>>>>>>>>>>>         ? composite_setup+0x224/0x1b90 [libcomposite]
>>>>>>>>>>>>>>>>         ? __dwc3_gadget_kick_transfer+0x160/0x400
>>>>>>>>>>>>>>>>         ? dwc3_gadget_ep_queue+0xf3/0x1a0
>>>>>>>>>>>>>>>>         ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>>>>>>>>>>         ? configfs_composite_setup+0x6b/0x90 [libcomposite]
>>>>>>>>>>>>>>>>         ? dwc3_ep0_interrupt+0x459/0xa40
>>>>>>>>>>>>>>>>         ? dwc3_thread_interrupt+0x8ee/0xf40
>>>>>>>>>>>>>>>>         ? __schedule+0x235/0x6c0
>>>>>>>>>>>>>>>>         ? disable_irq_nosync+0x10/0x10
>>>>>>>>>>>>>>>>         ? irq_thread_fn+0x1b/0x60
>>>>>>>>>>>>>>>>         ? irq_thread+0xc0/0x160
>>>>>>>>>>>>>>>>         ? irq_thread_check_affinity+0x70/0x70
>>>>>>>>>>>>>>>>         ? irq_forced_thread_fn+0x70/0x70
>>>>>>>>>>>>>>>>         ? kthread+0x122/0x140
>>>>>>>>>>>>>>>>         ? set_kthread_struct+0x40/0x40
>>>>>>>>>>>>>>>>         ? ret_from_fork+0x22/0x30
>>>>>>>>>>>>>>> Do you mind enabling dwc3 traces and collecting them? 
>>>>>>>>>>>>>>> Trying to figure
>>>>>>>>>>>>>>> out how we got here.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'll try if I can get the same error by booting with USB 
>>>>>>>>>>>>>> in host mode
>>>>>>>>>>>>>> and then switch to device mode. If so I can enable traces 
>>>>>>>>>>>>>> and collect as
>>>>>>>>>>>>>> you explained me before.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'll try before monday, as then I fly for a holiday and 
>>>>>>>>>>>>>> will not be
>>>>>>>>>>>>>> available before rc5.
>>>>>>>>>>>>> you can enable all of those with kernel cmdline :-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html 
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> you need ftrace_dump_on_oops=1 and also need the correct 
>>>>>>>>>>>>> options on
>>>>>>>>>>>>> trace_buf_size and trace_event.
>>>>>>>>>>>>>
>>>>>>>>>>>> On Edison-Arduino I have a switch to go to device mode, 
>>>>>>>>>>>> after which
>>>>>>>>>>>> udev triggers a script configure gadgets through configfs.
>>>>>>>>>>>>
>>>>>>>>>>>> I tried to log following these instructions:
>>>>>>>>>>>>
>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs 
>>>>>>>>>>>> <https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs> 
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Unfortunately the kernel crashes so badly I can not get to 
>>>>>>>>>>>> the ` cp
>>>>>>>>>>>> /t/trace /root/trace.txt` line (after a while the watchdog 
>>>>>>>>>>>> kicks).
>>>>>>>>>>>>
>>>>>>>>>>>> What to do next?
>>>>>>>>>>> Pass ftrace_dump_on_oops to kernel cmdline.
>>>>>>>>>>>
>>>>>>>>>> No sure if I did this right, on oops everything is pushed to 
>>>>>>>>>> console
>>>>>>>>>> (115k2 serial), I hope nothing essential is lost.
>>>>>>>>>>
>>>>>>>>>> I copied the screen buffer to file see attached.
>>>>>>>>> Thank you, I bet it took quite a some time :-) Anyway, looking at
>>>>>>>>> the logs around Set Interface requests, we can track every 
>>>>>>>>> endpoint
>>>>>>>>> that's disabled. I'll take a guess and assume we're failing at 
>>>>>>>>> the last
>>>>>>>>> Set Interface, that means we should have something odd with 
>>>>>>>>> ep6in, but
>>>>>>>>> everything looks fine in the trace output:
>>>>>>>>>
>>>>>>>>> [   75.823107] irq/14-d-596       0d... 42789194us : 
>>>>>>>>> dwc3_gadget_ep_enable: ep6in: mps 192/346 streams 16 burst 0 
>>>>>>>>> ring 0/0 flags E:swbp:<
>>>>>>>>> [   75.835472] irq/14-d-596       0d... 42789198us : 
>>>>>>>>> dwc3_alloc_request: ep6in: req 0000000002c71409 length 0/0 zsI 
>>>>>>>>> ==> 0
>>>>>>>>> [   75.846416] irq/14-d-596       0d... 42789202us : 
>>>>>>>>> dwc3_ep_queue: ep6in: req 0000000002c71409 length 0/192 zsI ==> 
>>>>>>>>> -115
>>>>>>>>> [   75.857360] irq/14-d-596       0d... 42789204us : 
>>>>>>>>> dwc3_alloc_request: ep6in: req 00000000a324f5d0 length 0/0 zsI 
>>>>>>>>> ==> 0
>>>>>>>>> [   75.868301] irq/14-d-596       0d... 42789206us : 
>>>>>>>>> dwc3_ep_queue: ep6in: req 00000000a324f5d0 length 0/192 zsI ==> 
>>>>>>>>> -115
>>>>>>>>> [   75.879244] irq/14-d-596       0d... 42789209us : 
>>>>>>>>> dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0] 
>>>>>>>>> (Not Active) [Status Phase]
>>>>>>>>> [   75.891880] irq/14-d-596       0d... 42789211us : 
>>>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>> 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>>>>>>> [   75.989131] irq/14-d-596       0d... 42789224us : 
>>>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 
>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>> [   76.096261] irq/14-d-596       0d... 42789272us : 
>>>>>>>>> dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) 
>>>>>>>>> [Status Phase]
>>>>>>>>> [   76.107834] irq/14-d-596       0d... 42789275us : 
>>>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>> 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>>>>>>> [   76.122944] irq/14-d-596       0d... 42789277us : 
>>>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 
>>>>>>>>> zsI ==> 0
>>>>>>>>> [   76.134160] irq/14-d-596       0d... 42789280us : 
>>>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>> 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>>>>>>> [   76.231322] irq/14-d-596       0d... 42789292us : 
>>>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 
>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>> [   76.297418] kworker/-23        0d... 42789670us : 
>>>>>>>>> dwc3_ep_queue: ep3in: req 0000000029586135 length 0/96 ZsI ==> 
>>>>>>>>> -115
>>>>>>>>> [   76.308278] kworker/-23        0d... 42789695us : 
>>>>>>>>> dwc3_prepare_trb: ep3in: trb 00000000b81213d6 (E1:D0) buf 
>>>>>>>>> 0000000003b7a800 size 96 ctrl 00000811 (Hlcs:sC:normal)
>>>>>>>>> [   76.395294] kworker/-23        0d... 42789707us : 
>>>>>>>>> dwc3_gadget_ep_cmd: ep3in: cmd 'Update Transfer' [60007] params 
>>>>>>>>> 00000000 00000000 00000000 --> status: Successful
>>>>>>>>> [   76.471900] irq/14-d-596       0d... 42789842us : 
>>>>>>>>> dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL) 
>>>>>>>>> [Setup Phase]
>>>>>>>>> [   76.489308] irq/14-d-596       0d... 42789845us : 
>>>>>>>>> dwc3_ctrl_req: Set Interface(Intf = 5, Alt.Setting = 0)
>>>>>>>>> [   76.505650] irq/14-d-596       0d... 42789851us : 
>>>>>>>>> dwc3_ep_dequeue: ep6in: req 0000000002c71409 length 0/192 zsI 
>>>>>>>>> ==> -115
>>>>>>>>> [   76.523315] irq/14-d-596       0d... 42789854us : 
>>>>>>>>> dwc3_gadget_giveback: ep6in: req 0000000002c71409 length 0/192 
>>>>>>>>> zsI ==> -104
>>>>>>>>> [   76.541427] irq/14-d-596       0d... 42789857us : 
>>>>>>>>> dwc3_free_request: ep6in: req 0000000002c71409 length 0/192 zsI 
>>>>>>>>> ==> -104
>>>>>>>>> [   76.559267] irq/14-d-596       0d... 42789859us : 
>>>>>>>>> dwc3_ep_dequeue: ep6in: req 00000000a324f5d0 length 0/192 zsI 
>>>>>>>>> ==> -115
>>>>>>>>> [   76.576937] irq/14-d-596       0d... 42789861us : 
>>>>>>>>> dwc3_gadget_giveback: ep6in: req 00000000a324f5d0 length 0/192 
>>>>>>>>> zsI ==> -104
>>>>>>>>> [   76.595046] irq/14-d-596       0d... 42789862us : 
>>>>>>>>> dwc3_free_request: ep6in: req 00000000a324f5d0 length 0/192 zsI 
>>>>>>>>> ==> -104
>>>>>>>>> [   76.612892] irq/14-d-596       0d... 42789865us : 
>>>>>>>>> dwc3_gadget_ep_disable: ep6in: mps 192/346 streams 16 burst 0 
>>>>>>>>> ring 0/0 flags E:swbp:<
>>>>>>>>> [   76.665535] irq/14-d-596       0d... 42789873us : 
>>>>>>>>> dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0] 
>>>>>>>>> (Not Active) [Status Phase]
>>>>>>>>> [   76.684716] irq/14-d-596       0d... 42789875us : 
>>>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>> 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>>>>>>> [   76.819195] irq/14-d-596       0d... 42789886us : 
>>>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 
>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>> [   76.926324] irq/14-d-596       0d... 42789930us : 
>>>>>>>>> dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) 
>>>>>>>>> [Status Phase]
>>>>>>>>> [   76.937892] irq/14-d-596       0d... 42789933us : 
>>>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>> 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>>>>>>> [   76.953003] irq/14-d-596       0d... 42789935us : 
>>>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 
>>>>>>>>> zsI ==> 0
>>>>>>>>> [   76.964217] irq/14-d-596       0d... 42789938us : 
>>>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>> 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>>>>>>> [   77.061379] irq/14-d-596       0d... 42789950us : 
>>>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 
>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>> [   77.168595] irq/14-d-596       0d... 42790509us : 
>>>>>>>>> dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL) 
>>>>>>>>> [Setup Phase]
>>>>>>>>> [   77.180159] irq/14-d-596       0d... 42790512us : 
>>>>>>>>> dwc3_ctrl_req: Get String Descriptor(Index = 18, Length = 255)
>>>>>>>>> [   77.190578] irq/14-d-596       0d... 42790537us : 
>>>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>> 0000000003b68000 size 36 ctrl 00000c53 (HLcs:SC:data)
>>>>>>>>> [   77.287648] irq/14-d-596       0d... 42790550us : 
>>>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 
>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>> [   77.333107] irq/14-d-596       0d... 42790557us : 
>>>>>>>>> dwc3_event: event (000010c2): ep0in: Transfer Not Ready [0] 
>>>>>>>>> (Not Active) [Data Phase]
>>>>>>>>> [   77.407223] irq/14-d-596       0d... 42790575us : 
>>>>>>>>> dwc3_event: event (000090c2): ep0in: Transfer Not Ready [0] 
>>>>>>>>> (Active) [Data Phase]
>>>>>>>>> [   77.480985] irq/14-d-596       0d... 42790588us : 
>>>>>>>>> dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) 
>>>>>>>>> [Data Phase]
>>>>>>>>> [   77.492376] irq/14-d-596       0d... 42790590us : 
>>>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>> 0000000003b68000 size 0 ctrl 00000c52 (hLcs:SC:data)
>>>>>>>>> [   77.507221] irq/14-d-596       0d... 42790595us : 
>>>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 36/36 
>>>>>>>>> ZsI ==> 0
>>>>>>>>> [   77.518609] irq/14-d-596       0d... 42790597us : 
>>>>>>>>> dwc3_event: event (000020c0): ep0out: Transfer Not Ready [0] 
>>>>>>>>> (Not Active) [Status Phase]
>>>>>>>>> [   77.531332] irq/14-d-596       0d... 42790598us : 
>>>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>> 000000001bded000 size 0 ctrl 00000c43 (HLcs:SC:status3)
>>>>>>>>> [   77.628669] irq/14-d-596       0d... 42790609us : 
>>>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 
>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>
>>>>>>>>> Do you mind adding a few prints in dwc3_remove_requests to tell 
>>>>>>>>> us which
>>>>>>>>> endpoint is being processed? Then we'll know for sure which one 
>>>>>>>>> caused
>>>>>>>>> the crash.
>>>>>>>>>
>>>>>>>> I wouldn't mind but am leaving on a holiday, won't have time 
>>>>>>>> until 6 aug.
>>>>>>> not a problem, we'll still be here when you're back :-)
>>>>>> Well, let's go then :-)
>>>>>>
>>>>>> To get back in the mood I have retested 5.13.0, 5.14.0-rc1, 
>>>>>> 5.14.0-rc2
>>>>>> and 5.14.0-rc5.
>>>>>>
>>>>>> I find that 5.13.0 works fine, and the issue starts from 5.14.0-rc1.
>>>>> That's great finding. We have a bisection point. There are a total of
>>>>> 13764 commits between v5.13 and v5.14-rc1
>>>>>
>>>>>     $ git rev-list  --count v5.13..v5.14-rc1
>>>>>     13764
>>>>>
>>>>> git bisect should find the offending commit in at most 14 tries. 
>>>>> That's
>>>>> not too bad.
>>>> I correctly guesstimated that the problem got introduced by the usb 
>>>> merge 79160a60
>>>>
>>>> "Merge tag 'usb-5.14-rc1' of 
>>>> git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb"
>>>>
>>>> 116 commits(7 bisects).
>>>>
>>>> 24f779dac8f3efb9629adc0e486914d93dc45517 is the first bad commit
>>>>
>>>> "usb: gadget: f_uac2/u_audio: add feedback endpoint support"
>>>>
>>>> Ruslan's 3 patches are related to each other so I reverted all three 
>>>> 24f779da...e89bb428 and applied the reverts to rc1.
>>>>
>>>> I can confirm this indeed resolves the problem in rc1.
>>>>
>>>> Is late now, tomorrow evening I will apply the reverts to rc6.
>>>
>>> With these reverts rc6 works fine as well.
>>>
>>> So, where do we go from here?
>>>
>>
>> I know the patches have been tested on dwc2 (by me and others).  I do 
>> not know if Ruslan or Jerome tested them on dwc3 but probably not. 
>> Ruslan has talked about RPi (my case too) and BeagleboneBlack, both 
>> with dwc2. Perhaps the dwc2 behaves a bit differently than dwc3?
>>
>> The patches add a new EP-IN for async feedback. I am sorry I have not 
>> followed your long thread (it started as unrelated to uac). Does the 
>> problem appear with f_uac1 or f_uac2? Please how have you reached the 
>> above problem?
> 
> I'm sorry too. I first believed the issue was related to the patch 
> mentioned in the subject line.
> 
> The problem appaers with f_uac2. I bost Edison_Arduino board in host 
> mode (there is a switch allowing to select host/device mode). When 
> flipping the switch to device mode udev run a script:
> But as I am using configfs (excerpt follows) and just disabling the last 
> 2 line resolves the issue, I'm guessing uac2 is the issue. Or exceeding 
> the available resources.
> 
> # Create directory structure
> mkdir "${GADGET_BASE_DIR}"
> cd "${GADGET_BASE_DIR}"
> mkdir -p configs/c.1/strings/0x409
> mkdir -p strings/0x409
> 
> # Serial device
> mkdir functions/gser.usb0
> ln -s functions/gser.usb0 configs/c.1/
> ###
> 
> # Ethernet device
> mkdir functions/eem.usb0
> echo "${DEV_ETH_ADDR}" > functions/eem.usb0/dev_addr
> echo "${HOST_ETH_ADDR}" > functions/eem.usb0/host_addr
> ln -s functions/eem.usb0 configs/c.1/
> 
> # Mass Storage device
> mkdir functions/mass_storage.usb0
> echo 1 > functions/mass_storage.usb0/stall
> echo 0 > functions/mass_storage.usb0/lun.0/cdrom
> echo 0 > functions/mass_storage.usb0/lun.0/ro
> echo 0 > functions/mass_storage.usb0/lun.0/nofua
> echo "${USBDISK}" > functions/mass_storage.usb0/lun.0/file
> ln -s functions/mass_storage.usb0 configs/c.1/
> 
> # UAC2 device
> mkdir functions/uac2.usb0
> ln -s functions/uac2.usb0 configs/c.1
> ....
> 

As you say, could perhaps the reason be that the extra EP-IN added in 
those patches (previously 1, now 2 with the default config you use) 
exceeds your EP-IN max count or available fifos somehow?  You have a 
number of functions initialized. If you change the load order of the 
functions, do you get the error later with a different function? Just 
guessing...

You should be able to switch the default async EP-OUT (which configures 
the new feedback EP-IN ) to adaptive EP-OUT (which requires no feedback 
EP) with c_sync=8 parameter of f_uac2.

https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c#L47

https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c#L830

https://elixir.bootlin.com/linux/v5.14-rc6/source/include/uapi/linux/usb/ch9.h#L453

Does that fix the problem?

Thanks,

Pavel.

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-08-19 21:04                                             ` Pavel Hofman
@ 2021-08-21  2:57                                               ` Thinh Nguyen
  2021-08-22 20:10                                                 ` Ferry Toth
  2021-09-02  0:28                                                 ` Jack Pham
  2021-08-22 19:43                                               ` Ferry Toth
  1 sibling, 2 replies; 73+ messages in thread
From: Thinh Nguyen @ 2021-08-21  2:57 UTC (permalink / raw)
  To: Pavel Hofman, Ferry Toth, Felipe Balbi
  Cc: Wesley Cheng, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list, linux-usb, linux-kernel, devicetree, linux-arm-msm,
	jackp, heikki.krogerus, andy.shevchenko, Ruslan Bilovol

Pavel Hofman wrote:
> 
> Dne 19. 08. 21 v 22:10 Ferry Toth napsal(a):
>> Hi
>>
>> Op 19-08-2021 om 09:51 schreef Pavel Hofman:
>>> Hi,
>>>
>>> Dne 18. 08. 21 v 21:07 Ferry Toth napsal(a):
>>>> Hi,
>>>>
>>>> Op 18-08-2021 om 00:00 schreef Ferry Toth:
>>>>> Hi,
>>>>>
>>>>> Op 16-08-2021 om 07:18 schreef Felipe Balbi:
>>>>>> Hi,
>>>>>>
>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>>>>>>>>>> Hardware name: Intel Corporation Merrifield/BODEGA BAY,
>>>>>>>>>>>>>>>>> BIOS 542
>>>>>>>>>>>>>>>>> 2015.01.21:18.19.48
>>>>>>>>>>>>>>>>> RIP: 0010:0x500000000
>>>>>>>>>>>>>>>>> Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
>>>>>>>>>>>>>>>>> RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
>>>>>>>>>>>>>>>>> RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX:
>>>>>>>>>>>>>>>>> 0000000000000000
>>>>>>>>>>>>>>>>> RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI:
>>>>>>>>>>>>>>>>> ffff8cd546aed200
>>>>>>>>>>>>>>>>> RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09:
>>>>>>>>>>>>>>>>> 0000000000000001
>>>>>>>>>>>>>>>>> R10: ffff8cd541fd28c0 R11: 0000000000000000 R12:
>>>>>>>>>>>>>>>>> ffff8cd547342828
>>>>>>>>>>>>>>>>> R13: ffff8cd546aed248 R14: 0000000000000000 R15:
>>>>>>>>>>>>>>>>> ffff8cd548b1d000
>>>>>>>>>>>>>>>>> FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000)
>>>>>>>>>>>>>>>>> knlGS:0000000000000000
>>>>>>>>>>>>>>>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>>>>>>>>>>>>> CR2: 0000000500000000 CR3: 000000000311e000 CR4:
>>>>>>>>>>>>>>>>> 00000000001006f0
>>>>>>>>>>>>>>>>> Call Trace:
>>>>>>>>>>>>>>>>>         ? dwc3_remove_requests.constprop.0+0x14d/0x170
>>>>>>>>>>>>>>>>>         ? __dwc3_gadget_ep_disable+0x7a/0x160
>>>>>>>>>>>>>>>>>         ? dwc3_gadget_ep_disable+0x3d/0xd0
>>>>>>>>>>>>>>>>>         ? usb_ep_disable+0x1c/0x
>>>>>>>>>>>>>>>>>         ? u_audio_stop_capture+0x79/0x120 [u_audio]
>>>>>>>>>>>>>>>>>         ? afunc_set_alt+0x73/0x80 [usb_f_uac2]
>>>>>>>>>> So this is triggered by a SetInterface request...
>>>>>>>>>>
>>>>>>>>>>>>>>>>>         ? composite_setup+0x224/0x1b90 [libcomposite]
>>>>>>>>>>>>>>>>>         ? __dwc3_gadget_kick_transfer+0x160/0x400
>>>>>>>>>>>>>>>>>         ? dwc3_gadget_ep_queue+0xf3/0x1a0
>>>>>>>>>>>>>>>>>         ? configfs_composite_setup+0x6b/0x90
>>>>>>>>>>>>>>>>> [libcomposite]
>>>>>>>>>>>>>>>>>         ? configfs_composite_setup+0x6b/0x90
>>>>>>>>>>>>>>>>> [libcomposite]
>>>>>>>>>>>>>>>>>         ? dwc3_ep0_interrupt+0x459/0xa40
>>>>>>>>>>>>>>>>>         ? dwc3_thread_interrupt+0x8ee/0xf40
>>>>>>>>>>>>>>>>>         ? __schedule+0x235/0x6c0
>>>>>>>>>>>>>>>>>         ? disable_irq_nosync+0x10/0x10
>>>>>>>>>>>>>>>>>         ? irq_thread_fn+0x1b/0x60
>>>>>>>>>>>>>>>>>         ? irq_thread+0xc0/0x160
>>>>>>>>>>>>>>>>>         ? irq_thread_check_affinity+0x70/0x70
>>>>>>>>>>>>>>>>>         ? irq_forced_thread_fn+0x70/0x70
>>>>>>>>>>>>>>>>>         ? kthread+0x122/0x140
>>>>>>>>>>>>>>>>>         ? set_kthread_struct+0x40/0x40
>>>>>>>>>>>>>>>>>         ? ret_from_fork+0x22/0x30
>>>>>>>>>>>>>>>> Do you mind enabling dwc3 traces and collecting them?
>>>>>>>>>>>>>>>> Trying to figure
>>>>>>>>>>>>>>>> out how we got here.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'll try if I can get the same error by booting with USB
>>>>>>>>>>>>>>> in host mode
>>>>>>>>>>>>>>> and then switch to device mode. If so I can enable traces
>>>>>>>>>>>>>>> and collect as
>>>>>>>>>>>>>>> you explained me before.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'll try before monday, as then I fly for a holiday and
>>>>>>>>>>>>>>> will not be
>>>>>>>>>>>>>>> available before rc5.
>>>>>>>>>>>>>> you can enable all of those with kernel cmdline :-)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://urldefense.com/v3/__https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html__;!!A4F2R9G_pg!PZHOWnH3HDs-9a9bz4KLDShZpsubSfG84fu3wEZyK4Q_Wvv2lhHKuk7LJpWPAPch0qja$ 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> you need ftrace_dump_on_oops=1 and also need the correct
>>>>>>>>>>>>>> options on
>>>>>>>>>>>>>> trace_buf_size and trace_event.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> On Edison-Arduino I have a switch to go to device mode,
>>>>>>>>>>>>> after which
>>>>>>>>>>>>> udev triggers a script configure gadgets through configfs.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I tried to log following these instructions:
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://urldefense.com/v3/__https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html*reporting-bugs__;Iw!!A4F2R9G_pg!PZHOWnH3HDs-9a9bz4KLDShZpsubSfG84fu3wEZyK4Q_Wvv2lhHKuk7LJpWPAG64agnA$ 
>>>>>>>>>>>>> <https://urldefense.com/v3/__https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html*reporting-bugs__;Iw!!A4F2R9G_pg!PZHOWnH3HDs-9a9bz4KLDShZpsubSfG84fu3wEZyK4Q_Wvv2lhHKuk7LJpWPAG64agnA$
>>>>>>>>>>>>> >
>>>>>>>>>>>>>
>>>>>>>>>>>>> Unfortunately the kernel crashes so badly I can not get to
>>>>>>>>>>>>> the ` cp
>>>>>>>>>>>>> /t/trace /root/trace.txt` line (after a while the watchdog
>>>>>>>>>>>>> kicks).
>>>>>>>>>>>>>
>>>>>>>>>>>>> What to do next?
>>>>>>>>>>>> Pass ftrace_dump_on_oops to kernel cmdline.
>>>>>>>>>>>>
>>>>>>>>>>> No sure if I did this right, on oops everything is pushed to
>>>>>>>>>>> console
>>>>>>>>>>> (115k2 serial), I hope nothing essential is lost.
>>>>>>>>>>>
>>>>>>>>>>> I copied the screen buffer to file see attached.
>>>>>>>>>> Thank you, I bet it took quite a some time :-) Anyway, looking at
>>>>>>>>>> the logs around Set Interface requests, we can track every
>>>>>>>>>> endpoint
>>>>>>>>>> that's disabled. I'll take a guess and assume we're failing at
>>>>>>>>>> the last
>>>>>>>>>> Set Interface, that means we should have something odd with
>>>>>>>>>> ep6in, but
>>>>>>>>>> everything looks fine in the trace output:
>>>>>>>>>>
>>>>>>>>>> [   75.823107] irq/14-d-596       0d... 42789194us :
>>>>>>>>>> dwc3_gadget_ep_enable: ep6in: mps 192/346 streams 16 burst 0
>>>>>>>>>> ring 0/0 flags E:swbp:<
>>>>>>>>>> [   75.835472] irq/14-d-596       0d... 42789198us :
>>>>>>>>>> dwc3_alloc_request: ep6in: req 0000000002c71409 length 0/0 zsI
>>>>>>>>>> ==> 0
>>>>>>>>>> [   75.846416] irq/14-d-596       0d... 42789202us :
>>>>>>>>>> dwc3_ep_queue: ep6in: req 0000000002c71409 length 0/192 zsI
>>>>>>>>>> ==> -115
>>>>>>>>>> [   75.857360] irq/14-d-596       0d... 42789204us :
>>>>>>>>>> dwc3_alloc_request: ep6in: req 00000000a324f5d0 length 0/0 zsI
>>>>>>>>>> ==> 0
>>>>>>>>>> [   75.868301] irq/14-d-596       0d... 42789206us :
>>>>>>>>>> dwc3_ep_queue: ep6in: req 00000000a324f5d0 length 0/192 zsI
>>>>>>>>>> ==> -115
>>>>>>>>>> [   75.879244] irq/14-d-596       0d... 42789209us :
>>>>>>>>>> dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0]
>>>>>>>>>> (Not Active) [Status Phase]
>>>>>>>>>> [   75.891880] irq/14-d-596       0d... 42789211us :
>>>>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>>>>>>>> [   75.989131] irq/14-d-596       0d... 42789224us :
>>>>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params
>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>> [   76.096261] irq/14-d-596       0d... 42789272us :
>>>>>>>>>> dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL)
>>>>>>>>>> [Status Phase]
>>>>>>>>>> [   76.107834] irq/14-d-596       0d... 42789275us :
>>>>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>>>>>>>> [   76.122944] irq/14-d-596       0d... 42789277us :
>>>>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0
>>>>>>>>>> zsI ==> 0
>>>>>>>>>> [   76.134160] irq/14-d-596       0d... 42789280us :
>>>>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>> 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>>>>>>>> [   76.231322] irq/14-d-596       0d... 42789292us :
>>>>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params
>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>> [   76.297418] kworker/-23        0d... 42789670us :
>>>>>>>>>> dwc3_ep_queue: ep3in: req 0000000029586135 length 0/96 ZsI ==>
>>>>>>>>>> -115
>>>>>>>>>> [   76.308278] kworker/-23        0d... 42789695us :
>>>>>>>>>> dwc3_prepare_trb: ep3in: trb 00000000b81213d6 (E1:D0) buf
>>>>>>>>>> 0000000003b7a800 size 96 ctrl 00000811 (Hlcs:sC:normal)
>>>>>>>>>> [   76.395294] kworker/-23        0d... 42789707us :
>>>>>>>>>> dwc3_gadget_ep_cmd: ep3in: cmd 'Update Transfer' [60007]
>>>>>>>>>> params 00000000 00000000 00000000 --> status: Successful
>>>>>>>>>> [   76.471900] irq/14-d-596       0d... 42789842us :
>>>>>>>>>> dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL)
>>>>>>>>>> [Setup Phase]
>>>>>>>>>> [   76.489308] irq/14-d-596       0d... 42789845us :
>>>>>>>>>> dwc3_ctrl_req: Set Interface(Intf = 5, Alt.Setting = 0)
>>>>>>>>>> [   76.505650] irq/14-d-596       0d... 42789851us :
>>>>>>>>>> dwc3_ep_dequeue: ep6in: req 0000000002c71409 length 0/192 zsI
>>>>>>>>>> ==> -115
>>>>>>>>>> [   76.523315] irq/14-d-596       0d... 42789854us :
>>>>>>>>>> dwc3_gadget_giveback: ep6in: req 0000000002c71409 length 0/192
>>>>>>>>>> zsI ==> -104
>>>>>>>>>> [   76.541427] irq/14-d-596       0d... 42789857us :
>>>>>>>>>> dwc3_free_request: ep6in: req 0000000002c71409 length 0/192
>>>>>>>>>> zsI ==> -104
>>>>>>>>>> [   76.559267] irq/14-d-596       0d... 42789859us :
>>>>>>>>>> dwc3_ep_dequeue: ep6in: req 00000000a324f5d0 length 0/192 zsI
>>>>>>>>>> ==> -115
>>>>>>>>>> [   76.576937] irq/14-d-596       0d... 42789861us :
>>>>>>>>>> dwc3_gadget_giveback: ep6in: req 00000000a324f5d0 length 0/192
>>>>>>>>>> zsI ==> -104
>>>>>>>>>> [   76.595046] irq/14-d-596       0d... 42789862us :
>>>>>>>>>> dwc3_free_request: ep6in: req 00000000a324f5d0 length 0/192
>>>>>>>>>> zsI ==> -104
>>>>>>>>>> [   76.612892] irq/14-d-596       0d... 42789865us :
>>>>>>>>>> dwc3_gadget_ep_disable: ep6in: mps 192/346 streams 16 burst 0
>>>>>>>>>> ring 0/0 flags E:swbp:<
>>>>>>>>>> [   76.665535] irq/14-d-596       0d... 42789873us :
>>>>>>>>>> dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0]
>>>>>>>>>> (Not Active) [Status Phase]
>>>>>>>>>> [   76.684716] irq/14-d-596       0d... 42789875us :
>>>>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>>>>>>>> [   76.819195] irq/14-d-596       0d... 42789886us :
>>>>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params
>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>> [   76.926324] irq/14-d-596       0d... 42789930us :
>>>>>>>>>> dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL)
>>>>>>>>>> [Status Phase]
>>>>>>>>>> [   76.937892] irq/14-d-596       0d... 42789933us :
>>>>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>>>>>>>> [   76.953003] irq/14-d-596       0d... 42789935us :
>>>>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0
>>>>>>>>>> zsI ==> 0
>>>>>>>>>> [   76.964217] irq/14-d-596       0d... 42789938us :
>>>>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>> 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>>>>>>>> [   77.061379] irq/14-d-596       0d... 42789950us :
>>>>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params
>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>> [   77.168595] irq/14-d-596       0d... 42790509us :
>>>>>>>>>> dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL)
>>>>>>>>>> [Setup Phase]
>>>>>>>>>> [   77.180159] irq/14-d-596       0d... 42790512us :
>>>>>>>>>> dwc3_ctrl_req: Get String Descriptor(Index = 18, Length = 255)
>>>>>>>>>> [   77.190578] irq/14-d-596       0d... 42790537us :
>>>>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>> 0000000003b68000 size 36 ctrl 00000c53 (HLcs:SC:data)
>>>>>>>>>> [   77.287648] irq/14-d-596       0d... 42790550us :
>>>>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params
>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>> [   77.333107] irq/14-d-596       0d... 42790557us :
>>>>>>>>>> dwc3_event: event (000010c2): ep0in: Transfer Not Ready [0]
>>>>>>>>>> (Not Active) [Data Phase]
>>>>>>>>>> [   77.407223] irq/14-d-596       0d... 42790575us :
>>>>>>>>>> dwc3_event: event (000090c2): ep0in: Transfer Not Ready [0]
>>>>>>>>>> (Active) [Data Phase]
>>>>>>>>>> [   77.480985] irq/14-d-596       0d... 42790588us :
>>>>>>>>>> dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL)
>>>>>>>>>> [Data Phase]
>>>>>>>>>> [   77.492376] irq/14-d-596       0d... 42790590us :
>>>>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>> 0000000003b68000 size 0 ctrl 00000c52 (hLcs:SC:data)
>>>>>>>>>> [   77.507221] irq/14-d-596       0d... 42790595us :
>>>>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length
>>>>>>>>>> 36/36 ZsI ==> 0
>>>>>>>>>> [   77.518609] irq/14-d-596       0d... 42790597us :
>>>>>>>>>> dwc3_event: event (000020c0): ep0out: Transfer Not Ready [0]
>>>>>>>>>> (Not Active) [Status Phase]
>>>>>>>>>> [   77.531332] irq/14-d-596       0d... 42790598us :
>>>>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c43 (HLcs:SC:status3)
>>>>>>>>>> [   77.628669] irq/14-d-596       0d... 42790609us :
>>>>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params
>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>>
>>>>>>>>>> Do you mind adding a few prints in dwc3_remove_requests to
>>>>>>>>>> tell us which
>>>>>>>>>> endpoint is being processed? Then we'll know for sure which
>>>>>>>>>> one caused
>>>>>>>>>> the crash.
>>>>>>>>>>
>>>>>>>>> I wouldn't mind but am leaving on a holiday, won't have time
>>>>>>>>> until 6 aug.
>>>>>>>> not a problem, we'll still be here when you're back :-)
>>>>>>> Well, let's go then :-)
>>>>>>>
>>>>>>> To get back in the mood I have retested 5.13.0, 5.14.0-rc1,
>>>>>>> 5.14.0-rc2
>>>>>>> and 5.14.0-rc5.
>>>>>>>
>>>>>>> I find that 5.13.0 works fine, and the issue starts from 5.14.0-rc1.
>>>>>> That's great finding. We have a bisection point. There are a total of
>>>>>> 13764 commits between v5.13 and v5.14-rc1
>>>>>>
>>>>>>     $ git rev-list  --count v5.13..v5.14-rc1
>>>>>>     13764
>>>>>>
>>>>>> git bisect should find the offending commit in at most 14 tries.
>>>>>> That's
>>>>>> not too bad.
>>>>> I correctly guesstimated that the problem got introduced by the usb
>>>>> merge 79160a60
>>>>>
>>>>> "Merge tag 'usb-5.14-rc1' of
>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb"
>>>>>
>>>>> 116 commits(7 bisects).
>>>>>
>>>>> 24f779dac8f3efb9629adc0e486914d93dc45517 is the first bad commit
>>>>>
>>>>> "usb: gadget: f_uac2/u_audio: add feedback endpoint support"
>>>>>
>>>>> Ruslan's 3 patches are related to each other so I reverted all
>>>>> three 24f779da...e89bb428 and applied the reverts to rc1.
>>>>>
>>>>> I can confirm this indeed resolves the problem in rc1.
>>>>>
>>>>> Is late now, tomorrow evening I will apply the reverts to rc6.
>>>>
>>>> With these reverts rc6 works fine as well.
>>>>
>>>> So, where do we go from here?
>>>>
>>>
>>> I know the patches have been tested on dwc2 (by me and others).  I do
>>> not know if Ruslan or Jerome tested them on dwc3 but probably not.
>>> Ruslan has talked about RPi (my case too) and BeagleboneBlack, both
>>> with dwc2. Perhaps the dwc2 behaves a bit differently than dwc3?
>>>
>>> The patches add a new EP-IN for async feedback. I am sorry I have not
>>> followed your long thread (it started as unrelated to uac). Does the
>>> problem appear with f_uac1 or f_uac2? Please how have you reached the
>>> above problem?
>>
>> I'm sorry too. I first believed the issue was related to the patch
>> mentioned in the subject line.
>>
>> The problem appaers with f_uac2. I bost Edison_Arduino board in host
>> mode (there is a switch allowing to select host/device mode). When
>> flipping the switch to device mode udev run a script:
>> But as I am using configfs (excerpt follows) and just disabling the
>> last 2 line resolves the issue, I'm guessing uac2 is the issue. Or
>> exceeding the available resources.
>>
>> # Create directory structure
>> mkdir "${GADGET_BASE_DIR}"
>> cd "${GADGET_BASE_DIR}"
>> mkdir -p configs/c.1/strings/0x409
>> mkdir -p strings/0x409
>>
>> # Serial device
>> mkdir functions/gser.usb0
>> ln -s functions/gser.usb0 configs/c.1/
>> ###
>>
>> # Ethernet device
>> mkdir functions/eem.usb0
>> echo "${DEV_ETH_ADDR}" > functions/eem.usb0/dev_addr
>> echo "${HOST_ETH_ADDR}" > functions/eem.usb0/host_addr
>> ln -s functions/eem.usb0 configs/c.1/
>>
>> # Mass Storage device
>> mkdir functions/mass_storage.usb0
>> echo 1 > functions/mass_storage.usb0/stall
>> echo 0 > functions/mass_storage.usb0/lun.0/cdrom
>> echo 0 > functions/mass_storage.usb0/lun.0/ro
>> echo 0 > functions/mass_storage.usb0/lun.0/nofua
>> echo "${USBDISK}" > functions/mass_storage.usb0/lun.0/file
>> ln -s functions/mass_storage.usb0 configs/c.1/
>>
>> # UAC2 device
>> mkdir functions/uac2.usb0
>> ln -s functions/uac2.usb0 configs/c.1
>> ....
>>
> 
> As you say, could perhaps the reason be that the extra EP-IN added in
> those patches (previously 1, now 2 with the default config you use)
> exceeds your EP-IN max count or available fifos somehow?  You have a
> number of functions initialized. If you change the load order of the
> functions, do you get the error later with a different function? Just
> guessing...
> 
> You should be able to switch the default async EP-OUT (which configures
> the new feedback EP-IN ) to adaptive EP-OUT (which requires no feedback
> EP) with c_sync=8 parameter of f_uac2.
> 
> https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c*L47__;Iw!!A4F2R9G_pg!PZHOWnH3HDs-9a9bz4KLDShZpsubSfG84fu3wEZyK4Q_Wvv2lhHKuk7LJpWPAHQ6Lj4y$
> 
> https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c*L830__;Iw!!A4F2R9G_pg!PZHOWnH3HDs-9a9bz4KLDShZpsubSfG84fu3wEZyK4Q_Wvv2lhHKuk7LJpWPAC7mfgo7$
> 
> https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14-rc6/source/include/uapi/linux/usb/ch9.h*L453__;Iw!!A4F2R9G_pg!PZHOWnH3HDs-9a9bz4KLDShZpsubSfG84fu3wEZyK4Q_Wvv2lhHKuk7LJpWPANdeuGIx$
> 
> Does that fix the problem?
> 

Hi Pavel,

I took a look at 24f779dac8f3 ("usb: gadget: f_uac2/u_audio: add
feedback endpoint support") that Ferry reported the issue from
bisection. I see at least a couple problems in the new UAC2 changes.

1) usb_ep_dequeue() is asynchronous. Don't free requests before the
controller driver give them back.

2) Did you test with SuperSpeed? I don't see companion descriptor.

Please fix them.

Thanks,
Thinh

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-08-19 21:04                                             ` Pavel Hofman
  2021-08-21  2:57                                               ` Thinh Nguyen
@ 2021-08-22 19:43                                               ` Ferry Toth
  2021-08-23 14:59                                                 ` Pavel Hofman
  1 sibling, 1 reply; 73+ messages in thread
From: Ferry Toth @ 2021-08-22 19:43 UTC (permalink / raw)
  To: Pavel Hofman, Felipe Balbi
  Cc: Wesley Cheng, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list, linux-usb, linux-kernel, devicetree, linux-arm-msm,
	jackp, heikki.krogerus, andy.shevchenko, Ruslan Bilovol

Op 19-08-2021 om 23:04 schreef Pavel Hofman:
> 
> Dne 19. 08. 21 v 22:10 Ferry Toth napsal(a):
>> Hi
>>
>> Op 19-08-2021 om 09:51 schreef Pavel Hofman:
>>> Hi,
>>>
>>> Dne 18. 08. 21 v 21:07 Ferry Toth napsal(a):
>>>> Hi,
>>>>
>>>> Op 18-08-2021 om 00:00 schreef Ferry Toth:
>>>>> Hi,
>>>>>
>>>>> Op 16-08-2021 om 07:18 schreef Felipe Balbi:
>>>>>> Hi,
>>>>>>
>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>>>>>>>>>> Hardware name: Intel Corporation Merrifield/BODEGA BAY, 
>>>>>>>>>>>>>>>>> BIOS 542
>>>>>>>>>>>>>>>>> 2015.01.21:18.19.48
>>>>>>>>>>>>>>>>> RIP: 0010:0x500000000
>>>>>>>>>>>>>>>>> Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
>>>>>>>>>>>>>>>>> RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
>>>>>>>>>>>>>>>>> RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX: 
>>>>>>>>>>>>>>>>> 0000000000000000
>>>>>>>>>>>>>>>>> RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI: 
>>>>>>>>>>>>>>>>> ffff8cd546aed200
>>>>>>>>>>>>>>>>> RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09: 
>>>>>>>>>>>>>>>>> 0000000000000001
>>>>>>>>>>>>>>>>> R10: ffff8cd541fd28c0 R11: 0000000000000000 R12: 
>>>>>>>>>>>>>>>>> ffff8cd547342828
>>>>>>>>>>>>>>>>> R13: ffff8cd546aed248 R14: 0000000000000000 R15: 
>>>>>>>>>>>>>>>>> ffff8cd548b1d000
>>>>>>>>>>>>>>>>> FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000) 
>>>>>>>>>>>>>>>>> knlGS:0000000000000000
>>>>>>>>>>>>>>>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>>>>>>>>>>>>> CR2: 0000000500000000 CR3: 000000000311e000 CR4: 
>>>>>>>>>>>>>>>>> 00000000001006f0
>>>>>>>>>>>>>>>>> Call Trace:
>>>>>>>>>>>>>>>>>         ? dwc3_remove_requests.constprop.0+0x14d/0x170
>>>>>>>>>>>>>>>>>         ? __dwc3_gadget_ep_disable+0x7a/0x160
>>>>>>>>>>>>>>>>>         ? dwc3_gadget_ep_disable+0x3d/0xd0
>>>>>>>>>>>>>>>>>         ? usb_ep_disable+0x1c/0x
>>>>>>>>>>>>>>>>>         ? u_audio_stop_capture+0x79/0x120 [u_audio]
>>>>>>>>>>>>>>>>>         ? afunc_set_alt+0x73/0x80 [usb_f_uac2]
>>>>>>>>>> So this is triggered by a SetInterface request...
>>>>>>>>>>
>>>>>>>>>>>>>>>>>         ? composite_setup+0x224/0x1b90 [libcomposite]
>>>>>>>>>>>>>>>>>         ? __dwc3_gadget_kick_transfer+0x160/0x400
>>>>>>>>>>>>>>>>>         ? dwc3_gadget_ep_queue+0xf3/0x1a0
>>>>>>>>>>>>>>>>>         ? configfs_composite_setup+0x6b/0x90 
>>>>>>>>>>>>>>>>> [libcomposite]
>>>>>>>>>>>>>>>>>         ? configfs_composite_setup+0x6b/0x90 
>>>>>>>>>>>>>>>>> [libcomposite]
>>>>>>>>>>>>>>>>>         ? dwc3_ep0_interrupt+0x459/0xa40
>>>>>>>>>>>>>>>>>         ? dwc3_thread_interrupt+0x8ee/0xf40
>>>>>>>>>>>>>>>>>         ? __schedule+0x235/0x6c0
>>>>>>>>>>>>>>>>>         ? disable_irq_nosync+0x10/0x10
>>>>>>>>>>>>>>>>>         ? irq_thread_fn+0x1b/0x60
>>>>>>>>>>>>>>>>>         ? irq_thread+0xc0/0x160
>>>>>>>>>>>>>>>>>         ? irq_thread_check_affinity+0x70/0x70
>>>>>>>>>>>>>>>>>         ? irq_forced_thread_fn+0x70/0x70
>>>>>>>>>>>>>>>>>         ? kthread+0x122/0x140
>>>>>>>>>>>>>>>>>         ? set_kthread_struct+0x40/0x40
>>>>>>>>>>>>>>>>>         ? ret_from_fork+0x22/0x30
>>>>>>>>>>>>>>>> Do you mind enabling dwc3 traces and collecting them? 
>>>>>>>>>>>>>>>> Trying to figure
>>>>>>>>>>>>>>>> out how we got here.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'll try if I can get the same error by booting with USB 
>>>>>>>>>>>>>>> in host mode
>>>>>>>>>>>>>>> and then switch to device mode. If so I can enable traces 
>>>>>>>>>>>>>>> and collect as
>>>>>>>>>>>>>>> you explained me before.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'll try before monday, as then I fly for a holiday and 
>>>>>>>>>>>>>>> will not be
>>>>>>>>>>>>>>> available before rc5.
>>>>>>>>>>>>>> you can enable all of those with kernel cmdline :-)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> you need ftrace_dump_on_oops=1 and also need the correct 
>>>>>>>>>>>>>> options on
>>>>>>>>>>>>>> trace_buf_size and trace_event.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> On Edison-Arduino I have a switch to go to device mode, 
>>>>>>>>>>>>> after which
>>>>>>>>>>>>> udev triggers a script configure gadgets through configfs.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I tried to log following these instructions:
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs 
>>>>>>>>>>>>> <https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs> 
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Unfortunately the kernel crashes so badly I can not get to 
>>>>>>>>>>>>> the ` cp
>>>>>>>>>>>>> /t/trace /root/trace.txt` line (after a while the watchdog 
>>>>>>>>>>>>> kicks).
>>>>>>>>>>>>>
>>>>>>>>>>>>> What to do next?
>>>>>>>>>>>> Pass ftrace_dump_on_oops to kernel cmdline.
>>>>>>>>>>>>
>>>>>>>>>>> No sure if I did this right, on oops everything is pushed to 
>>>>>>>>>>> console
>>>>>>>>>>> (115k2 serial), I hope nothing essential is lost.
>>>>>>>>>>>
>>>>>>>>>>> I copied the screen buffer to file see attached.
>>>>>>>>>> Thank you, I bet it took quite a some time :-) Anyway, looking at
>>>>>>>>>> the logs around Set Interface requests, we can track every 
>>>>>>>>>> endpoint
>>>>>>>>>> that's disabled. I'll take a guess and assume we're failing at 
>>>>>>>>>> the last
>>>>>>>>>> Set Interface, that means we should have something odd with 
>>>>>>>>>> ep6in, but
>>>>>>>>>> everything looks fine in the trace output:
>>>>>>>>>>
>>>>>>>>>> [   75.823107] irq/14-d-596       0d... 42789194us : 
>>>>>>>>>> dwc3_gadget_ep_enable: ep6in: mps 192/346 streams 16 burst 0 
>>>>>>>>>> ring 0/0 flags E:swbp:<
>>>>>>>>>> [   75.835472] irq/14-d-596       0d... 42789198us : 
>>>>>>>>>> dwc3_alloc_request: ep6in: req 0000000002c71409 length 0/0 zsI 
>>>>>>>>>> ==> 0
>>>>>>>>>> [   75.846416] irq/14-d-596       0d... 42789202us : 
>>>>>>>>>> dwc3_ep_queue: ep6in: req 0000000002c71409 length 0/192 zsI 
>>>>>>>>>> ==> -115
>>>>>>>>>> [   75.857360] irq/14-d-596       0d... 42789204us : 
>>>>>>>>>> dwc3_alloc_request: ep6in: req 00000000a324f5d0 length 0/0 zsI 
>>>>>>>>>> ==> 0
>>>>>>>>>> [   75.868301] irq/14-d-596       0d... 42789206us : 
>>>>>>>>>> dwc3_ep_queue: ep6in: req 00000000a324f5d0 length 0/192 zsI 
>>>>>>>>>> ==> -115
>>>>>>>>>> [   75.879244] irq/14-d-596       0d... 42789209us : 
>>>>>>>>>> dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0] 
>>>>>>>>>> (Not Active) [Status Phase]
>>>>>>>>>> [   75.891880] irq/14-d-596       0d... 42789211us : 
>>>>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>>>>>>>> [   75.989131] irq/14-d-596       0d... 42789224us : 
>>>>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 
>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>> [   76.096261] irq/14-d-596       0d... 42789272us : 
>>>>>>>>>> dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) 
>>>>>>>>>> [Status Phase]
>>>>>>>>>> [   76.107834] irq/14-d-596       0d... 42789275us : 
>>>>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>>>>>>>> [   76.122944] irq/14-d-596       0d... 42789277us : 
>>>>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 
>>>>>>>>>> zsI ==> 0
>>>>>>>>>> [   76.134160] irq/14-d-596       0d... 42789280us : 
>>>>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>>> 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>>>>>>>> [   76.231322] irq/14-d-596       0d... 42789292us : 
>>>>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 
>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>> [   76.297418] kworker/-23        0d... 42789670us : 
>>>>>>>>>> dwc3_ep_queue: ep3in: req 0000000029586135 length 0/96 ZsI ==> 
>>>>>>>>>> -115
>>>>>>>>>> [   76.308278] kworker/-23        0d... 42789695us : 
>>>>>>>>>> dwc3_prepare_trb: ep3in: trb 00000000b81213d6 (E1:D0) buf 
>>>>>>>>>> 0000000003b7a800 size 96 ctrl 00000811 (Hlcs:sC:normal)
>>>>>>>>>> [   76.395294] kworker/-23        0d... 42789707us : 
>>>>>>>>>> dwc3_gadget_ep_cmd: ep3in: cmd 'Update Transfer' [60007] 
>>>>>>>>>> params 00000000 00000000 00000000 --> status: Successful
>>>>>>>>>> [   76.471900] irq/14-d-596       0d... 42789842us : 
>>>>>>>>>> dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL) 
>>>>>>>>>> [Setup Phase]
>>>>>>>>>> [   76.489308] irq/14-d-596       0d... 42789845us : 
>>>>>>>>>> dwc3_ctrl_req: Set Interface(Intf = 5, Alt.Setting = 0)
>>>>>>>>>> [   76.505650] irq/14-d-596       0d... 42789851us : 
>>>>>>>>>> dwc3_ep_dequeue: ep6in: req 0000000002c71409 length 0/192 zsI 
>>>>>>>>>> ==> -115
>>>>>>>>>> [   76.523315] irq/14-d-596       0d... 42789854us : 
>>>>>>>>>> dwc3_gadget_giveback: ep6in: req 0000000002c71409 length 0/192 
>>>>>>>>>> zsI ==> -104
>>>>>>>>>> [   76.541427] irq/14-d-596       0d... 42789857us : 
>>>>>>>>>> dwc3_free_request: ep6in: req 0000000002c71409 length 0/192 
>>>>>>>>>> zsI ==> -104
>>>>>>>>>> [   76.559267] irq/14-d-596       0d... 42789859us : 
>>>>>>>>>> dwc3_ep_dequeue: ep6in: req 00000000a324f5d0 length 0/192 zsI 
>>>>>>>>>> ==> -115
>>>>>>>>>> [   76.576937] irq/14-d-596       0d... 42789861us : 
>>>>>>>>>> dwc3_gadget_giveback: ep6in: req 00000000a324f5d0 length 0/192 
>>>>>>>>>> zsI ==> -104
>>>>>>>>>> [   76.595046] irq/14-d-596       0d... 42789862us : 
>>>>>>>>>> dwc3_free_request: ep6in: req 00000000a324f5d0 length 0/192 
>>>>>>>>>> zsI ==> -104
>>>>>>>>>> [   76.612892] irq/14-d-596       0d... 42789865us : 
>>>>>>>>>> dwc3_gadget_ep_disable: ep6in: mps 192/346 streams 16 burst 0 
>>>>>>>>>> ring 0/0 flags E:swbp:<
>>>>>>>>>> [   76.665535] irq/14-d-596       0d... 42789873us : 
>>>>>>>>>> dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0] 
>>>>>>>>>> (Not Active) [Status Phase]
>>>>>>>>>> [   76.684716] irq/14-d-596       0d... 42789875us : 
>>>>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>>>>>>>> [   76.819195] irq/14-d-596       0d... 42789886us : 
>>>>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 
>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>> [   76.926324] irq/14-d-596       0d... 42789930us : 
>>>>>>>>>> dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) 
>>>>>>>>>> [Status Phase]
>>>>>>>>>> [   76.937892] irq/14-d-596       0d... 42789933us : 
>>>>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>>>>>>>> [   76.953003] irq/14-d-596       0d... 42789935us : 
>>>>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 
>>>>>>>>>> zsI ==> 0
>>>>>>>>>> [   76.964217] irq/14-d-596       0d... 42789938us : 
>>>>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>>> 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>>>>>>>> [   77.061379] irq/14-d-596       0d... 42789950us : 
>>>>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 
>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>> [   77.168595] irq/14-d-596       0d... 42790509us : 
>>>>>>>>>> dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL) 
>>>>>>>>>> [Setup Phase]
>>>>>>>>>> [   77.180159] irq/14-d-596       0d... 42790512us : 
>>>>>>>>>> dwc3_ctrl_req: Get String Descriptor(Index = 18, Length = 255)
>>>>>>>>>> [   77.190578] irq/14-d-596       0d... 42790537us : 
>>>>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>>> 0000000003b68000 size 36 ctrl 00000c53 (HLcs:SC:data)
>>>>>>>>>> [   77.287648] irq/14-d-596       0d... 42790550us : 
>>>>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 
>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>> [   77.333107] irq/14-d-596       0d... 42790557us : 
>>>>>>>>>> dwc3_event: event (000010c2): ep0in: Transfer Not Ready [0] 
>>>>>>>>>> (Not Active) [Data Phase]
>>>>>>>>>> [   77.407223] irq/14-d-596       0d... 42790575us : 
>>>>>>>>>> dwc3_event: event (000090c2): ep0in: Transfer Not Ready [0] 
>>>>>>>>>> (Active) [Data Phase]
>>>>>>>>>> [   77.480985] irq/14-d-596       0d... 42790588us : 
>>>>>>>>>> dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) 
>>>>>>>>>> [Data Phase]
>>>>>>>>>> [   77.492376] irq/14-d-596       0d... 42790590us : 
>>>>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>>> 0000000003b68000 size 0 ctrl 00000c52 (hLcs:SC:data)
>>>>>>>>>> [   77.507221] irq/14-d-596       0d... 42790595us : 
>>>>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 
>>>>>>>>>> 36/36 ZsI ==> 0
>>>>>>>>>> [   77.518609] irq/14-d-596       0d... 42790597us : 
>>>>>>>>>> dwc3_event: event (000020c0): ep0out: Transfer Not Ready [0] 
>>>>>>>>>> (Not Active) [Status Phase]
>>>>>>>>>> [   77.531332] irq/14-d-596       0d... 42790598us : 
>>>>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c43 (HLcs:SC:status3)
>>>>>>>>>> [   77.628669] irq/14-d-596       0d... 42790609us : 
>>>>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 
>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>>
>>>>>>>>>> Do you mind adding a few prints in dwc3_remove_requests to 
>>>>>>>>>> tell us which
>>>>>>>>>> endpoint is being processed? Then we'll know for sure which 
>>>>>>>>>> one caused
>>>>>>>>>> the crash.
>>>>>>>>>>
>>>>>>>>> I wouldn't mind but am leaving on a holiday, won't have time 
>>>>>>>>> until 6 aug.
>>>>>>>> not a problem, we'll still be here when you're back :-)
>>>>>>> Well, let's go then :-)
>>>>>>>
>>>>>>> To get back in the mood I have retested 5.13.0, 5.14.0-rc1, 
>>>>>>> 5.14.0-rc2
>>>>>>> and 5.14.0-rc5.
>>>>>>>
>>>>>>> I find that 5.13.0 works fine, and the issue starts from 5.14.0-rc1.
>>>>>> That's great finding. We have a bisection point. There are a total of
>>>>>> 13764 commits between v5.13 and v5.14-rc1
>>>>>>
>>>>>>     $ git rev-list  --count v5.13..v5.14-rc1
>>>>>>     13764
>>>>>>
>>>>>> git bisect should find the offending commit in at most 14 tries. 
>>>>>> That's
>>>>>> not too bad.
>>>>> I correctly guesstimated that the problem got introduced by the usb 
>>>>> merge 79160a60
>>>>>
>>>>> "Merge tag 'usb-5.14-rc1' of 
>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb"
>>>>>
>>>>> 116 commits(7 bisects).
>>>>>
>>>>> 24f779dac8f3efb9629adc0e486914d93dc45517 is the first bad commit
>>>>>
>>>>> "usb: gadget: f_uac2/u_audio: add feedback endpoint support"
>>>>>
>>>>> Ruslan's 3 patches are related to each other so I reverted all 
>>>>> three 24f779da...e89bb428 and applied the reverts to rc1.
>>>>>
>>>>> I can confirm this indeed resolves the problem in rc1.
>>>>>
>>>>> Is late now, tomorrow evening I will apply the reverts to rc6.
>>>>
>>>> With these reverts rc6 works fine as well.
>>>>
>>>> So, where do we go from here?
>>>>
>>>
>>> I know the patches have been tested on dwc2 (by me and others).  I do 
>>> not know if Ruslan or Jerome tested them on dwc3 but probably not. 
>>> Ruslan has talked about RPi (my case too) and BeagleboneBlack, both 
>>> with dwc2. Perhaps the dwc2 behaves a bit differently than dwc3?
>>>
>>> The patches add a new EP-IN for async feedback. I am sorry I have not 
>>> followed your long thread (it started as unrelated to uac). Does the 
>>> problem appear with f_uac1 or f_uac2? Please how have you reached the 
>>> above problem?
>>
>> I'm sorry too. I first believed the issue was related to the patch 
>> mentioned in the subject line.
>>
>> The problem appaers with f_uac2. I bost Edison_Arduino board in host 
>> mode (there is a switch allowing to select host/device mode). When 
>> flipping the switch to device mode udev run a script:
>> But as I am using configfs (excerpt follows) and just disabling the 
>> last 2 line resolves the issue, I'm guessing uac2 is the issue. Or 
>> exceeding the available resources.
>>
>> # Create directory structure
>> mkdir "${GADGET_BASE_DIR}"
>> cd "${GADGET_BASE_DIR}"
>> mkdir -p configs/c.1/strings/0x409
>> mkdir -p strings/0x409
>>
>> # Serial device
>> mkdir functions/gser.usb0
>> ln -s functions/gser.usb0 configs/c.1/
>> ###
>>
>> # Ethernet device
>> mkdir functions/eem.usb0
>> echo "${DEV_ETH_ADDR}" > functions/eem.usb0/dev_addr
>> echo "${HOST_ETH_ADDR}" > functions/eem.usb0/host_addr
>> ln -s functions/eem.usb0 configs/c.1/
>>
>> # Mass Storage device
>> mkdir functions/mass_storage.usb0
>> echo 1 > functions/mass_storage.usb0/stall
>> echo 0 > functions/mass_storage.usb0/lun.0/cdrom
>> echo 0 > functions/mass_storage.usb0/lun.0/ro
>> echo 0 > functions/mass_storage.usb0/lun.0/nofua
>> echo "${USBDISK}" > functions/mass_storage.usb0/lun.0/file
>> ln -s functions/mass_storage.usb0 configs/c.1/
>>
>> # UAC2 device
>> mkdir functions/uac2.usb0
>> ln -s functions/uac2.usb0 configs/c.1
>> ....
>>
> 
> As you say, could perhaps the reason be that the extra EP-IN added in 
> those patches (previously 1, now 2 with the default config you use) 
> exceeds your EP-IN max count or available fifos somehow?  You have a 
> number of functions initialized. If you change the load order of the 
> functions, do you get the error later with a different function? Just 
> guessing...
> 
> You should be able to switch the default async EP-OUT (which configures 
> the new feedback EP-IN ) to adaptive EP-OUT (which requires no feedback 
> EP) with c_sync=8 parameter of f_uac2.
> 
> https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c#L47 
> 
> 
> https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c#L830 
> 
> 
> https://elixir.bootlin.com/linux/v5.14-rc6/source/include/uapi/linux/usb/ch9.h#L453 
> 
> 
> Does that fix the problem?

Not sure how to do that. Do you mean the module should have a parameter 
called c_sync? `modinfo` list no parameters for usb_f_uac2.

> Thanks,
> 
> Pavel.


^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-08-21  2:57                                               ` Thinh Nguyen
@ 2021-08-22 20:10                                                 ` Ferry Toth
  2021-08-23  7:57                                                   ` Pavel Hofman
  2021-09-02  0:28                                                 ` Jack Pham
  1 sibling, 1 reply; 73+ messages in thread
From: Ferry Toth @ 2021-08-22 20:10 UTC (permalink / raw)
  To: Thinh Nguyen, Pavel Hofman, Felipe Balbi
  Cc: Wesley Cheng, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list, linux-usb, linux-kernel, devicetree, linux-arm-msm,
	jackp, heikki.krogerus, andy.shevchenko, Ruslan Bilovol

Op 21-08-2021 om 04:57 schreef Thinh Nguyen:
> Pavel Hofman wrote:
>>
>> Dne 19. 08. 21 v 22:10 Ferry Toth napsal(a):
>>> Hi
>>>
>>> Op 19-08-2021 om 09:51 schreef Pavel Hofman:
>>>> Hi,
>>>>
>>>> Dne 18. 08. 21 v 21:07 Ferry Toth napsal(a):
>>>>> Hi,
>>>>>
>>>>> Op 18-08-2021 om 00:00 schreef Ferry Toth:
>>>>>> Hi,
>>>>>>
>>>>>> Op 16-08-2021 om 07:18 schreef Felipe Balbi:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>>>>>>>>>>> Hardware name: Intel Corporation Merrifield/BODEGA BAY,
>>>>>>>>>>>>>>>>>> BIOS 542
>>>>>>>>>>>>>>>>>> 2015.01.21:18.19.48
>>>>>>>>>>>>>>>>>> RIP: 0010:0x500000000
>>>>>>>>>>>>>>>>>> Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
>>>>>>>>>>>>>>>>>> RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
>>>>>>>>>>>>>>>>>> RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX:
>>>>>>>>>>>>>>>>>> 0000000000000000
>>>>>>>>>>>>>>>>>> RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI:
>>>>>>>>>>>>>>>>>> ffff8cd546aed200
>>>>>>>>>>>>>>>>>> RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09:
>>>>>>>>>>>>>>>>>> 0000000000000001
>>>>>>>>>>>>>>>>>> R10: ffff8cd541fd28c0 R11: 0000000000000000 R12:
>>>>>>>>>>>>>>>>>> ffff8cd547342828
>>>>>>>>>>>>>>>>>> R13: ffff8cd546aed248 R14: 0000000000000000 R15:
>>>>>>>>>>>>>>>>>> ffff8cd548b1d000
>>>>>>>>>>>>>>>>>> FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000)
>>>>>>>>>>>>>>>>>> knlGS:0000000000000000
>>>>>>>>>>>>>>>>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>>>>>>>>>>>>>> CR2: 0000000500000000 CR3: 000000000311e000 CR4:
>>>>>>>>>>>>>>>>>> 00000000001006f0
>>>>>>>>>>>>>>>>>> Call Trace:
>>>>>>>>>>>>>>>>>>          ? dwc3_remove_requests.constprop.0+0x14d/0x170
>>>>>>>>>>>>>>>>>>          ? __dwc3_gadget_ep_disable+0x7a/0x160
>>>>>>>>>>>>>>>>>>          ? dwc3_gadget_ep_disable+0x3d/0xd0
>>>>>>>>>>>>>>>>>>          ? usb_ep_disable+0x1c/0x
>>>>>>>>>>>>>>>>>>          ? u_audio_stop_capture+0x79/0x120 [u_audio]
>>>>>>>>>>>>>>>>>>          ? afunc_set_alt+0x73/0x80 [usb_f_uac2]
>>>>>>>>>>> So this is triggered by a SetInterface request...
>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>          ? composite_setup+0x224/0x1b90 [libcomposite]
>>>>>>>>>>>>>>>>>>          ? __dwc3_gadget_kick_transfer+0x160/0x400
>>>>>>>>>>>>>>>>>>          ? dwc3_gadget_ep_queue+0xf3/0x1a0
>>>>>>>>>>>>>>>>>>          ? configfs_composite_setup+0x6b/0x90
>>>>>>>>>>>>>>>>>> [libcomposite]
>>>>>>>>>>>>>>>>>>          ? configfs_composite_setup+0x6b/0x90
>>>>>>>>>>>>>>>>>> [libcomposite]
>>>>>>>>>>>>>>>>>>          ? dwc3_ep0_interrupt+0x459/0xa40
>>>>>>>>>>>>>>>>>>          ? dwc3_thread_interrupt+0x8ee/0xf40
>>>>>>>>>>>>>>>>>>          ? __schedule+0x235/0x6c0
>>>>>>>>>>>>>>>>>>          ? disable_irq_nosync+0x10/0x10
>>>>>>>>>>>>>>>>>>          ? irq_thread_fn+0x1b/0x60
>>>>>>>>>>>>>>>>>>          ? irq_thread+0xc0/0x160
>>>>>>>>>>>>>>>>>>          ? irq_thread_check_affinity+0x70/0x70
>>>>>>>>>>>>>>>>>>          ? irq_forced_thread_fn+0x70/0x70
>>>>>>>>>>>>>>>>>>          ? kthread+0x122/0x140
>>>>>>>>>>>>>>>>>>          ? set_kthread_struct+0x40/0x40
>>>>>>>>>>>>>>>>>>          ? ret_from_fork+0x22/0x30
>>>>>>>>>>>>>>>>> Do you mind enabling dwc3 traces and collecting them?
>>>>>>>>>>>>>>>>> Trying to figure
>>>>>>>>>>>>>>>>> out how we got here.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'll try if I can get the same error by booting with USB
>>>>>>>>>>>>>>>> in host mode
>>>>>>>>>>>>>>>> and then switch to device mode. If so I can enable traces
>>>>>>>>>>>>>>>> and collect as
>>>>>>>>>>>>>>>> you explained me before.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'll try before monday, as then I fly for a holiday and
>>>>>>>>>>>>>>>> will not be
>>>>>>>>>>>>>>>> available before rc5.
>>>>>>>>>>>>>>> you can enable all of those with kernel cmdline :-)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://urldefense.com/v3/__https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html__;!!A4F2R9G_pg!PZHOWnH3HDs-9a9bz4KLDShZpsubSfG84fu3wEZyK4Q_Wvv2lhHKuk7LJpWPAPch0qja$
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> you need ftrace_dump_on_oops=1 and also need the correct
>>>>>>>>>>>>>>> options on
>>>>>>>>>>>>>>> trace_buf_size and trace_event.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Edison-Arduino I have a switch to go to device mode,
>>>>>>>>>>>>>> after which
>>>>>>>>>>>>>> udev triggers a script configure gadgets through configfs.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I tried to log following these instructions:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://urldefense.com/v3/__https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html*reporting-bugs__;Iw!!A4F2R9G_pg!PZHOWnH3HDs-9a9bz4KLDShZpsubSfG84fu3wEZyK4Q_Wvv2lhHKuk7LJpWPAG64agnA$
>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html*reporting-bugs__;Iw!!A4F2R9G_pg!PZHOWnH3HDs-9a9bz4KLDShZpsubSfG84fu3wEZyK4Q_Wvv2lhHKuk7LJpWPAG64agnA$
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Unfortunately the kernel crashes so badly I can not get to
>>>>>>>>>>>>>> the ` cp
>>>>>>>>>>>>>> /t/trace /root/trace.txt` line (after a while the watchdog
>>>>>>>>>>>>>> kicks).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What to do next?
>>>>>>>>>>>>> Pass ftrace_dump_on_oops to kernel cmdline.
>>>>>>>>>>>>>
>>>>>>>>>>>> No sure if I did this right, on oops everything is pushed to
>>>>>>>>>>>> console
>>>>>>>>>>>> (115k2 serial), I hope nothing essential is lost.
>>>>>>>>>>>>
>>>>>>>>>>>> I copied the screen buffer to file see attached.
>>>>>>>>>>> Thank you, I bet it took quite a some time :-) Anyway, looking at
>>>>>>>>>>> the logs around Set Interface requests, we can track every
>>>>>>>>>>> endpoint
>>>>>>>>>>> that's disabled. I'll take a guess and assume we're failing at
>>>>>>>>>>> the last
>>>>>>>>>>> Set Interface, that means we should have something odd with
>>>>>>>>>>> ep6in, but
>>>>>>>>>>> everything looks fine in the trace output:
>>>>>>>>>>>
>>>>>>>>>>> [   75.823107] irq/14-d-596       0d... 42789194us :
>>>>>>>>>>> dwc3_gadget_ep_enable: ep6in: mps 192/346 streams 16 burst 0
>>>>>>>>>>> ring 0/0 flags E:swbp:<
>>>>>>>>>>> [   75.835472] irq/14-d-596       0d... 42789198us :
>>>>>>>>>>> dwc3_alloc_request: ep6in: req 0000000002c71409 length 0/0 zsI
>>>>>>>>>>> ==> 0
>>>>>>>>>>> [   75.846416] irq/14-d-596       0d... 42789202us :
>>>>>>>>>>> dwc3_ep_queue: ep6in: req 0000000002c71409 length 0/192 zsI
>>>>>>>>>>> ==> -115
>>>>>>>>>>> [   75.857360] irq/14-d-596       0d... 42789204us :
>>>>>>>>>>> dwc3_alloc_request: ep6in: req 00000000a324f5d0 length 0/0 zsI
>>>>>>>>>>> ==> 0
>>>>>>>>>>> [   75.868301] irq/14-d-596       0d... 42789206us :
>>>>>>>>>>> dwc3_ep_queue: ep6in: req 00000000a324f5d0 length 0/192 zsI
>>>>>>>>>>> ==> -115
>>>>>>>>>>> [   75.879244] irq/14-d-596       0d... 42789209us :
>>>>>>>>>>> dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0]
>>>>>>>>>>> (Not Active) [Status Phase]
>>>>>>>>>>> [   75.891880] irq/14-d-596       0d... 42789211us :
>>>>>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>>>>>>>>> [   75.989131] irq/14-d-596       0d... 42789224us :
>>>>>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params
>>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>>> [   76.096261] irq/14-d-596       0d... 42789272us :
>>>>>>>>>>> dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL)
>>>>>>>>>>> [Status Phase]
>>>>>>>>>>> [   76.107834] irq/14-d-596       0d... 42789275us :
>>>>>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>>>>>>>>> [   76.122944] irq/14-d-596       0d... 42789277us :
>>>>>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0
>>>>>>>>>>> zsI ==> 0
>>>>>>>>>>> [   76.134160] irq/14-d-596       0d... 42789280us :
>>>>>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>>> 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>>>>>>>>> [   76.231322] irq/14-d-596       0d... 42789292us :
>>>>>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params
>>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>>> [   76.297418] kworker/-23        0d... 42789670us :
>>>>>>>>>>> dwc3_ep_queue: ep3in: req 0000000029586135 length 0/96 ZsI ==>
>>>>>>>>>>> -115
>>>>>>>>>>> [   76.308278] kworker/-23        0d... 42789695us :
>>>>>>>>>>> dwc3_prepare_trb: ep3in: trb 00000000b81213d6 (E1:D0) buf
>>>>>>>>>>> 0000000003b7a800 size 96 ctrl 00000811 (Hlcs:sC:normal)
>>>>>>>>>>> [   76.395294] kworker/-23        0d... 42789707us :
>>>>>>>>>>> dwc3_gadget_ep_cmd: ep3in: cmd 'Update Transfer' [60007]
>>>>>>>>>>> params 00000000 00000000 00000000 --> status: Successful
>>>>>>>>>>> [   76.471900] irq/14-d-596       0d... 42789842us :
>>>>>>>>>>> dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL)
>>>>>>>>>>> [Setup Phase]
>>>>>>>>>>> [   76.489308] irq/14-d-596       0d... 42789845us :
>>>>>>>>>>> dwc3_ctrl_req: Set Interface(Intf = 5, Alt.Setting = 0)
>>>>>>>>>>> [   76.505650] irq/14-d-596       0d... 42789851us :
>>>>>>>>>>> dwc3_ep_dequeue: ep6in: req 0000000002c71409 length 0/192 zsI
>>>>>>>>>>> ==> -115
>>>>>>>>>>> [   76.523315] irq/14-d-596       0d... 42789854us :
>>>>>>>>>>> dwc3_gadget_giveback: ep6in: req 0000000002c71409 length 0/192
>>>>>>>>>>> zsI ==> -104
>>>>>>>>>>> [   76.541427] irq/14-d-596       0d... 42789857us :
>>>>>>>>>>> dwc3_free_request: ep6in: req 0000000002c71409 length 0/192
>>>>>>>>>>> zsI ==> -104
>>>>>>>>>>> [   76.559267] irq/14-d-596       0d... 42789859us :
>>>>>>>>>>> dwc3_ep_dequeue: ep6in: req 00000000a324f5d0 length 0/192 zsI
>>>>>>>>>>> ==> -115
>>>>>>>>>>> [   76.576937] irq/14-d-596       0d... 42789861us :
>>>>>>>>>>> dwc3_gadget_giveback: ep6in: req 00000000a324f5d0 length 0/192
>>>>>>>>>>> zsI ==> -104
>>>>>>>>>>> [   76.595046] irq/14-d-596       0d... 42789862us :
>>>>>>>>>>> dwc3_free_request: ep6in: req 00000000a324f5d0 length 0/192
>>>>>>>>>>> zsI ==> -104
>>>>>>>>>>> [   76.612892] irq/14-d-596       0d... 42789865us :
>>>>>>>>>>> dwc3_gadget_ep_disable: ep6in: mps 192/346 streams 16 burst 0
>>>>>>>>>>> ring 0/0 flags E:swbp:<
>>>>>>>>>>> [   76.665535] irq/14-d-596       0d... 42789873us :
>>>>>>>>>>> dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0]
>>>>>>>>>>> (Not Active) [Status Phase]
>>>>>>>>>>> [   76.684716] irq/14-d-596       0d... 42789875us :
>>>>>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>>>>>>>>> [   76.819195] irq/14-d-596       0d... 42789886us :
>>>>>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params
>>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>>> [   76.926324] irq/14-d-596       0d... 42789930us :
>>>>>>>>>>> dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL)
>>>>>>>>>>> [Status Phase]
>>>>>>>>>>> [   76.937892] irq/14-d-596       0d... 42789933us :
>>>>>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>>>>>>>>> [   76.953003] irq/14-d-596       0d... 42789935us :
>>>>>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0
>>>>>>>>>>> zsI ==> 0
>>>>>>>>>>> [   76.964217] irq/14-d-596       0d... 42789938us :
>>>>>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>>> 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>>>>>>>>> [   77.061379] irq/14-d-596       0d... 42789950us :
>>>>>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params
>>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>>> [   77.168595] irq/14-d-596       0d... 42790509us :
>>>>>>>>>>> dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL)
>>>>>>>>>>> [Setup Phase]
>>>>>>>>>>> [   77.180159] irq/14-d-596       0d... 42790512us :
>>>>>>>>>>> dwc3_ctrl_req: Get String Descriptor(Index = 18, Length = 255)
>>>>>>>>>>> [   77.190578] irq/14-d-596       0d... 42790537us :
>>>>>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>>> 0000000003b68000 size 36 ctrl 00000c53 (HLcs:SC:data)
>>>>>>>>>>> [   77.287648] irq/14-d-596       0d... 42790550us :
>>>>>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params
>>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>>> [   77.333107] irq/14-d-596       0d... 42790557us :
>>>>>>>>>>> dwc3_event: event (000010c2): ep0in: Transfer Not Ready [0]
>>>>>>>>>>> (Not Active) [Data Phase]
>>>>>>>>>>> [   77.407223] irq/14-d-596       0d... 42790575us :
>>>>>>>>>>> dwc3_event: event (000090c2): ep0in: Transfer Not Ready [0]
>>>>>>>>>>> (Active) [Data Phase]
>>>>>>>>>>> [   77.480985] irq/14-d-596       0d... 42790588us :
>>>>>>>>>>> dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL)
>>>>>>>>>>> [Data Phase]
>>>>>>>>>>> [   77.492376] irq/14-d-596       0d... 42790590us :
>>>>>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>>> 0000000003b68000 size 0 ctrl 00000c52 (hLcs:SC:data)
>>>>>>>>>>> [   77.507221] irq/14-d-596       0d... 42790595us :
>>>>>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length
>>>>>>>>>>> 36/36 ZsI ==> 0
>>>>>>>>>>> [   77.518609] irq/14-d-596       0d... 42790597us :
>>>>>>>>>>> dwc3_event: event (000020c0): ep0out: Transfer Not Ready [0]
>>>>>>>>>>> (Not Active) [Status Phase]
>>>>>>>>>>> [   77.531332] irq/14-d-596       0d... 42790598us :
>>>>>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c43 (HLcs:SC:status3)
>>>>>>>>>>> [   77.628669] irq/14-d-596       0d... 42790609us :
>>>>>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params
>>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>>>
>>>>>>>>>>> Do you mind adding a few prints in dwc3_remove_requests to
>>>>>>>>>>> tell us which
>>>>>>>>>>> endpoint is being processed? Then we'll know for sure which
>>>>>>>>>>> one caused
>>>>>>>>>>> the crash.
>>>>>>>>>>>
>>>>>>>>>> I wouldn't mind but am leaving on a holiday, won't have time
>>>>>>>>>> until 6 aug.
>>>>>>>>> not a problem, we'll still be here when you're back :-)
>>>>>>>> Well, let's go then :-)
>>>>>>>>
>>>>>>>> To get back in the mood I have retested 5.13.0, 5.14.0-rc1,
>>>>>>>> 5.14.0-rc2
>>>>>>>> and 5.14.0-rc5.
>>>>>>>>
>>>>>>>> I find that 5.13.0 works fine, and the issue starts from 5.14.0-rc1.
>>>>>>> That's great finding. We have a bisection point. There are a total of
>>>>>>> 13764 commits between v5.13 and v5.14-rc1
>>>>>>>
>>>>>>>      $ git rev-list  --count v5.13..v5.14-rc1
>>>>>>>      13764
>>>>>>>
>>>>>>> git bisect should find the offending commit in at most 14 tries.
>>>>>>> That's
>>>>>>> not too bad.
>>>>>> I correctly guesstimated that the problem got introduced by the usb
>>>>>> merge 79160a60
>>>>>>
>>>>>> "Merge tag 'usb-5.14-rc1' of
>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb"
>>>>>>
>>>>>> 116 commits(7 bisects).
>>>>>>
>>>>>> 24f779dac8f3efb9629adc0e486914d93dc45517 is the first bad commit
>>>>>>
>>>>>> "usb: gadget: f_uac2/u_audio: add feedback endpoint support"
>>>>>>
>>>>>> Ruslan's 3 patches are related to each other so I reverted all
>>>>>> three 24f779da...e89bb428 and applied the reverts to rc1.
>>>>>>
>>>>>> I can confirm this indeed resolves the problem in rc1.
>>>>>>
>>>>>> Is late now, tomorrow evening I will apply the reverts to rc6.
>>>>>
>>>>> With these reverts rc6 works fine as well.
>>>>>
>>>>> So, where do we go from here?
>>>>>
>>>>
>>>> I know the patches have been tested on dwc2 (by me and others).  I do
>>>> not know if Ruslan or Jerome tested them on dwc3 but probably not.
>>>> Ruslan has talked about RPi (my case too) and BeagleboneBlack, both
>>>> with dwc2. Perhaps the dwc2 behaves a bit differently than dwc3?
>>>>
>>>> The patches add a new EP-IN for async feedback. I am sorry I have not
>>>> followed your long thread (it started as unrelated to uac). Does the
>>>> problem appear with f_uac1 or f_uac2? Please how have you reached the
>>>> above problem?
>>>
>>> I'm sorry too. I first believed the issue was related to the patch
>>> mentioned in the subject line.
>>>
>>> The problem appaers with f_uac2. I bost Edison_Arduino board in host
>>> mode (there is a switch allowing to select host/device mode). When
>>> flipping the switch to device mode udev run a script:
>>> But as I am using configfs (excerpt follows) and just disabling the
>>> last 2 line resolves the issue, I'm guessing uac2 is the issue. Or
>>> exceeding the available resources.
>>>
>>> # Create directory structure
>>> mkdir "${GADGET_BASE_DIR}"
>>> cd "${GADGET_BASE_DIR}"
>>> mkdir -p configs/c.1/strings/0x409
>>> mkdir -p strings/0x409
>>>
>>> # Serial device
>>> mkdir functions/gser.usb0
>>> ln -s functions/gser.usb0 configs/c.1/
>>> ###
>>>
>>> # Ethernet device
>>> mkdir functions/eem.usb0
>>> echo "${DEV_ETH_ADDR}" > functions/eem.usb0/dev_addr
>>> echo "${HOST_ETH_ADDR}" > functions/eem.usb0/host_addr
>>> ln -s functions/eem.usb0 configs/c.1/
>>>
>>> # Mass Storage device
>>> mkdir functions/mass_storage.usb0
>>> echo 1 > functions/mass_storage.usb0/stall
>>> echo 0 > functions/mass_storage.usb0/lun.0/cdrom
>>> echo 0 > functions/mass_storage.usb0/lun.0/ro
>>> echo 0 > functions/mass_storage.usb0/lun.0/nofua
>>> echo "${USBDISK}" > functions/mass_storage.usb0/lun.0/file
>>> ln -s functions/mass_storage.usb0 configs/c.1/
>>>
>>> # UAC2 device
>>> mkdir functions/uac2.usb0
>>> ln -s functions/uac2.usb0 configs/c.1
>>> ....
>>>
>>
>> As you say, could perhaps the reason be that the extra EP-IN added in
>> those patches (previously 1, now 2 with the default config you use)
>> exceeds your EP-IN max count or available fifos somehow?  You have a
>> number of functions initialized. If you change the load order of the
>> functions, do you get the error later with a different function? Just
>> guessing...
>>
>> You should be able to switch the default async EP-OUT (which configures
>> the new feedback EP-IN ) to adaptive EP-OUT (which requires no feedback
>> EP) with c_sync=8 parameter of f_uac2.
>>
>> https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c*L47__;Iw!!A4F2R9G_pg!PZHOWnH3HDs-9a9bz4KLDShZpsubSfG84fu3wEZyK4Q_Wvv2lhHKuk7LJpWPAHQ6Lj4y$
>>
>> https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c*L830__;Iw!!A4F2R9G_pg!PZHOWnH3HDs-9a9bz4KLDShZpsubSfG84fu3wEZyK4Q_Wvv2lhHKuk7LJpWPAC7mfgo7$
>>
>> https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14-rc6/source/include/uapi/linux/usb/ch9.h*L453__;Iw!!A4F2R9G_pg!PZHOWnH3HDs-9a9bz4KLDShZpsubSfG84fu3wEZyK4Q_Wvv2lhHKuk7LJpWPANdeuGIx$
>>
>> Does that fix the problem?
>>
> 
> Hi Pavel,
> 
> I took a look at 24f779dac8f3 ("usb: gadget: f_uac2/u_audio: add
> feedback endpoint support") that Ferry reported the issue from
> bisection. I see at least a couple problems in the new UAC2 changes.
> 
> 1) usb_ep_dequeue() is asynchronous. Don't free requests before the
> controller driver give them back.
> 
> 2) Did you test with SuperSpeed? I don't see companion descriptor.
> 
> Please fix them.

Given it's late in the cycle I agree with Andy it would be best to 
revert the series now.

I will be happy to test on dwc3 when a fixed series arrives.

> Thanks,
> Thinh
> 


^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-08-22 20:10                                                 ` Ferry Toth
@ 2021-08-23  7:57                                                   ` Pavel Hofman
  0 siblings, 0 replies; 73+ messages in thread
From: Pavel Hofman @ 2021-08-23  7:57 UTC (permalink / raw)
  To: Ferry Toth, Thinh Nguyen, Felipe Balbi
  Cc: Wesley Cheng, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list, linux-usb, linux-kernel, devicetree, linux-arm-msm,
	jackp, heikki.krogerus, andy.shevchenko, Ruslan Bilovol,
	Jerome Brunet



Dne 22. 08. 21 v 22:10 Ferry Toth napsal(a):
> Op 21-08-2021 om 04:57 schreef Thinh Nguyen:
>> Pavel Hofman wrote:
>>>
>>> Dne 19. 08. 21 v 22:10 Ferry Toth napsal(a):
>>>> Hi
>>>>
>>>> Op 19-08-2021 om 09:51 schreef Pavel Hofman:
>>>>> Hi,
>>>>>
>>>>> Dne 18. 08. 21 v 21:07 Ferry Toth napsal(a):
>>>>>> Hi,
>>>>>>
>>>>>> Op 18-08-2021 om 00:00 schreef Ferry Toth:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Op 16-08-2021 om 07:18 schreef Felipe Balbi:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>>>>>>>>>>>> Hardware name: Intel Corporation Merrifield/BODEGA BAY,
>>>>>>>>>>>>>>>>>>> BIOS 542
>>>>>>>>>>>>>>>>>>> 2015.01.21:18.19.48
>>>>>>>>>>>>>>>>>>> RIP: 0010:0x500000000
>>>>>>>>>>>>>>>>>>> Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
>>>>>>>>>>>>>>>>>>> RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
>>>>>>>>>>>>>>>>>>> RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX:
>>>>>>>>>>>>>>>>>>> 0000000000000000
>>>>>>>>>>>>>>>>>>> RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI:
>>>>>>>>>>>>>>>>>>> ffff8cd546aed200
>>>>>>>>>>>>>>>>>>> RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09:
>>>>>>>>>>>>>>>>>>> 0000000000000001
>>>>>>>>>>>>>>>>>>> R10: ffff8cd541fd28c0 R11: 0000000000000000 R12:
>>>>>>>>>>>>>>>>>>> ffff8cd547342828
>>>>>>>>>>>>>>>>>>> R13: ffff8cd546aed248 R14: 0000000000000000 R15:
>>>>>>>>>>>>>>>>>>> ffff8cd548b1d000
>>>>>>>>>>>>>>>>>>> FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000)
>>>>>>>>>>>>>>>>>>> knlGS:0000000000000000
>>>>>>>>>>>>>>>>>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>>>>>>>>>>>>>>> CR2: 0000000500000000 CR3: 000000000311e000 CR4:
>>>>>>>>>>>>>>>>>>> 00000000001006f0
>>>>>>>>>>>>>>>>>>> Call Trace:
>>>>>>>>>>>>>>>>>>>          ? dwc3_remove_requests.constprop.0+0x14d/0x170
>>>>>>>>>>>>>>>>>>>          ? __dwc3_gadget_ep_disable+0x7a/0x160
>>>>>>>>>>>>>>>>>>>          ? dwc3_gadget_ep_disable+0x3d/0xd0
>>>>>>>>>>>>>>>>>>>          ? usb_ep_disable+0x1c/0x
>>>>>>>>>>>>>>>>>>>          ? u_audio_stop_capture+0x79/0x120 [u_audio]
>>>>>>>>>>>>>>>>>>>          ? afunc_set_alt+0x73/0x80 [usb_f_uac2]
>>>>>>>>>>>> So this is triggered by a SetInterface request...
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>          ? composite_setup+0x224/0x1b90 [libcomposite]
>>>>>>>>>>>>>>>>>>>          ? __dwc3_gadget_kick_transfer+0x160/0x400
>>>>>>>>>>>>>>>>>>>          ? dwc3_gadget_ep_queue+0xf3/0x1a0
>>>>>>>>>>>>>>>>>>>          ? configfs_composite_setup+0x6b/0x90
>>>>>>>>>>>>>>>>>>> [libcomposite]
>>>>>>>>>>>>>>>>>>>          ? configfs_composite_setup+0x6b/0x90
>>>>>>>>>>>>>>>>>>> [libcomposite]
>>>>>>>>>>>>>>>>>>>          ? dwc3_ep0_interrupt+0x459/0xa40
>>>>>>>>>>>>>>>>>>>          ? dwc3_thread_interrupt+0x8ee/0xf40
>>>>>>>>>>>>>>>>>>>          ? __schedule+0x235/0x6c0
>>>>>>>>>>>>>>>>>>>          ? disable_irq_nosync+0x10/0x10
>>>>>>>>>>>>>>>>>>>          ? irq_thread_fn+0x1b/0x60
>>>>>>>>>>>>>>>>>>>          ? irq_thread+0xc0/0x160
>>>>>>>>>>>>>>>>>>>          ? irq_thread_check_affinity+0x70/0x70
>>>>>>>>>>>>>>>>>>>          ? irq_forced_thread_fn+0x70/0x70
>>>>>>>>>>>>>>>>>>>          ? kthread+0x122/0x140
>>>>>>>>>>>>>>>>>>>          ? set_kthread_struct+0x40/0x40
>>>>>>>>>>>>>>>>>>>          ? ret_from_fork+0x22/0x30
>>>>>>>>>>>>>>>>>> Do you mind enabling dwc3 traces and collecting them?
>>>>>>>>>>>>>>>>>> Trying to figure
>>>>>>>>>>>>>>>>>> out how we got here.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'll try if I can get the same error by booting with USB
>>>>>>>>>>>>>>>>> in host mode
>>>>>>>>>>>>>>>>> and then switch to device mode. If so I can enable traces
>>>>>>>>>>>>>>>>> and collect as
>>>>>>>>>>>>>>>>> you explained me before.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'll try before monday, as then I fly for a holiday and
>>>>>>>>>>>>>>>>> will not be
>>>>>>>>>>>>>>>>> available before rc5.
>>>>>>>>>>>>>>>> you can enable all of those with kernel cmdline :-)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> https://urldefense.com/v3/__https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html__;!!A4F2R9G_pg!PZHOWnH3HDs-9a9bz4KLDShZpsubSfG84fu3wEZyK4Q_Wvv2lhHKuk7LJpWPAPch0qja$ 
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> you need ftrace_dump_on_oops=1 and also need the correct
>>>>>>>>>>>>>>>> options on
>>>>>>>>>>>>>>>> trace_buf_size and trace_event.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Edison-Arduino I have a switch to go to device mode,
>>>>>>>>>>>>>>> after which
>>>>>>>>>>>>>>> udev triggers a script configure gadgets through configfs.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I tried to log following these instructions:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://urldefense.com/v3/__https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html*reporting-bugs__;Iw!!A4F2R9G_pg!PZHOWnH3HDs-9a9bz4KLDShZpsubSfG84fu3wEZyK4Q_Wvv2lhHKuk7LJpWPAG64agnA$ 
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html*reporting-bugs__;Iw!!A4F2R9G_pg!PZHOWnH3HDs-9a9bz4KLDShZpsubSfG84fu3wEZyK4Q_Wvv2lhHKuk7LJpWPAG64agnA$ 
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Unfortunately the kernel crashes so badly I can not get to
>>>>>>>>>>>>>>> the ` cp
>>>>>>>>>>>>>>> /t/trace /root/trace.txt` line (after a while the watchdog
>>>>>>>>>>>>>>> kicks).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What to do next?
>>>>>>>>>>>>>> Pass ftrace_dump_on_oops to kernel cmdline.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> No sure if I did this right, on oops everything is pushed to
>>>>>>>>>>>>> console
>>>>>>>>>>>>> (115k2 serial), I hope nothing essential is lost.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I copied the screen buffer to file see attached.
>>>>>>>>>>>> Thank you, I bet it took quite a some time :-) Anyway, 
>>>>>>>>>>>> looking at
>>>>>>>>>>>> the logs around Set Interface requests, we can track every
>>>>>>>>>>>> endpoint
>>>>>>>>>>>> that's disabled. I'll take a guess and assume we're failing at
>>>>>>>>>>>> the last
>>>>>>>>>>>> Set Interface, that means we should have something odd with
>>>>>>>>>>>> ep6in, but
>>>>>>>>>>>> everything looks fine in the trace output:
>>>>>>>>>>>>
>>>>>>>>>>>> [   75.823107] irq/14-d-596       0d... 42789194us :
>>>>>>>>>>>> dwc3_gadget_ep_enable: ep6in: mps 192/346 streams 16 burst 0
>>>>>>>>>>>> ring 0/0 flags E:swbp:<
>>>>>>>>>>>> [   75.835472] irq/14-d-596       0d... 42789198us :
>>>>>>>>>>>> dwc3_alloc_request: ep6in: req 0000000002c71409 length 0/0 zsI
>>>>>>>>>>>> ==> 0
>>>>>>>>>>>> [   75.846416] irq/14-d-596       0d... 42789202us :
>>>>>>>>>>>> dwc3_ep_queue: ep6in: req 0000000002c71409 length 0/192 zsI
>>>>>>>>>>>> ==> -115
>>>>>>>>>>>> [   75.857360] irq/14-d-596       0d... 42789204us :
>>>>>>>>>>>> dwc3_alloc_request: ep6in: req 00000000a324f5d0 length 0/0 zsI
>>>>>>>>>>>> ==> 0
>>>>>>>>>>>> [   75.868301] irq/14-d-596       0d... 42789206us :
>>>>>>>>>>>> dwc3_ep_queue: ep6in: req 00000000a324f5d0 length 0/192 zsI
>>>>>>>>>>>> ==> -115
>>>>>>>>>>>> [   75.879244] irq/14-d-596       0d... 42789209us :
>>>>>>>>>>>> dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0]
>>>>>>>>>>>> (Not Active) [Status Phase]
>>>>>>>>>>>> [   75.891880] irq/14-d-596       0d... 42789211us :
>>>>>>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>>>>>>>>>> [   75.989131] irq/14-d-596       0d... 42789224us :
>>>>>>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params
>>>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>>>> [   76.096261] irq/14-d-596       0d... 42789272us :
>>>>>>>>>>>> dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL)
>>>>>>>>>>>> [Status Phase]
>>>>>>>>>>>> [   76.107834] irq/14-d-596       0d... 42789275us :
>>>>>>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>>>>>>>>>> [   76.122944] irq/14-d-596       0d... 42789277us :
>>>>>>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0
>>>>>>>>>>>> zsI ==> 0
>>>>>>>>>>>> [   76.134160] irq/14-d-596       0d... 42789280us :
>>>>>>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>>>> 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>>>>>>>>>> [   76.231322] irq/14-d-596       0d... 42789292us :
>>>>>>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params
>>>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>>>> [   76.297418] kworker/-23        0d... 42789670us :
>>>>>>>>>>>> dwc3_ep_queue: ep3in: req 0000000029586135 length 0/96 ZsI ==>
>>>>>>>>>>>> -115
>>>>>>>>>>>> [   76.308278] kworker/-23        0d... 42789695us :
>>>>>>>>>>>> dwc3_prepare_trb: ep3in: trb 00000000b81213d6 (E1:D0) buf
>>>>>>>>>>>> 0000000003b7a800 size 96 ctrl 00000811 (Hlcs:sC:normal)
>>>>>>>>>>>> [   76.395294] kworker/-23        0d... 42789707us :
>>>>>>>>>>>> dwc3_gadget_ep_cmd: ep3in: cmd 'Update Transfer' [60007]
>>>>>>>>>>>> params 00000000 00000000 00000000 --> status: Successful
>>>>>>>>>>>> [   76.471900] irq/14-d-596       0d... 42789842us :
>>>>>>>>>>>> dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL)
>>>>>>>>>>>> [Setup Phase]
>>>>>>>>>>>> [   76.489308] irq/14-d-596       0d... 42789845us :
>>>>>>>>>>>> dwc3_ctrl_req: Set Interface(Intf = 5, Alt.Setting = 0)
>>>>>>>>>>>> [   76.505650] irq/14-d-596       0d... 42789851us :
>>>>>>>>>>>> dwc3_ep_dequeue: ep6in: req 0000000002c71409 length 0/192 zsI
>>>>>>>>>>>> ==> -115
>>>>>>>>>>>> [   76.523315] irq/14-d-596       0d... 42789854us :
>>>>>>>>>>>> dwc3_gadget_giveback: ep6in: req 0000000002c71409 length 0/192
>>>>>>>>>>>> zsI ==> -104
>>>>>>>>>>>> [   76.541427] irq/14-d-596       0d... 42789857us :
>>>>>>>>>>>> dwc3_free_request: ep6in: req 0000000002c71409 length 0/192
>>>>>>>>>>>> zsI ==> -104
>>>>>>>>>>>> [   76.559267] irq/14-d-596       0d... 42789859us :
>>>>>>>>>>>> dwc3_ep_dequeue: ep6in: req 00000000a324f5d0 length 0/192 zsI
>>>>>>>>>>>> ==> -115
>>>>>>>>>>>> [   76.576937] irq/14-d-596       0d... 42789861us :
>>>>>>>>>>>> dwc3_gadget_giveback: ep6in: req 00000000a324f5d0 length 0/192
>>>>>>>>>>>> zsI ==> -104
>>>>>>>>>>>> [   76.595046] irq/14-d-596       0d... 42789862us :
>>>>>>>>>>>> dwc3_free_request: ep6in: req 00000000a324f5d0 length 0/192
>>>>>>>>>>>> zsI ==> -104
>>>>>>>>>>>> [   76.612892] irq/14-d-596       0d... 42789865us :
>>>>>>>>>>>> dwc3_gadget_ep_disable: ep6in: mps 192/346 streams 16 burst 0
>>>>>>>>>>>> ring 0/0 flags E:swbp:<
>>>>>>>>>>>> [   76.665535] irq/14-d-596       0d... 42789873us :
>>>>>>>>>>>> dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0]
>>>>>>>>>>>> (Not Active) [Status Phase]
>>>>>>>>>>>> [   76.684716] irq/14-d-596       0d... 42789875us :
>>>>>>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>>>>>>>>>> [   76.819195] irq/14-d-596       0d... 42789886us :
>>>>>>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params
>>>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>>>> [   76.926324] irq/14-d-596       0d... 42789930us :
>>>>>>>>>>>> dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL)
>>>>>>>>>>>> [Status Phase]
>>>>>>>>>>>> [   76.937892] irq/14-d-596       0d... 42789933us :
>>>>>>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>>>>>>>>>> [   76.953003] irq/14-d-596       0d... 42789935us :
>>>>>>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0
>>>>>>>>>>>> zsI ==> 0
>>>>>>>>>>>> [   76.964217] irq/14-d-596       0d... 42789938us :
>>>>>>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>>>> 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>>>>>>>>>> [   77.061379] irq/14-d-596       0d... 42789950us :
>>>>>>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params
>>>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>>>> [   77.168595] irq/14-d-596       0d... 42790509us :
>>>>>>>>>>>> dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL)
>>>>>>>>>>>> [Setup Phase]
>>>>>>>>>>>> [   77.180159] irq/14-d-596       0d... 42790512us :
>>>>>>>>>>>> dwc3_ctrl_req: Get String Descriptor(Index = 18, Length = 255)
>>>>>>>>>>>> [   77.190578] irq/14-d-596       0d... 42790537us :
>>>>>>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>>>> 0000000003b68000 size 36 ctrl 00000c53 (HLcs:SC:data)
>>>>>>>>>>>> [   77.287648] irq/14-d-596       0d... 42790550us :
>>>>>>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params
>>>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>>>> [   77.333107] irq/14-d-596       0d... 42790557us :
>>>>>>>>>>>> dwc3_event: event (000010c2): ep0in: Transfer Not Ready [0]
>>>>>>>>>>>> (Not Active) [Data Phase]
>>>>>>>>>>>> [   77.407223] irq/14-d-596       0d... 42790575us :
>>>>>>>>>>>> dwc3_event: event (000090c2): ep0in: Transfer Not Ready [0]
>>>>>>>>>>>> (Active) [Data Phase]
>>>>>>>>>>>> [   77.480985] irq/14-d-596       0d... 42790588us :
>>>>>>>>>>>> dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL)
>>>>>>>>>>>> [Data Phase]
>>>>>>>>>>>> [   77.492376] irq/14-d-596       0d... 42790590us :
>>>>>>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>>>> 0000000003b68000 size 0 ctrl 00000c52 (hLcs:SC:data)
>>>>>>>>>>>> [   77.507221] irq/14-d-596       0d... 42790595us :
>>>>>>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length
>>>>>>>>>>>> 36/36 ZsI ==> 0
>>>>>>>>>>>> [   77.518609] irq/14-d-596       0d... 42790597us :
>>>>>>>>>>>> dwc3_event: event (000020c0): ep0out: Transfer Not Ready [0]
>>>>>>>>>>>> (Not Active) [Status Phase]
>>>>>>>>>>>> [   77.531332] irq/14-d-596       0d... 42790598us :
>>>>>>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf
>>>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c43 (HLcs:SC:status3)
>>>>>>>>>>>> [   77.628669] irq/14-d-596       0d... 42790609us :
>>>>>>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params
>>>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>>>>
>>>>>>>>>>>> Do you mind adding a few prints in dwc3_remove_requests to
>>>>>>>>>>>> tell us which
>>>>>>>>>>>> endpoint is being processed? Then we'll know for sure which
>>>>>>>>>>>> one caused
>>>>>>>>>>>> the crash.
>>>>>>>>>>>>
>>>>>>>>>>> I wouldn't mind but am leaving on a holiday, won't have time
>>>>>>>>>>> until 6 aug.
>>>>>>>>>> not a problem, we'll still be here when you're back :-)
>>>>>>>>> Well, let's go then :-)
>>>>>>>>>
>>>>>>>>> To get back in the mood I have retested 5.13.0, 5.14.0-rc1,
>>>>>>>>> 5.14.0-rc2
>>>>>>>>> and 5.14.0-rc5.
>>>>>>>>>
>>>>>>>>> I find that 5.13.0 works fine, and the issue starts from 
>>>>>>>>> 5.14.0-rc1.
>>>>>>>> That's great finding. We have a bisection point. There are a 
>>>>>>>> total of
>>>>>>>> 13764 commits between v5.13 and v5.14-rc1
>>>>>>>>
>>>>>>>>      $ git rev-list  --count v5.13..v5.14-rc1
>>>>>>>>      13764
>>>>>>>>
>>>>>>>> git bisect should find the offending commit in at most 14 tries.
>>>>>>>> That's
>>>>>>>> not too bad.
>>>>>>> I correctly guesstimated that the problem got introduced by the usb
>>>>>>> merge 79160a60
>>>>>>>
>>>>>>> "Merge tag 'usb-5.14-rc1' of
>>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb"
>>>>>>>
>>>>>>> 116 commits(7 bisects).
>>>>>>>
>>>>>>> 24f779dac8f3efb9629adc0e486914d93dc45517 is the first bad commit
>>>>>>>
>>>>>>> "usb: gadget: f_uac2/u_audio: add feedback endpoint support"
>>>>>>>
>>>>>>> Ruslan's 3 patches are related to each other so I reverted all
>>>>>>> three 24f779da...e89bb428 and applied the reverts to rc1.
>>>>>>>
>>>>>>> I can confirm this indeed resolves the problem in rc1.
>>>>>>>
>>>>>>> Is late now, tomorrow evening I will apply the reverts to rc6.
>>>>>>
>>>>>> With these reverts rc6 works fine as well.
>>>>>>
>>>>>> So, where do we go from here?
>>>>>>
>>>>>
>>>>> I know the patches have been tested on dwc2 (by me and others).  I do
>>>>> not know if Ruslan or Jerome tested them on dwc3 but probably not.
>>>>> Ruslan has talked about RPi (my case too) and BeagleboneBlack, both
>>>>> with dwc2. Perhaps the dwc2 behaves a bit differently than dwc3?
>>>>>
>>>>> The patches add a new EP-IN for async feedback. I am sorry I have not
>>>>> followed your long thread (it started as unrelated to uac). Does the
>>>>> problem appear with f_uac1 or f_uac2? Please how have you reached the
>>>>> above problem?
>>>>
>>>> I'm sorry too. I first believed the issue was related to the patch
>>>> mentioned in the subject line.
>>>>
>>>> The problem appaers with f_uac2. I bost Edison_Arduino board in host
>>>> mode (there is a switch allowing to select host/device mode). When
>>>> flipping the switch to device mode udev run a script:
>>>> But as I am using configfs (excerpt follows) and just disabling the
>>>> last 2 line resolves the issue, I'm guessing uac2 is the issue. Or
>>>> exceeding the available resources.
>>>>
>>>> # Create directory structure
>>>> mkdir "${GADGET_BASE_DIR}"
>>>> cd "${GADGET_BASE_DIR}"
>>>> mkdir -p configs/c.1/strings/0x409
>>>> mkdir -p strings/0x409
>>>>
>>>> # Serial device
>>>> mkdir functions/gser.usb0
>>>> ln -s functions/gser.usb0 configs/c.1/
>>>> ###
>>>>
>>>> # Ethernet device
>>>> mkdir functions/eem.usb0
>>>> echo "${DEV_ETH_ADDR}" > functions/eem.usb0/dev_addr
>>>> echo "${HOST_ETH_ADDR}" > functions/eem.usb0/host_addr
>>>> ln -s functions/eem.usb0 configs/c.1/
>>>>
>>>> # Mass Storage device
>>>> mkdir functions/mass_storage.usb0
>>>> echo 1 > functions/mass_storage.usb0/stall
>>>> echo 0 > functions/mass_storage.usb0/lun.0/cdrom
>>>> echo 0 > functions/mass_storage.usb0/lun.0/ro
>>>> echo 0 > functions/mass_storage.usb0/lun.0/nofua
>>>> echo "${USBDISK}" > functions/mass_storage.usb0/lun.0/file
>>>> ln -s functions/mass_storage.usb0 configs/c.1/
>>>>
>>>> # UAC2 device
>>>> mkdir functions/uac2.usb0
>>>> ln -s functions/uac2.usb0 configs/c.1
>>>> ....
>>>>
>>>
>>> As you say, could perhaps the reason be that the extra EP-IN added in
>>> those patches (previously 1, now 2 with the default config you use)
>>> exceeds your EP-IN max count or available fifos somehow?  You have a
>>> number of functions initialized. If you change the load order of the
>>> functions, do you get the error later with a different function? Just
>>> guessing...
>>>
>>> You should be able to switch the default async EP-OUT (which configures
>>> the new feedback EP-IN ) to adaptive EP-OUT (which requires no feedback
>>> EP) with c_sync=8 parameter of f_uac2.
>>>
>>> https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c*L47__;Iw!!A4F2R9G_pg!PZHOWnH3HDs-9a9bz4KLDShZpsubSfG84fu3wEZyK4Q_Wvv2lhHKuk7LJpWPAHQ6Lj4y$ 
>>>
>>>
>>> https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c*L830__;Iw!!A4F2R9G_pg!PZHOWnH3HDs-9a9bz4KLDShZpsubSfG84fu3wEZyK4Q_Wvv2lhHKuk7LJpWPAC7mfgo7$ 
>>>
>>>
>>> https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14-rc6/source/include/uapi/linux/usb/ch9.h*L453__;Iw!!A4F2R9G_pg!PZHOWnH3HDs-9a9bz4KLDShZpsubSfG84fu3wEZyK4Q_Wvv2lhHKuk7LJpWPANdeuGIx$ 
>>>
>>>
>>> Does that fix the problem?
>>>
>>
>> Hi Pavel,
>>
>> I took a look at 24f779dac8f3 ("usb: gadget: f_uac2/u_audio: add
>> feedback endpoint support") that Ferry reported the issue from
>> bisection. I see at least a couple problems in the new UAC2 changes.
>>
>> 1) usb_ep_dequeue() is asynchronous. Don't free requests before the
>> controller driver give them back.
>>
>> 2) Did you test with SuperSpeed? I don't see companion descriptor.
>>
>> Please fix them.
> 
> Given it's late in the cycle I agree with Andy it would be best to 
> revert the series now.
> 
> I will be happy to test on dwc3 when a fixed series arrives.

Yes, I agree it's the best solution now. I have asked the patch devs 
Ruslan and Jerome for help with these issues, I am mostly a tester of 
the UAC2 gadget changes (with dwc2 USB2 only available for now).

Best regards,

Pavel.

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-08-22 19:43                                               ` Ferry Toth
@ 2021-08-23 14:59                                                 ` Pavel Hofman
  2021-08-23 15:21                                                   ` Andy Shevchenko
  0 siblings, 1 reply; 73+ messages in thread
From: Pavel Hofman @ 2021-08-23 14:59 UTC (permalink / raw)
  To: Ferry Toth, Felipe Balbi
  Cc: Wesley Cheng, gregkh, robh+dt, agross, bjorn.andersson,
	frowand.list, linux-usb, linux-kernel, devicetree, linux-arm-msm,
	jackp, heikki.krogerus, andy.shevchenko, Ruslan Bilovol


Dne 22. 08. 21 v 21:43 Ferry Toth napsal(a):
> Op 19-08-2021 om 23:04 schreef Pavel Hofman:
>>
>> Dne 19. 08. 21 v 22:10 Ferry Toth napsal(a):
>>> Hi
>>>
>>> Op 19-08-2021 om 09:51 schreef Pavel Hofman:
>>>> Hi,
>>>>
>>>> Dne 18. 08. 21 v 21:07 Ferry Toth napsal(a):
>>>>> Hi,
>>>>>
>>>>> Op 18-08-2021 om 00:00 schreef Ferry Toth:
>>>>>> Hi,
>>>>>>
>>>>>> Op 16-08-2021 om 07:18 schreef Felipe Balbi:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>>>>>> Ferry Toth <fntoth@gmail.com> writes:
>>>>>>>>>>>>>>>>>> Hardware name: Intel Corporation Merrifield/BODEGA 
>>>>>>>>>>>>>>>>>> BAY, BIOS 542
>>>>>>>>>>>>>>>>>> 2015.01.21:18.19.48
>>>>>>>>>>>>>>>>>> RIP: 0010:0x500000000
>>>>>>>>>>>>>>>>>> Code: Unable to access opcode bytes at RIP 0x4ffffffd6.
>>>>>>>>>>>>>>>>>> RSP: 0018:ffffa4d00045fc28 EFLAGS: 00010046
>>>>>>>>>>>>>>>>>> RAX: 0000000500000000 RBX: ffff8cd546aed200 RCX: 
>>>>>>>>>>>>>>>>>> 0000000000000000
>>>>>>>>>>>>>>>>>> RDX: 0000000000000000 RSI: ffff8cd547bfcae0 RDI: 
>>>>>>>>>>>>>>>>>> ffff8cd546aed200
>>>>>>>>>>>>>>>>>> RBP: ffff8cd547bfcae0 R08: 0000000000000000 R09: 
>>>>>>>>>>>>>>>>>> 0000000000000001
>>>>>>>>>>>>>>>>>> R10: ffff8cd541fd28c0 R11: 0000000000000000 R12: 
>>>>>>>>>>>>>>>>>> ffff8cd547342828
>>>>>>>>>>>>>>>>>> R13: ffff8cd546aed248 R14: 0000000000000000 R15: 
>>>>>>>>>>>>>>>>>> ffff8cd548b1d000
>>>>>>>>>>>>>>>>>> FS:  0000000000000000(0000) GS:ffff8cd57e200000(0000) 
>>>>>>>>>>>>>>>>>> knlGS:0000000000000000
>>>>>>>>>>>>>>>>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>>>>>>>>>>>>>> CR2: 0000000500000000 CR3: 000000000311e000 CR4: 
>>>>>>>>>>>>>>>>>> 00000000001006f0
>>>>>>>>>>>>>>>>>> Call Trace:
>>>>>>>>>>>>>>>>>>         ? dwc3_remove_requests.constprop.0+0x14d/0x170
>>>>>>>>>>>>>>>>>>         ? __dwc3_gadget_ep_disable+0x7a/0x160
>>>>>>>>>>>>>>>>>>         ? dwc3_gadget_ep_disable+0x3d/0xd0
>>>>>>>>>>>>>>>>>>         ? usb_ep_disable+0x1c/0x
>>>>>>>>>>>>>>>>>>         ? u_audio_stop_capture+0x79/0x120 [u_audio]
>>>>>>>>>>>>>>>>>>         ? afunc_set_alt+0x73/0x80 [usb_f_uac2]
>>>>>>>>>>> So this is triggered by a SetInterface request...
>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>         ? composite_setup+0x224/0x1b90 [libcomposite]
>>>>>>>>>>>>>>>>>>         ? __dwc3_gadget_kick_transfer+0x160/0x400
>>>>>>>>>>>>>>>>>>         ? dwc3_gadget_ep_queue+0xf3/0x1a0
>>>>>>>>>>>>>>>>>>         ? configfs_composite_setup+0x6b/0x90 
>>>>>>>>>>>>>>>>>> [libcomposite]
>>>>>>>>>>>>>>>>>>         ? configfs_composite_setup+0x6b/0x90 
>>>>>>>>>>>>>>>>>> [libcomposite]
>>>>>>>>>>>>>>>>>>         ? dwc3_ep0_interrupt+0x459/0xa40
>>>>>>>>>>>>>>>>>>         ? dwc3_thread_interrupt+0x8ee/0xf40
>>>>>>>>>>>>>>>>>>         ? __schedule+0x235/0x6c0
>>>>>>>>>>>>>>>>>>         ? disable_irq_nosync+0x10/0x10
>>>>>>>>>>>>>>>>>>         ? irq_thread_fn+0x1b/0x60
>>>>>>>>>>>>>>>>>>         ? irq_thread+0xc0/0x160
>>>>>>>>>>>>>>>>>>         ? irq_thread_check_affinity+0x70/0x70
>>>>>>>>>>>>>>>>>>         ? irq_forced_thread_fn+0x70/0x70
>>>>>>>>>>>>>>>>>>         ? kthread+0x122/0x140
>>>>>>>>>>>>>>>>>>         ? set_kthread_struct+0x40/0x40
>>>>>>>>>>>>>>>>>>         ? ret_from_fork+0x22/0x30
>>>>>>>>>>>>>>>>> Do you mind enabling dwc3 traces and collecting them? 
>>>>>>>>>>>>>>>>> Trying to figure
>>>>>>>>>>>>>>>>> out how we got here.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'll try if I can get the same error by booting with USB 
>>>>>>>>>>>>>>>> in host mode
>>>>>>>>>>>>>>>> and then switch to device mode. If so I can enable 
>>>>>>>>>>>>>>>> traces and collect as
>>>>>>>>>>>>>>>> you explained me before.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'll try before monday, as then I fly for a holiday and 
>>>>>>>>>>>>>>>> will not be
>>>>>>>>>>>>>>>> available before rc5.
>>>>>>>>>>>>>>> you can enable all of those with kernel cmdline :-)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html 
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> you need ftrace_dump_on_oops=1 and also need the correct 
>>>>>>>>>>>>>>> options on
>>>>>>>>>>>>>>> trace_buf_size and trace_event.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Edison-Arduino I have a switch to go to device mode, 
>>>>>>>>>>>>>> after which
>>>>>>>>>>>>>> udev triggers a script configure gadgets through configfs.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I tried to log following these instructions:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs 
>>>>>>>>>>>>>> <https://www.kernel.org/doc/html/latest/driver-api/usb/dwc3.html#reporting-bugs> 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Unfortunately the kernel crashes so badly I can not get to 
>>>>>>>>>>>>>> the ` cp
>>>>>>>>>>>>>> /t/trace /root/trace.txt` line (after a while the watchdog 
>>>>>>>>>>>>>> kicks).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What to do next?
>>>>>>>>>>>>> Pass ftrace_dump_on_oops to kernel cmdline.
>>>>>>>>>>>>>
>>>>>>>>>>>> No sure if I did this right, on oops everything is pushed to 
>>>>>>>>>>>> console
>>>>>>>>>>>> (115k2 serial), I hope nothing essential is lost.
>>>>>>>>>>>>
>>>>>>>>>>>> I copied the screen buffer to file see attached.
>>>>>>>>>>> Thank you, I bet it took quite a some time :-) Anyway, 
>>>>>>>>>>> looking at
>>>>>>>>>>> the logs around Set Interface requests, we can track every 
>>>>>>>>>>> endpoint
>>>>>>>>>>> that's disabled. I'll take a guess and assume we're failing 
>>>>>>>>>>> at the last
>>>>>>>>>>> Set Interface, that means we should have something odd with 
>>>>>>>>>>> ep6in, but
>>>>>>>>>>> everything looks fine in the trace output:
>>>>>>>>>>>
>>>>>>>>>>> [   75.823107] irq/14-d-596       0d... 42789194us : 
>>>>>>>>>>> dwc3_gadget_ep_enable: ep6in: mps 192/346 streams 16 burst 0 
>>>>>>>>>>> ring 0/0 flags E:swbp:<
>>>>>>>>>>> [   75.835472] irq/14-d-596       0d... 42789198us : 
>>>>>>>>>>> dwc3_alloc_request: ep6in: req 0000000002c71409 length 0/0 
>>>>>>>>>>> zsI ==> 0
>>>>>>>>>>> [   75.846416] irq/14-d-596       0d... 42789202us : 
>>>>>>>>>>> dwc3_ep_queue: ep6in: req 0000000002c71409 length 0/192 zsI 
>>>>>>>>>>> ==> -115
>>>>>>>>>>> [   75.857360] irq/14-d-596       0d... 42789204us : 
>>>>>>>>>>> dwc3_alloc_request: ep6in: req 00000000a324f5d0 length 0/0 
>>>>>>>>>>> zsI ==> 0
>>>>>>>>>>> [   75.868301] irq/14-d-596       0d... 42789206us : 
>>>>>>>>>>> dwc3_ep_queue: ep6in: req 00000000a324f5d0 length 0/192 zsI 
>>>>>>>>>>> ==> -115
>>>>>>>>>>> [   75.879244] irq/14-d-596       0d... 42789209us : 
>>>>>>>>>>> dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0] 
>>>>>>>>>>> (Not Active) [Status Phase]
>>>>>>>>>>> [   75.891880] irq/14-d-596       0d... 42789211us : 
>>>>>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>>>>>>>>> [   75.989131] irq/14-d-596       0d... 42789224us : 
>>>>>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 
>>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>>> [   76.096261] irq/14-d-596       0d... 42789272us : 
>>>>>>>>>>> dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) 
>>>>>>>>>>> [Status Phase]
>>>>>>>>>>> [   76.107834] irq/14-d-596       0d... 42789275us : 
>>>>>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>>>>>>>>> [   76.122944] irq/14-d-596       0d... 42789277us : 
>>>>>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 
>>>>>>>>>>> zsI ==> 0
>>>>>>>>>>> [   76.134160] irq/14-d-596       0d... 42789280us : 
>>>>>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>>>> 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>>>>>>>>> [   76.231322] irq/14-d-596       0d... 42789292us : 
>>>>>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 
>>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>>> [   76.297418] kworker/-23        0d... 42789670us : 
>>>>>>>>>>> dwc3_ep_queue: ep3in: req 0000000029586135 length 0/96 ZsI 
>>>>>>>>>>> ==> -115
>>>>>>>>>>> [   76.308278] kworker/-23        0d... 42789695us : 
>>>>>>>>>>> dwc3_prepare_trb: ep3in: trb 00000000b81213d6 (E1:D0) buf 
>>>>>>>>>>> 0000000003b7a800 size 96 ctrl 00000811 (Hlcs:sC:normal)
>>>>>>>>>>> [   76.395294] kworker/-23        0d... 42789707us : 
>>>>>>>>>>> dwc3_gadget_ep_cmd: ep3in: cmd 'Update Transfer' [60007] 
>>>>>>>>>>> params 00000000 00000000 00000000 --> status: Successful
>>>>>>>>>>> [   76.471900] irq/14-d-596       0d... 42789842us : 
>>>>>>>>>>> dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL) 
>>>>>>>>>>> [Setup Phase]
>>>>>>>>>>> [   76.489308] irq/14-d-596       0d... 42789845us : 
>>>>>>>>>>> dwc3_ctrl_req: Set Interface(Intf = 5, Alt.Setting = 0)
>>>>>>>>>>> [   76.505650] irq/14-d-596       0d... 42789851us : 
>>>>>>>>>>> dwc3_ep_dequeue: ep6in: req 0000000002c71409 length 0/192 zsI 
>>>>>>>>>>> ==> -115
>>>>>>>>>>> [   76.523315] irq/14-d-596       0d... 42789854us : 
>>>>>>>>>>> dwc3_gadget_giveback: ep6in: req 0000000002c71409 length 
>>>>>>>>>>> 0/192 zsI ==> -104
>>>>>>>>>>> [   76.541427] irq/14-d-596       0d... 42789857us : 
>>>>>>>>>>> dwc3_free_request: ep6in: req 0000000002c71409 length 0/192 
>>>>>>>>>>> zsI ==> -104
>>>>>>>>>>> [   76.559267] irq/14-d-596       0d... 42789859us : 
>>>>>>>>>>> dwc3_ep_dequeue: ep6in: req 00000000a324f5d0 length 0/192 zsI 
>>>>>>>>>>> ==> -115
>>>>>>>>>>> [   76.576937] irq/14-d-596       0d... 42789861us : 
>>>>>>>>>>> dwc3_gadget_giveback: ep6in: req 00000000a324f5d0 length 
>>>>>>>>>>> 0/192 zsI ==> -104
>>>>>>>>>>> [   76.595046] irq/14-d-596       0d... 42789862us : 
>>>>>>>>>>> dwc3_free_request: ep6in: req 00000000a324f5d0 length 0/192 
>>>>>>>>>>> zsI ==> -104
>>>>>>>>>>> [   76.612892] irq/14-d-596       0d... 42789865us : 
>>>>>>>>>>> dwc3_gadget_ep_disable: ep6in: mps 192/346 streams 16 burst 0 
>>>>>>>>>>> ring 0/0 flags E:swbp:<
>>>>>>>>>>> [   76.665535] irq/14-d-596       0d... 42789873us : 
>>>>>>>>>>> dwc3_event: event (000020c2): ep0in: Transfer Not Ready [0] 
>>>>>>>>>>> (Not Active) [Status Phase]
>>>>>>>>>>> [   76.684716] irq/14-d-596       0d... 42789875us : 
>>>>>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c33 (HLcs:SC:status2)
>>>>>>>>>>> [   76.819195] irq/14-d-596       0d... 42789886us : 
>>>>>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 
>>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>>> [   76.926324] irq/14-d-596       0d... 42789930us : 
>>>>>>>>>>> dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) 
>>>>>>>>>>> [Status Phase]
>>>>>>>>>>> [   76.937892] irq/14-d-596       0d... 42789933us : 
>>>>>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c32 (hLcs:SC:status2)
>>>>>>>>>>> [   76.953003] irq/14-d-596       0d... 42789935us : 
>>>>>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 0/0 
>>>>>>>>>>> zsI ==> 0
>>>>>>>>>>> [   76.964217] irq/14-d-596       0d... 42789938us : 
>>>>>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>>>> 000000001bded000 size 8 ctrl 00000c23 (HLcs:SC:setup)
>>>>>>>>>>> [   77.061379] irq/14-d-596       0d... 42789950us : 
>>>>>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 
>>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>>> [   77.168595] irq/14-d-596       0d... 42790509us : 
>>>>>>>>>>> dwc3_event: event (0000c040): ep0out: Transfer Complete (sIL) 
>>>>>>>>>>> [Setup Phase]
>>>>>>>>>>> [   77.180159] irq/14-d-596       0d... 42790512us : 
>>>>>>>>>>> dwc3_ctrl_req: Get String Descriptor(Index = 18, Length = 255)
>>>>>>>>>>> [   77.190578] irq/14-d-596       0d... 42790537us : 
>>>>>>>>>>> dwc3_prepare_trb: ep0in: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>>>> 0000000003b68000 size 36 ctrl 00000c53 (HLcs:SC:data)
>>>>>>>>>>> [   77.287648] irq/14-d-596       0d... 42790550us : 
>>>>>>>>>>> dwc3_gadget_ep_cmd: ep0in: cmd 'Start Transfer' [406] params 
>>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>>> [   77.333107] irq/14-d-596       0d... 42790557us : 
>>>>>>>>>>> dwc3_event: event (000010c2): ep0in: Transfer Not Ready [0] 
>>>>>>>>>>> (Not Active) [Data Phase]
>>>>>>>>>>> [   77.407223] irq/14-d-596       0d... 42790575us : 
>>>>>>>>>>> dwc3_event: event (000090c2): ep0in: Transfer Not Ready [0] 
>>>>>>>>>>> (Active) [Data Phase]
>>>>>>>>>>> [   77.480985] irq/14-d-596       0d... 42790588us : 
>>>>>>>>>>> dwc3_event: event (0000c042): ep0in: Transfer Complete (sIL) 
>>>>>>>>>>> [Data Phase]
>>>>>>>>>>> [   77.492376] irq/14-d-596       0d... 42790590us : 
>>>>>>>>>>> dwc3_complete_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>>>> 0000000003b68000 size 0 ctrl 00000c52 (hLcs:SC:data)
>>>>>>>>>>> [   77.507221] irq/14-d-596       0d... 42790595us : 
>>>>>>>>>>> dwc3_gadget_giveback: ep0out: req 00000000cb1bd3cd length 
>>>>>>>>>>> 36/36 ZsI ==> 0
>>>>>>>>>>> [   77.518609] irq/14-d-596       0d... 42790597us : 
>>>>>>>>>>> dwc3_event: event (000020c0): ep0out: Transfer Not Ready [0] 
>>>>>>>>>>> (Not Active) [Status Phase]
>>>>>>>>>>> [   77.531332] irq/14-d-596       0d... 42790598us : 
>>>>>>>>>>> dwc3_prepare_trb: ep0out: trb 000000004c0ae319 (E0:D0) buf 
>>>>>>>>>>> 000000001bded000 size 0 ctrl 00000c43 (HLcs:SC:status3)
>>>>>>>>>>> [   77.628669] irq/14-d-596       0d... 42790609us : 
>>>>>>>>>>> dwc3_gadget_ep_cmd: ep0out: cmd 'Start Transfer' [406] params 
>>>>>>>>>>> 00000000 1bded000 00000000 --> status: Successful
>>>>>>>>>>>
>>>>>>>>>>> Do you mind adding a few prints in dwc3_remove_requests to 
>>>>>>>>>>> tell us which
>>>>>>>>>>> endpoint is being processed? Then we'll know for sure which 
>>>>>>>>>>> one caused
>>>>>>>>>>> the crash.
>>>>>>>>>>>
>>>>>>>>>> I wouldn't mind but am leaving on a holiday, won't have time 
>>>>>>>>>> until 6 aug.
>>>>>>>>> not a problem, we'll still be here when you're back :-)
>>>>>>>> Well, let's go then :-)
>>>>>>>>
>>>>>>>> To get back in the mood I have retested 5.13.0, 5.14.0-rc1, 
>>>>>>>> 5.14.0-rc2
>>>>>>>> and 5.14.0-rc5.
>>>>>>>>
>>>>>>>> I find that 5.13.0 works fine, and the issue starts from 
>>>>>>>> 5.14.0-rc1.
>>>>>>> That's great finding. We have a bisection point. There are a 
>>>>>>> total of
>>>>>>> 13764 commits between v5.13 and v5.14-rc1
>>>>>>>
>>>>>>>     $ git rev-list  --count v5.13..v5.14-rc1
>>>>>>>     13764
>>>>>>>
>>>>>>> git bisect should find the offending commit in at most 14 tries. 
>>>>>>> That's
>>>>>>> not too bad.
>>>>>> I correctly guesstimated that the problem got introduced by the 
>>>>>> usb merge 79160a60
>>>>>>
>>>>>> "Merge tag 'usb-5.14-rc1' of 
>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb"
>>>>>>
>>>>>> 116 commits(7 bisects).
>>>>>>
>>>>>> 24f779dac8f3efb9629adc0e486914d93dc45517 is the first bad commit
>>>>>>
>>>>>> "usb: gadget: f_uac2/u_audio: add feedback endpoint support"
>>>>>>
>>>>>> Ruslan's 3 patches are related to each other so I reverted all 
>>>>>> three 24f779da...e89bb428 and applied the reverts to rc1.
>>>>>>
>>>>>> I can confirm this indeed resolves the problem in rc1.
>>>>>>
>>>>>> Is late now, tomorrow evening I will apply the reverts to rc6.
>>>>>
>>>>> With these reverts rc6 works fine as well.
>>>>>
>>>>> So, where do we go from here?
>>>>>
>>>>
>>>> I know the patches have been tested on dwc2 (by me and others).  I 
>>>> do not know if Ruslan or Jerome tested them on dwc3 but probably 
>>>> not. Ruslan has talked about RPi (my case too) and BeagleboneBlack, 
>>>> both with dwc2. Perhaps the dwc2 behaves a bit differently than dwc3?
>>>>
>>>> The patches add a new EP-IN for async feedback. I am sorry I have 
>>>> not followed your long thread (it started as unrelated to uac). Does 
>>>> the problem appear with f_uac1 or f_uac2? Please how have you 
>>>> reached the above problem?
>>>
>>> I'm sorry too. I first believed the issue was related to the patch 
>>> mentioned in the subject line.
>>>
>>> The problem appaers with f_uac2. I bost Edison_Arduino board in host 
>>> mode (there is a switch allowing to select host/device mode). When 
>>> flipping the switch to device mode udev run a script:
>>> But as I am using configfs (excerpt follows) and just disabling the 
>>> last 2 line resolves the issue, I'm guessing uac2 is the issue. Or 
>>> exceeding the available resources.
>>>
>>> # Create directory structure
>>> mkdir "${GADGET_BASE_DIR}"
>>> cd "${GADGET_BASE_DIR}"
>>> mkdir -p configs/c.1/strings/0x409
>>> mkdir -p strings/0x409
>>>
>>> # Serial device
>>> mkdir functions/gser.usb0
>>> ln -s functions/gser.usb0 configs/c.1/
>>> ###
>>>
>>> # Ethernet device
>>> mkdir functions/eem.usb0
>>> echo "${DEV_ETH_ADDR}" > functions/eem.usb0/dev_addr
>>> echo "${HOST_ETH_ADDR}" > functions/eem.usb0/host_addr
>>> ln -s functions/eem.usb0 configs/c.1/
>>>
>>> # Mass Storage device
>>> mkdir functions/mass_storage.usb0
>>> echo 1 > functions/mass_storage.usb0/stall
>>> echo 0 > functions/mass_storage.usb0/lun.0/cdrom
>>> echo 0 > functions/mass_storage.usb0/lun.0/ro
>>> echo 0 > functions/mass_storage.usb0/lun.0/nofua
>>> echo "${USBDISK}" > functions/mass_storage.usb0/lun.0/file
>>> ln -s functions/mass_storage.usb0 configs/c.1/
>>>
>>> # UAC2 device
>>> mkdir functions/uac2.usb0
>>> ln -s functions/uac2.usb0 configs/c.1
>>> ....
>>>
>>
>> As you say, could perhaps the reason be that the extra EP-IN added in 
>> those patches (previously 1, now 2 with the default config you use) 
>> exceeds your EP-IN max count or available fifos somehow?  You have a 
>> number of functions initialized. If you change the load order of the 
>> functions, do you get the error later with a different function? Just 
>> guessing...
>>
>> You should be able to switch the default async EP-OUT (which 
>> configures the new feedback EP-IN ) to adaptive EP-OUT (which requires 
>> no feedback EP) with c_sync=8 parameter of f_uac2.
>>
>> https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c#L47 
>>
>>
>> https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c#L830 
>>
>>
>> https://elixir.bootlin.com/linux/v5.14-rc6/source/include/uapi/linux/usb/ch9.h#L453 
>>
>>
>> Does that fix the problem?
> 
> Not sure how to do that. Do you mean the module should have a parameter 
> called c_sync? `modinfo` list no parameters for usb_f_uac2.
> 

Those are configfs params, not available in modinfo.

I checked and the value is "adaptive" 
https://elixir.bootlin.com/linux/v5.14-rc7/source/drivers/usb/gadget/function/f_uac2.c#L1312

In your configfs script:

# UAC2 device
mkdir functions/uac2.usb0
ln -s functions/uac2.usb0 configs/c.1


On the USB Host:
cat /proc/asound/Gadget/stream:
Playback:
   Status: Stop
   Interface 1
     Altset 1
     Format: S16_LE
     Channels: 2
     Endpoint: 0x01 (1 OUT) (ASYNC)
     Rates: 64000
     Data packet interval: 125 us
     Bits: 16
     Channel map: FL FR
     Sync Endpoint: 0x82 (2 IN)
     Sync EP Interface: 1
     Sync EP Altset: 1
     Implicit Feedback Mode: No


lsusb -v -d 1d6b:0104 | | grep EP.*IN
         bEndpointAddress     0x81  EP 1 IN
         bEndpointAddress     0x82  EP 2 IN
         bEndpointAddress     0x83  EP 3 IN

I have additional patches applied which define controls via EP IN 
interrupt mode, not part of that patchset.


Switching to the adaptive mode:
# UAC2 device
mkdir functions/uac2.usb0
echo "adaptive" > functions/uac2.usb0/c_sync
ln -s functions/uac2.usb0 configs/c.1

On the USB Host:
cat /proc/asound/Gadget/stream:
Playback:
   Status: Stop
   Interface 1
     Altset 1
     Format: S16_LE
     Channels: 2
     Endpoint: 0x01 (1 OUT) (ADAPTIVE)
     Rates: 64000
     Data packet interval: 125 us
     Bits: 16
     Channel map: FL FR


lsusb -v -d 1d6b:0104 | grep EP.*IN
         bEndpointAddress     0x81  EP 1 IN
         bEndpointAddress     0x82  EP 2 IN


The feedback EP-IN is gone because the mode is adaptive now.

If you think the new input endpoints should not be created by default 
for resource-compatibility reasons, the adaptive mode can be set by 
default in a fixed patch.

Also the patches defining the new interrupt endpoints can have the 
controls disabled by default (and not allocate the EP-IN). These patches 
on top of the async patches are already accepted by Greg for usb-next 
branch 
https://kernel.googlesource.com/pub/scm/linux/kernel/git/gregkh/usb/+/eaf6cbe0992052a46d93047dc122fad5126aa3bd 
. That's why I was trying to sort out the async patches without 
reverting them as it will call for more reverts.

Thanks,

Pavel.

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-08-23 14:59                                                 ` Pavel Hofman
@ 2021-08-23 15:21                                                   ` Andy Shevchenko
  2021-08-23 15:34                                                     ` Pavel Hofman
  0 siblings, 1 reply; 73+ messages in thread
From: Andy Shevchenko @ 2021-08-23 15:21 UTC (permalink / raw)
  To: Pavel Hofman
  Cc: Ferry Toth, Felipe Balbi, Wesley Cheng, gregkh, robh+dt, agross,
	bjorn.andersson, frowand.list, linux-usb, linux-kernel,
	devicetree, linux-arm-msm, jackp, heikki.krogerus,
	Ruslan Bilovol

On Mon, Aug 23, 2021 at 5:59 PM Pavel Hofman <pavel.hofman@ivitera.com> wrote:
> Dne 22. 08. 21 v 21:43 Ferry Toth napsal(a):
> > Op 19-08-2021 om 23:04 schreef Pavel Hofman:
> >> Dne 19. 08. 21 v 22:10 Ferry Toth napsal(a):
> >>> Op 19-08-2021 om 09:51 schreef Pavel Hofman:
> >>>> Dne 18. 08. 21 v 21:07 Ferry Toth napsal(a):
> >>>>> Op 18-08-2021 om 00:00 schreef Ferry Toth:

...

> >>>>> So, where do we go from here?
> >>>>
> >>>> I know the patches have been tested on dwc2 (by me and others).  I
> >>>> do not know if Ruslan or Jerome tested them on dwc3 but probably
> >>>> not. Ruslan has talked about RPi (my case too) and BeagleboneBlack,
> >>>> both with dwc2. Perhaps the dwc2 behaves a bit differently than dwc3?
> >>>>
> >>>> The patches add a new EP-IN for async feedback. I am sorry I have
> >>>> not followed your long thread (it started as unrelated to uac). Does
> >>>> the problem appear with f_uac1 or f_uac2? Please how have you
> >>>> reached the above problem?
> >>>
> >>> I'm sorry too. I first believed the issue was related to the patch
> >>> mentioned in the subject line.
> >>>
> >>> The problem appaers with f_uac2. I bost Edison_Arduino board in host
> >>> mode (there is a switch allowing to select host/device mode). When
> >>> flipping the switch to device mode udev run a script:
> >>> But as I am using configfs (excerpt follows) and just disabling the
> >>> last 2 line resolves the issue, I'm guessing uac2 is the issue. Or
> >>> exceeding the available resources.
> >>>
> >>> # Create directory structure
> >>> mkdir "${GADGET_BASE_DIR}"
> >>> cd "${GADGET_BASE_DIR}"
> >>> mkdir -p configs/c.1/strings/0x409
> >>> mkdir -p strings/0x409
> >>>
> >>> # Serial device
> >>> mkdir functions/gser.usb0
> >>> ln -s functions/gser.usb0 configs/c.1/
> >>> ###
> >>>
> >>> # Ethernet device
> >>> mkdir functions/eem.usb0
> >>> echo "${DEV_ETH_ADDR}" > functions/eem.usb0/dev_addr
> >>> echo "${HOST_ETH_ADDR}" > functions/eem.usb0/host_addr
> >>> ln -s functions/eem.usb0 configs/c.1/
> >>>
> >>> # Mass Storage device
> >>> mkdir functions/mass_storage.usb0
> >>> echo 1 > functions/mass_storage.usb0/stall
> >>> echo 0 > functions/mass_storage.usb0/lun.0/cdrom
> >>> echo 0 > functions/mass_storage.usb0/lun.0/ro
> >>> echo 0 > functions/mass_storage.usb0/lun.0/nofua
> >>> echo "${USBDISK}" > functions/mass_storage.usb0/lun.0/file
> >>> ln -s functions/mass_storage.usb0 configs/c.1/
> >>>
> >>> # UAC2 device
> >>> mkdir functions/uac2.usb0
> >>> ln -s functions/uac2.usb0 configs/c.1
> >>> ....
> >>
> >> As you say, could perhaps the reason be that the extra EP-IN added in
> >> those patches (previously 1, now 2 with the default config you use)
> >> exceeds your EP-IN max count or available fifos somehow?  You have a
> >> number of functions initialized. If you change the load order of the
> >> functions, do you get the error later with a different function? Just
> >> guessing...
> >>
> >> You should be able to switch the default async EP-OUT (which
> >> configures the new feedback EP-IN ) to adaptive EP-OUT (which requires
> >> no feedback EP) with c_sync=8 parameter of f_uac2.
> >>
> >> https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c#L47
> >>
> >> https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c#L830
> >>
> >> https://elixir.bootlin.com/linux/v5.14-rc6/source/include/uapi/linux/usb/ch9.h#L453
> >>
> >> Does that fix the problem?
> >
> > Not sure how to do that. Do you mean the module should have a parameter
> > called c_sync? `modinfo` list no parameters for usb_f_uac2.
>
> Those are configfs params, not available in modinfo.
>
> I checked and the value is "adaptive"
> https://elixir.bootlin.com/linux/v5.14-rc7/source/drivers/usb/gadget/function/f_uac2.c#L1312

> In your configfs script:

Kernel shouldn't crash with any available set of configuration
parameters, right? So, even if the parameter would fix the crash the
series is buggy and has to be reverted in my opinion.

> # UAC2 device
> mkdir functions/uac2.usb0
> ln -s functions/uac2.usb0 configs/c.1
>
>
> On the USB Host:
> cat /proc/asound/Gadget/stream:
> Playback:
>    Status: Stop
>    Interface 1
>      Altset 1
>      Format: S16_LE
>      Channels: 2
>      Endpoint: 0x01 (1 OUT) (ASYNC)
>      Rates: 64000
>      Data packet interval: 125 us
>      Bits: 16
>      Channel map: FL FR
>      Sync Endpoint: 0x82 (2 IN)
>      Sync EP Interface: 1
>      Sync EP Altset: 1
>      Implicit Feedback Mode: No
>
> lsusb -v -d 1d6b:0104 | | grep EP.*IN
>          bEndpointAddress     0x81  EP 1 IN
>          bEndpointAddress     0x82  EP 2 IN
>          bEndpointAddress     0x83  EP 3 IN
>
> I have additional patches applied which define controls via EP IN
> interrupt mode, not part of that patchset.
>
> Switching to the adaptive mode:
> # UAC2 device
> mkdir functions/uac2.usb0
> echo "adaptive" > functions/uac2.usb0/c_sync
> ln -s functions/uac2.usb0 configs/c.1
>
> On the USB Host:
> cat /proc/asound/Gadget/stream:
> Playback:
>    Status: Stop
>    Interface 1
>      Altset 1
>      Format: S16_LE
>      Channels: 2
>      Endpoint: 0x01 (1 OUT) (ADAPTIVE)
>      Rates: 64000
>      Data packet interval: 125 us
>      Bits: 16
>      Channel map: FL FR
>
> lsusb -v -d 1d6b:0104 | grep EP.*IN
>          bEndpointAddress     0x81  EP 1 IN
>          bEndpointAddress     0x82  EP 2 IN
>
> The feedback EP-IN is gone because the mode is adaptive now.
>
> If you think the new input endpoints should not be created by default
> for resource-compatibility reasons, the adaptive mode can be set by
> default in a fixed patch.

Would it be possible to change the mode? If so, then the user may
configure it and crash again.

> Also the patches defining the new interrupt endpoints can have the
> controls disabled by default (and not allocate the EP-IN). These patches
> on top of the async patches are already accepted by Greg for usb-next
> branch
> https://kernel.googlesource.com/pub/scm/linux/kernel/git/gregkh/usb/+/eaf6cbe0992052a46d93047dc122fad5126aa3bd
> . That's why I was trying to sort out the async patches without
> reverting them as it will call for more reverts.

-- 
With Best Regards,
Andy Shevchenko

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-08-23 15:21                                                   ` Andy Shevchenko
@ 2021-08-23 15:34                                                     ` Pavel Hofman
  2021-08-23 18:54                                                       ` Ferry Toth
  2021-08-23 22:50                                                       ` Thinh Nguyen
  0 siblings, 2 replies; 73+ messages in thread
From: Pavel Hofman @ 2021-08-23 15:34 UTC (permalink / raw)
  To: Andy Shevchenko
  Cc: Ferry Toth, Felipe Balbi, Wesley Cheng, gregkh, robh+dt, agross,
	bjorn.andersson, frowand.list, linux-usb, linux-kernel,
	devicetree, linux-arm-msm, jackp, heikki.krogerus,
	Ruslan Bilovol



Dne 23. 08. 21 v 17:21 Andy Shevchenko napsal(a):
> On Mon, Aug 23, 2021 at 5:59 PM Pavel Hofman <pavel.hofman@ivitera.com> wrote:
>> Dne 22. 08. 21 v 21:43 Ferry Toth napsal(a):
>>> Op 19-08-2021 om 23:04 schreef Pavel Hofman:
>>>> Dne 19. 08. 21 v 22:10 Ferry Toth napsal(a):
>>>>> Op 19-08-2021 om 09:51 schreef Pavel Hofman:
>>>>>> Dne 18. 08. 21 v 21:07 Ferry Toth napsal(a):
>>>>>>> Op 18-08-2021 om 00:00 schreef Ferry Toth:
> 
> ...
> 
>>>>>>> So, where do we go from here?
>>>>>>
>>>>>> I know the patches have been tested on dwc2 (by me and others).  I
>>>>>> do not know if Ruslan or Jerome tested them on dwc3 but probably
>>>>>> not. Ruslan has talked about RPi (my case too) and BeagleboneBlack,
>>>>>> both with dwc2. Perhaps the dwc2 behaves a bit differently than dwc3?
>>>>>>
>>>>>> The patches add a new EP-IN for async feedback. I am sorry I have
>>>>>> not followed your long thread (it started as unrelated to uac). Does
>>>>>> the problem appear with f_uac1 or f_uac2? Please how have you
>>>>>> reached the above problem?
>>>>>
>>>>> I'm sorry too. I first believed the issue was related to the patch
>>>>> mentioned in the subject line.
>>>>>
>>>>> The problem appaers with f_uac2. I bost Edison_Arduino board in host
>>>>> mode (there is a switch allowing to select host/device mode). When
>>>>> flipping the switch to device mode udev run a script:
>>>>> But as I am using configfs (excerpt follows) and just disabling the
>>>>> last 2 line resolves the issue, I'm guessing uac2 is the issue. Or
>>>>> exceeding the available resources.
>>>>>
>>>>> # Create directory structure
>>>>> mkdir "${GADGET_BASE_DIR}"
>>>>> cd "${GADGET_BASE_DIR}"
>>>>> mkdir -p configs/c.1/strings/0x409
>>>>> mkdir -p strings/0x409
>>>>>
>>>>> # Serial device
>>>>> mkdir functions/gser.usb0
>>>>> ln -s functions/gser.usb0 configs/c.1/
>>>>> ###
>>>>>
>>>>> # Ethernet device
>>>>> mkdir functions/eem.usb0
>>>>> echo "${DEV_ETH_ADDR}" > functions/eem.usb0/dev_addr
>>>>> echo "${HOST_ETH_ADDR}" > functions/eem.usb0/host_addr
>>>>> ln -s functions/eem.usb0 configs/c.1/
>>>>>
>>>>> # Mass Storage device
>>>>> mkdir functions/mass_storage.usb0
>>>>> echo 1 > functions/mass_storage.usb0/stall
>>>>> echo 0 > functions/mass_storage.usb0/lun.0/cdrom
>>>>> echo 0 > functions/mass_storage.usb0/lun.0/ro
>>>>> echo 0 > functions/mass_storage.usb0/lun.0/nofua
>>>>> echo "${USBDISK}" > functions/mass_storage.usb0/lun.0/file
>>>>> ln -s functions/mass_storage.usb0 configs/c.1/
>>>>>
>>>>> # UAC2 device
>>>>> mkdir functions/uac2.usb0
>>>>> ln -s functions/uac2.usb0 configs/c.1
>>>>> ....
>>>>
>>>> As you say, could perhaps the reason be that the extra EP-IN added in
>>>> those patches (previously 1, now 2 with the default config you use)
>>>> exceeds your EP-IN max count or available fifos somehow?  You have a
>>>> number of functions initialized. If you change the load order of the
>>>> functions, do you get the error later with a different function? Just
>>>> guessing...
>>>>
>>>> You should be able to switch the default async EP-OUT (which
>>>> configures the new feedback EP-IN ) to adaptive EP-OUT (which requires
>>>> no feedback EP) with c_sync=8 parameter of f_uac2.
>>>>
>>>> https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c#L47
>>>>
>>>> https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c#L830
>>>>
>>>> https://elixir.bootlin.com/linux/v5.14-rc6/source/include/uapi/linux/usb/ch9.h#L453
>>>>
>>>> Does that fix the problem?
>>>
>>> Not sure how to do that. Do you mean the module should have a parameter
>>> called c_sync? `modinfo` list no parameters for usb_f_uac2.
>>
>> Those are configfs params, not available in modinfo.
>>
>> I checked and the value is "adaptive"
>> https://elixir.bootlin.com/linux/v5.14-rc7/source/drivers/usb/gadget/function/f_uac2.c#L1312
> 
>> In your configfs script:
> 
> Kernel shouldn't crash with any available set of configuration
> parameters, right? So, even if the parameter would fix the crash the
> series is buggy and has to be reverted in my opinion.

Sure, no problem with reverting. I am just trying to start up some 
troubleshooting. A resource exhaustion was mentioned here, that's why I 
suggested a way to test the patch with the original number of endpoints 
allocated. I do not get any crashes on my setup which uses fewer gadget 
functions. That's why I asked what happens if the functions load order 
is changed. I am afraid this thread is so complex that the actual 
problem has been burried in the history.

Again, I am not the author of the patches, my USB knowledge is way 
unsufficient for that. I am trying to make them work as they are 
important and make the existing audio gadget actually usable.

Thanks,

Pavel.

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-08-23 15:34                                                     ` Pavel Hofman
@ 2021-08-23 18:54                                                       ` Ferry Toth
  2021-08-23 22:50                                                       ` Thinh Nguyen
  1 sibling, 0 replies; 73+ messages in thread
From: Ferry Toth @ 2021-08-23 18:54 UTC (permalink / raw)
  To: Pavel Hofman, Andy Shevchenko
  Cc: Felipe Balbi, Wesley Cheng, gregkh, robh+dt, agross,
	bjorn.andersson, frowand.list, linux-usb, linux-kernel,
	devicetree, linux-arm-msm, jackp, heikki.krogerus,
	Ruslan Bilovol

Op 23-08-2021 om 17:34 schreef Pavel Hofman:
> 
> 
> Dne 23. 08. 21 v 17:21 Andy Shevchenko napsal(a):
>> On Mon, Aug 23, 2021 at 5:59 PM Pavel Hofman 
>> <pavel.hofman@ivitera.com> wrote:
>>> Dne 22. 08. 21 v 21:43 Ferry Toth napsal(a):
>>>> Op 19-08-2021 om 23:04 schreef Pavel Hofman:
>>>>> Dne 19. 08. 21 v 22:10 Ferry Toth napsal(a):
>>>>>> Op 19-08-2021 om 09:51 schreef Pavel Hofman:
>>>>>>> Dne 18. 08. 21 v 21:07 Ferry Toth napsal(a):
>>>>>>>> Op 18-08-2021 om 00:00 schreef Ferry Toth:
>>
>> ...
>>
>>>>>>>> So, where do we go from here?
>>>>>>>
>>>>>>> I know the patches have been tested on dwc2 (by me and others).  I
>>>>>>> do not know if Ruslan or Jerome tested them on dwc3 but probably
>>>>>>> not. Ruslan has talked about RPi (my case too) and BeagleboneBlack,
>>>>>>> both with dwc2. Perhaps the dwc2 behaves a bit differently than 
>>>>>>> dwc3?
>>>>>>>
>>>>>>> The patches add a new EP-IN for async feedback. I am sorry I have
>>>>>>> not followed your long thread (it started as unrelated to uac). Does
>>>>>>> the problem appear with f_uac1 or f_uac2? Please how have you
>>>>>>> reached the above problem?
>>>>>>
>>>>>> I'm sorry too. I first believed the issue was related to the patch
>>>>>> mentioned in the subject line.
>>>>>>
>>>>>> The problem appaers with f_uac2. I bost Edison_Arduino board in host
>>>>>> mode (there is a switch allowing to select host/device mode). When
>>>>>> flipping the switch to device mode udev run a script:
>>>>>> But as I am using configfs (excerpt follows) and just disabling the
>>>>>> last 2 line resolves the issue, I'm guessing uac2 is the issue. Or
>>>>>> exceeding the available resources.
>>>>>>
>>>>>> # Create directory structure
>>>>>> mkdir "${GADGET_BASE_DIR}"
>>>>>> cd "${GADGET_BASE_DIR}"
>>>>>> mkdir -p configs/c.1/strings/0x409
>>>>>> mkdir -p strings/0x409
>>>>>>
>>>>>> # Serial device
>>>>>> mkdir functions/gser.usb0
>>>>>> ln -s functions/gser.usb0 configs/c.1/
>>>>>> ###
>>>>>>
>>>>>> # Ethernet device
>>>>>> mkdir functions/eem.usb0
>>>>>> echo "${DEV_ETH_ADDR}" > functions/eem.usb0/dev_addr
>>>>>> echo "${HOST_ETH_ADDR}" > functions/eem.usb0/host_addr
>>>>>> ln -s functions/eem.usb0 configs/c.1/
>>>>>>
>>>>>> # Mass Storage device
>>>>>> mkdir functions/mass_storage.usb0
>>>>>> echo 1 > functions/mass_storage.usb0/stall
>>>>>> echo 0 > functions/mass_storage.usb0/lun.0/cdrom
>>>>>> echo 0 > functions/mass_storage.usb0/lun.0/ro
>>>>>> echo 0 > functions/mass_storage.usb0/lun.0/nofua
>>>>>> echo "${USBDISK}" > functions/mass_storage.usb0/lun.0/file
>>>>>> ln -s functions/mass_storage.usb0 configs/c.1/
>>>>>>
>>>>>> # UAC2 device
>>>>>> mkdir functions/uac2.usb0
>>>>>> ln -s functions/uac2.usb0 configs/c.1
>>>>>> ....
>>>>>
>>>>> As you say, could perhaps the reason be that the extra EP-IN added in
>>>>> those patches (previously 1, now 2 with the default config you use)
>>>>> exceeds your EP-IN max count or available fifos somehow?  You have a
>>>>> number of functions initialized. If you change the load order of the
>>>>> functions, do you get the error later with a different function? Just
>>>>> guessing...
>>>>>
>>>>> You should be able to switch the default async EP-OUT (which
>>>>> configures the new feedback EP-IN ) to adaptive EP-OUT (which requires
>>>>> no feedback EP) with c_sync=8 parameter of f_uac2.
>>>>>
>>>>> https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c#L47 
>>>>>
>>>>>
>>>>> https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c#L830 
>>>>>
>>>>>
>>>>> https://elixir.bootlin.com/linux/v5.14-rc6/source/include/uapi/linux/usb/ch9.h#L453 
>>>>>
>>>>>
>>>>> Does that fix the problem?
>>>>
>>>> Not sure how to do that. Do you mean the module should have a parameter
>>>> called c_sync? `modinfo` list no parameters for usb_f_uac2.
>>>
>>> Those are configfs params, not available in modinfo.
>>>
>>> I checked and the value is "adaptive"
>>> https://elixir.bootlin.com/linux/v5.14-rc7/source/drivers/usb/gadget/function/f_uac2.c#L1312 
>>>
>>
>>> In your configfs script:
>>
>> Kernel shouldn't crash with any available set of configuration
>> parameters, right? So, even if the parameter would fix the crash the
>> series is buggy and has to be reverted in my opinion.

I will send in reverts tomorrow, unless someone has a better idea.

> Sure, no problem with reverting. I am just trying to start up some 
> troubleshooting. A resource exhaustion was mentioned here, that's why I 
> suggested a way to test the patch with the original number of endpoints 
> allocated. I do not get any crashes on my setup which uses fewer gadget 
> functions. That's why I asked what happens if the functions load order 
> is changed. I am afraid this thread is so complex that the actual 
> problem has been burried in the history.

I found some time to test your suggestions. So below 3 cases on rc6 with 
the reverts removed again:
1) the original configfs setup script
2) with uac2 moved to the top so it is created first (I hope)
3) same as 1) but with c_sync = adaptive

Summary:
1) still shows the crash as expected
2) has no effect
3) resolves the issue as you expect

1) Normal configfs order (uac2 last)
------------------------------------

root@yuna:~# [   50.975986] BUG: kernel NULL pointer dereference, 
address: 0000000000000008
[   50.983125] #PF: supervisor write access in kernel mode
[   50.988470] #PF: error_code(0x0002) - not-present page
[   50.993723] PGD 0 P4D 0
[   50.996327] Oops: 0002 [#1] SMP PTI
[   50.999899] CPU: 0 PID: 596 Comm: irq/14-dwc3 Tainted: G        W 
      5.14.0-rc6-edison-acpi-standard #1
[   51.009948] Hardware name: Intel Corporation Merrifield/BODEGA BAY, 
BIOS 542 2015.01.21:18.19.48
[   51.018922] RIP: 0010:dwc3_gadget_del_and_unmap_request+0x19/0xe0
[   51.025169] Code: 8b 78 08 e8 89 d6 ff ff 48 89 ef 5d e9 f0 61 a6 ff 
41 54 55 48 89 f5 53 48 8b 46 68 48 89 fb 48 8b 4e 60 4c 8b a7 90 00 00 
00 <48> 89 41 08 48 89 08 48 b8 00 01 00 00
00 00 ad de 80 a6 b4 00 00
[   51.044345] RSP: 0000:ffffb2ea40313c10 EFLAGS: 00010093
[   51.049692] RAX: ffff94dc03aa2c48 RBX: ffff94dc03aa2c00 RCX: 
0000000000000000
[   51.056985] RDX: 00000000ffffff94 RSI: ffff94dc02303d80 RDI: 
ffff94dc03aa2c00
[   51.064278] RBP: ffff94dc02303d80 R08: 0000000000000014 R09: 
0000000000000001
[   51.071572] R10: ffff94dc02303d80 R11: 0000000000000001 R12: 
ffff94dc08b43028
[   51.078864] R13: ffff94dc03aa2c48 R14: 0000000000000000 R15: 
ffff94dc031b3c00
[   51.086159] FS:  0000000000000000(0000) GS:ffff94dc3e200000(0000) 
knlGS:0000000000000000
[   51.094429] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   51.100304] CR2: 0000000000000008 CR3: 0000000007e20000 CR4: 
00000000001006f0
[   51.107596] Call Trace:
[   51.110103]  dwc3_remove_requests.constprop.0+0x12f/0x170
[   51.115637]  __dwc3_gadget_ep_disable+0x7a/0x160
[   51.120366]  dwc3_gadget_ep_disable+0x3d/0xd0
[   51.124827]  usb_ep_disable+0x1c/0x70
[   51.128586]  u_audio_stop_capture+0x79/0x120 [u_audio]
[   51.133857]  afunc_set_alt+0x73/0x80 [usb_f_uac2]
[   51.138683]  composite_setup+0x224/0x1b90 [libcomposite]
[   51.144137]  ? __dwc3_gadget_kick_transfer+0x160/0x400
[   51.149404]  ? configfs_composite_setup+0x6b/0x90 [libcomposite]
[   51.155562]  configfs_composite_setup+0x6b/0x90 [libcomposite]
[   51.161544]  dwc3_ep0_interrupt+0x466/0xa70
[   51.165831]  dwc3_thread_interrupt+0x829/0xe90
[   51.170379]  ? __schedule+0x235/0x6c0
[   51.174138]  ? disable_irq_nosync+0x10/0x10
[   51.178425]  irq_thread_fn+0x1b/0x60
[   51.182092]  irq_thread+0xc0/0x160
[   51.185580]  ? irq_thread_check_affinity+0x70/0x70
[   51.190486]  ? irq_forced_thread_fn+0x70/0x70
[   51.194948]  kthread+0x122/0x140
[   51.198259]  ? set_kthread_struct+0x40/0x40
[   51.202545]  ret_from_fork+0x22/0x30
[   51.206214] Modules linked in: usb_f_uac2 u_audio usb_f_mass_storage 
usb_f_eem u_ether usb_f_serial u_serial libcomposite rfcomm iptable_nat 
bnep spi_pxa2xx_platform snd_sof_nocodec dw_dmac smsc smsc95xx 
intel_mrfld_pwrbtn intel_mrfld_adc dw_dmac_pci pwm_lpss_pci dw_dmac_core 
pwm_lpss snd_sof_pci_intel_tng snd_sof_pci snd_sof_intel_ipc 
snd_sof_intel_atom snd_sof snd_sof_xtensa_dsp snd_soc_acpi 
spi_pxa2xx_pci brcmfmac brcmutil hci_uart btbcm ti_ads7950 
industrialio_triggered_buffer kfifo_buf leds_gpio ledtrig_timer 
ledtrig_heartbeat mmc_block extcon_intel_mrfld sdhci_pci cqhci sdhci 
led_class mmc_core intel_soc_pmic_mrfld btrfs libcrc32c xor 
zstd_compress zlib_deflate raid6_pq
[   51.267592] CR2: 0000000000000008
[   51.270981] ---[ end trace 135abe76b19414cc ]---

ferry@delfion:~$ cat /proc/asound/Gadget/stream0
USBArmory USBArmory Gadget at usb-0000:00:14.0-5, high speed : USB Audio

Playback:
   Status: Stop
   Interface 4
     Altset 1
     Format: S16_LE
     Channels: 2
     Endpoint: 0x04 (4 OUT) (ASYNC)
     Rates: 64000
     Data packet interval: 1000 us
     Bits: 16
     Channel map: FL FR
     Sync Endpoint: 0x85 (5 IN)
     Sync EP Interface: 4
     Sync EP Altset: 1
     Implicit Feedback Mode: No

Capture:
   Status: Stop
   Interface 5
     Altset 1
     Format: S16_LE
     Channels: 2
     Endpoint: 0x86 (6 IN) (ASYNC)
     Rates: 48000
     Data packet interval: 1000 us
     Bits: 16
     Channel map: FL FR

root@delfion:~# lsusb -v -d 1d6b:0104 | grep EP.*IN
         bEndpointAddress     0x82  EP 2 IN
         bEndpointAddress     0x83  EP 3 IN
         bEndpointAddress     0x84  EP 4 IN
         bEndpointAddress     0x85  EP 5 IN
         bEndpointAddress     0x86  EP 6 IN
can't get debug descriptor: Resource temporarily unavailable
cannot read device status, Resource temporarily unavailable (11)

2) uac2 first in configfs
-------------------------

root@yuna:~# [  172.094997] BUG: kernel NULL pointer dereference, 
address: 0000000000000008
[  172.102136] #PF: supervisor write access in kernel mode
[  172.107480] #PF: error_code(0x0002) - not-present page
[  172.112733] PGD 0 P4D 0
[  172.115338] Oops: 0002 [#1] SMP PTI
[  172.118909] CPU: 0 PID: 604 Comm: irq/14-dwc3 Tainted: G        W 
      5.14.0-rc6-edison-acpi-standard #1
[  172.128958] Hardware name: Intel Corporation Merrifield/BODEGA BAY, 
BIOS 542 2015.01.21:18.19.48
[  172.137931] RIP: 0010:dwc3_gadget_del_and_unmap_request+0x19/0xe0
[  172.144175] Code: 8b 78 08 e8 89 d6 ff ff 48 89 ef 5d e9 f0 61 a6 ff 
41 54 55 48 89 f5 53 48 8b 46 68 48 89 fb 48 8b 4e 60 4c 8b a7 90 00 00 
00 <48> 89 41 08 48 89 08 48 b8 00 01 00 00
00 00 ad de 80 a6 b4 00 00
[  172.163352] RSP: 0018:ffffa1438053bc10 EFLAGS: 00010097
[  172.168698] RAX: ffff9b76c7ec3848 RBX: ffff9b76c7ec3800 RCX: 
0000000000000000
[  172.175990] RDX: 00000000ffffff94 RSI: ffff9b76c79ffc00 RDI: 
ffff9b76c7ec3800
[  172.183283] RBP: ffff9b76c79ffc00 R08: 0000000000000014 R09: 
0000000000000001
[  172.190577] R10: ffff9b76c79ffc00 R11: 0000000000000002 R12: 
ffff9b76c8a66028
[  172.197870] R13: ffff9b76c7ec3848 R14: 0000000000000000 R15: 
ffff9b76c7255c00
[  172.205163] FS:  0000000000000000(0000) GS:ffff9b76fe200000(0000) 
knlGS:0000000000000000
[  172.213436] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  172.219309] CR2: 0000000000000008 CR3: 000000001360c000 CR4: 
00000000001006f0
[  172.226600] Call Trace:
[  172.229107]  dwc3_remove_requests.constprop.0+0x12f/0x170
[  172.234640]  __dwc3_gadget_ep_disable+0x7a/0x160
[  172.239372]  dwc3_gadget_ep_disable+0x3d/0xd0
[  172.243834]  usb_ep_disable+0x1c/0x70
[  172.247590]  u_audio_stop_capture+0x79/0x120 [u_audio]
[  172.252862]  afunc_set_alt+0x73/0x80 [usb_f_uac2]
[  172.257688]  composite_setup+0x224/0x1b90 [libcomposite]
[  172.263143]  ? __dwc3_gadget_kick_transfer+0x160/0x400
[  172.268410]  ? configfs_composite_setup+0x6b/0x90 [libcomposite]
[  172.274568]  configfs_composite_setup+0x6b/0x90 [libcomposite]
[  172.280551]  dwc3_ep0_interrupt+0x466/0xa70
[  172.284834]  ? newidle_balance.constprop.0+0xfa/0x380
[  172.290008]  dwc3_thread_interrupt+0x829/0xe90
[  172.294561]  ? __schedule+0x235/0x6c0
[  172.298316]  ? disable_irq_nosync+0x10/0x10
[  172.302606]  irq_thread_fn+0x1b/0x60
[  172.306272]  irq_thread+0xc0/0x160
[  172.309761]  ? irq_thread_check_affinity+0x70/0x70
[  172.314666]  ? irq_forced_thread_fn+0x70/0x70
[  172.319128]  kthread+0x122/0x140
[  172.322440]  ? set_kthread_struct+0x40/0x40
[  172.326724]  ret_from_fork+0x22/0x30
[  172.330390] Modules linked in: usb_f_mass_storage usb_f_eem u_ether 
usb_f_serial u_serial usb_f_uac2 u_audio libcomposite rfcomm iptable_nat 
bnep snd_sof_nocodec spi_pxa2xx_platform dw_dmac smsc smsc95xx 
pwm_lpss_pci pwm_lpss snd_sof_pci_intel_tng snd_sof_pci 
snd_sof_intel_ipc intel_mrfld_pwrbtn snd_sof_intel_atom intel_mrfld_adc 
dw_dmac_pci dw_dmac_core snd_sof snd_sof_xtensa_dsp snd_soc_acpi 
spi_pxa2xx_pci brcmfmac brcmutil hci_uart leds_gpio btbcm ti_ads7950 
industrialio_triggered_buffer kfifo_buf ledtrig_timer ledtrig_heartbeat 
mmc_block extcon_intel_mrfld sdhci_pci cqhci sdhci led_class mmc_core 
intel_soc_pmic_mrfld btrfs libcrc32c xor zstd_compress zlib_deflate raid6_pq
[  172.391766] CR2: 0000000000000008
[  172.395157] ---[ end trace 8272319264d5a2c0 ]---

3) With c_sync = adaptive
-------------------------
# UAC2 device
mkdir functions/uac2.usb0
echo "adaptive" > functions/uac2.usb0/c_sync
ln -s functions/uac2.usb0 configs/c.1

(No BUG)

root@delfion:~# cat /proc/asound/Gadget/stream0
USBArmory USBArmory Gadget at usb-0000:00:14.0-5, high speed : USB Audio

Playback:
   Status: Stop
   Interface 4
     Altset 1
     Format: S16_LE
     Channels: 2
     Endpoint: 0x04 (4 OUT) (ADAPTIVE)
     Rates: 64000
     Data packet interval: 1000 us
     Bits: 16
     Channel map: FL FR

Capture:
   Status: Stop
   Interface 5
     Altset 1
     Format: S16_LE
     Channels: 2
     Endpoint: 0x85 (5 IN) (ASYNC)
     Rates: 48000
     Data packet interval: 1000 us
     Bits: 16
     Channel map: FL FR


root@delfion:~# lsusb -v -d 1d6b:0104 |  grep EP.*IN
         bEndpointAddress     0x82  EP 2 IN
         bEndpointAddress     0x83  EP 3 IN
         bEndpointAddress     0x84  EP 4 IN
         bEndpointAddress     0x85  EP 5 IN
can't get debug descriptor: Resource temporarily unavailable

> Again, I am not the author of the patches, my USB knowledge is way 
> unsufficient for that. I am trying to make them work as they are 
> important and make the existing audio gadget actually usable.

Thanks for the suggestions, I didn't think to set params through configfs.

Maybe this will help in tracing the root cause, although that's way over 
my head.

> Thanks,
> 
> Pavel.


^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-08-23 15:34                                                     ` Pavel Hofman
  2021-08-23 18:54                                                       ` Ferry Toth
@ 2021-08-23 22:50                                                       ` Thinh Nguyen
  2021-08-24  5:39                                                         ` Pavel Hofman
  1 sibling, 1 reply; 73+ messages in thread
From: Thinh Nguyen @ 2021-08-23 22:50 UTC (permalink / raw)
  To: Pavel Hofman, Andy Shevchenko
  Cc: Ferry Toth, Felipe Balbi, Wesley Cheng, gregkh, robh+dt, agross,
	bjorn.andersson, frowand.list, linux-usb, linux-kernel,
	devicetree, linux-arm-msm, jackp, heikki.krogerus,
	Ruslan Bilovol

Pavel Hofman wrote:
> 
> 
> Dne 23. 08. 21 v 17:21 Andy Shevchenko napsal(a):
>> On Mon, Aug 23, 2021 at 5:59 PM Pavel Hofman
>> <pavel.hofman@ivitera.com> wrote:
>>> Dne 22. 08. 21 v 21:43 Ferry Toth napsal(a):
>>>> Op 19-08-2021 om 23:04 schreef Pavel Hofman:
>>>>> Dne 19. 08. 21 v 22:10 Ferry Toth napsal(a):
>>>>>> Op 19-08-2021 om 09:51 schreef Pavel Hofman:
>>>>>>> Dne 18. 08. 21 v 21:07 Ferry Toth napsal(a):
>>>>>>>> Op 18-08-2021 om 00:00 schreef Ferry Toth:
>>
>> ...
>>
>>>>>>>> So, where do we go from here?
>>>>>>>
>>>>>>> I know the patches have been tested on dwc2 (by me and others).  I
>>>>>>> do not know if Ruslan or Jerome tested them on dwc3 but probably
>>>>>>> not. Ruslan has talked about RPi (my case too) and BeagleboneBlack,
>>>>>>> both with dwc2. Perhaps the dwc2 behaves a bit differently than
>>>>>>> dwc3?
>>>>>>>
>>>>>>> The patches add a new EP-IN for async feedback. I am sorry I have
>>>>>>> not followed your long thread (it started as unrelated to uac). Does
>>>>>>> the problem appear with f_uac1 or f_uac2? Please how have you
>>>>>>> reached the above problem?
>>>>>>
>>>>>> I'm sorry too. I first believed the issue was related to the patch
>>>>>> mentioned in the subject line.
>>>>>>
>>>>>> The problem appaers with f_uac2. I bost Edison_Arduino board in host
>>>>>> mode (there is a switch allowing to select host/device mode). When
>>>>>> flipping the switch to device mode udev run a script:
>>>>>> But as I am using configfs (excerpt follows) and just disabling the
>>>>>> last 2 line resolves the issue, I'm guessing uac2 is the issue. Or
>>>>>> exceeding the available resources.
>>>>>>
>>>>>> # Create directory structure
>>>>>> mkdir "${GADGET_BASE_DIR}"
>>>>>> cd "${GADGET_BASE_DIR}"
>>>>>> mkdir -p configs/c.1/strings/0x409
>>>>>> mkdir -p strings/0x409
>>>>>>
>>>>>> # Serial device
>>>>>> mkdir functions/gser.usb0
>>>>>> ln -s functions/gser.usb0 configs/c.1/
>>>>>> ###
>>>>>>
>>>>>> # Ethernet device
>>>>>> mkdir functions/eem.usb0
>>>>>> echo "${DEV_ETH_ADDR}" > functions/eem.usb0/dev_addr
>>>>>> echo "${HOST_ETH_ADDR}" > functions/eem.usb0/host_addr
>>>>>> ln -s functions/eem.usb0 configs/c.1/
>>>>>>
>>>>>> # Mass Storage device
>>>>>> mkdir functions/mass_storage.usb0
>>>>>> echo 1 > functions/mass_storage.usb0/stall
>>>>>> echo 0 > functions/mass_storage.usb0/lun.0/cdrom
>>>>>> echo 0 > functions/mass_storage.usb0/lun.0/ro
>>>>>> echo 0 > functions/mass_storage.usb0/lun.0/nofua
>>>>>> echo "${USBDISK}" > functions/mass_storage.usb0/lun.0/file
>>>>>> ln -s functions/mass_storage.usb0 configs/c.1/
>>>>>>
>>>>>> # UAC2 device
>>>>>> mkdir functions/uac2.usb0
>>>>>> ln -s functions/uac2.usb0 configs/c.1
>>>>>> ....
>>>>>
>>>>> As you say, could perhaps the reason be that the extra EP-IN added in
>>>>> those patches (previously 1, now 2 with the default config you use)
>>>>> exceeds your EP-IN max count or available fifos somehow?  You have a
>>>>> number of functions initialized. If you change the load order of the
>>>>> functions, do you get the error later with a different function? Just
>>>>> guessing...
>>>>>
>>>>> You should be able to switch the default async EP-OUT (which
>>>>> configures the new feedback EP-IN ) to adaptive EP-OUT (which requires
>>>>> no feedback EP) with c_sync=8 parameter of f_uac2.
>>>>>
>>>>> https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c*L47__;Iw!!A4F2R9G_pg!LBySrM_zgMGV0x-zZ7nSrs54yJw1GlnpUVUVxdQE8PeszSMZ6OkFX8lSoigwRbWQzLcU$
>>>>>
>>>>> https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c*L830__;Iw!!A4F2R9G_pg!LBySrM_zgMGV0x-zZ7nSrs54yJw1GlnpUVUVxdQE8PeszSMZ6OkFX8lSoigwRfP5TdZC$
>>>>>
>>>>> https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14-rc6/source/include/uapi/linux/usb/ch9.h*L453__;Iw!!A4F2R9G_pg!LBySrM_zgMGV0x-zZ7nSrs54yJw1GlnpUVUVxdQE8PeszSMZ6OkFX8lSoigwRejzMbWO$
>>>>>
>>>>> Does that fix the problem?
>>>>
>>>> Not sure how to do that. Do you mean the module should have a parameter
>>>> called c_sync? `modinfo` list no parameters for usb_f_uac2.
>>>
>>> Those are configfs params, not available in modinfo.
>>>
>>> I checked and the value is "adaptive"
>>> https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14-rc7/source/drivers/usb/gadget/function/f_uac2.c*L1312__;Iw!!A4F2R9G_pg!LBySrM_zgMGV0x-zZ7nSrs54yJw1GlnpUVUVxdQE8PeszSMZ6OkFX8lSoigwRTETcbsN$
>>
>>
>>> In your configfs script:
>>
>> Kernel shouldn't crash with any available set of configuration
>> parameters, right? So, even if the parameter would fix the crash the
>> series is buggy and has to be reverted in my opinion.
> 
> Sure, no problem with reverting. I am just trying to start up some
> troubleshooting. A resource exhaustion was mentioned here, that's why I
> suggested a way to test the patch with the original number of endpoints
> allocated. I do not get any crashes on my setup which uses fewer gadget
> functions. That's why I asked what happens if the functions load order
> is changed. I am afraid this thread is so complex that the actual
> problem has been burried in the history.
> 

As I pointed out previously, the crash is probably because of f_uac2
prematurely freeing feedback request before its completion.
usb_ep_dequeue() is asynchronous. dwc2() may treat it as a synchronous
call so you didn't get a crash.

> Again, I am not the author of the patches, my USB knowledge is way
> unsufficient for that. I am trying to make them work as they are
> important and make the existing audio gadget actually usable.
> 

I'm not sure how easy it is for you to obtain/test a device with dwc3,
but it would be great to also have it as part of your testing suite. :)

Thanks,
Thinh

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-08-23 22:50                                                       ` Thinh Nguyen
@ 2021-08-24  5:39                                                         ` Pavel Hofman
  2021-08-24 13:44                                                           ` Ferry Toth
       [not found]                                                           ` <7ffab777-0f77-f949-f70f-7bf34c504381@gmail.com>
  0 siblings, 2 replies; 73+ messages in thread
From: Pavel Hofman @ 2021-08-24  5:39 UTC (permalink / raw)
  To: Thinh Nguyen, Andy Shevchenko
  Cc: Ferry Toth, Felipe Balbi, Wesley Cheng, gregkh, robh+dt, agross,
	bjorn.andersson, frowand.list, linux-usb, linux-kernel,
	devicetree, linux-arm-msm, jackp, heikki.krogerus,
	Ruslan Bilovol



Dne 24. 08. 21 v 0:50 Thinh Nguyen napsal(a):
> Pavel Hofman wrote:
>>
>>
>> Dne 23. 08. 21 v 17:21 Andy Shevchenko napsal(a):
>>> On Mon, Aug 23, 2021 at 5:59 PM Pavel Hofman
>>> <pavel.hofman@ivitera.com> wrote:
>>>> Dne 22. 08. 21 v 21:43 Ferry Toth napsal(a):
>>>>> Op 19-08-2021 om 23:04 schreef Pavel Hofman:
>>>>>> Dne 19. 08. 21 v 22:10 Ferry Toth napsal(a):
>>>>>>> Op 19-08-2021 om 09:51 schreef Pavel Hofman:
>>>>>>>> Dne 18. 08. 21 v 21:07 Ferry Toth napsal(a):
>>>>>>>>> Op 18-08-2021 om 00:00 schreef Ferry Toth:
>>>
>>> ...
>>>
>>>>>>>>> So, where do we go from here?
>>>>>>>>
>>>>>>>> I know the patches have been tested on dwc2 (by me and others).  I
>>>>>>>> do not know if Ruslan or Jerome tested them on dwc3 but probably
>>>>>>>> not. Ruslan has talked about RPi (my case too) and BeagleboneBlack,
>>>>>>>> both with dwc2. Perhaps the dwc2 behaves a bit differently than
>>>>>>>> dwc3?
>>>>>>>>
>>>>>>>> The patches add a new EP-IN for async feedback. I am sorry I have
>>>>>>>> not followed your long thread (it started as unrelated to uac). Does
>>>>>>>> the problem appear with f_uac1 or f_uac2? Please how have you
>>>>>>>> reached the above problem?
>>>>>>>
>>>>>>> I'm sorry too. I first believed the issue was related to the patch
>>>>>>> mentioned in the subject line.
>>>>>>>
>>>>>>> The problem appaers with f_uac2. I bost Edison_Arduino board in host
>>>>>>> mode (there is a switch allowing to select host/device mode). When
>>>>>>> flipping the switch to device mode udev run a script:
>>>>>>> But as I am using configfs (excerpt follows) and just disabling the
>>>>>>> last 2 line resolves the issue, I'm guessing uac2 is the issue. Or
>>>>>>> exceeding the available resources.
>>>>>>>
>>>>>>> # Create directory structure
>>>>>>> mkdir "${GADGET_BASE_DIR}"
>>>>>>> cd "${GADGET_BASE_DIR}"
>>>>>>> mkdir -p configs/c.1/strings/0x409
>>>>>>> mkdir -p strings/0x409
>>>>>>>
>>>>>>> # Serial device
>>>>>>> mkdir functions/gser.usb0
>>>>>>> ln -s functions/gser.usb0 configs/c.1/
>>>>>>> ###
>>>>>>>
>>>>>>> # Ethernet device
>>>>>>> mkdir functions/eem.usb0
>>>>>>> echo "${DEV_ETH_ADDR}" > functions/eem.usb0/dev_addr
>>>>>>> echo "${HOST_ETH_ADDR}" > functions/eem.usb0/host_addr
>>>>>>> ln -s functions/eem.usb0 configs/c.1/
>>>>>>>
>>>>>>> # Mass Storage device
>>>>>>> mkdir functions/mass_storage.usb0
>>>>>>> echo 1 > functions/mass_storage.usb0/stall
>>>>>>> echo 0 > functions/mass_storage.usb0/lun.0/cdrom
>>>>>>> echo 0 > functions/mass_storage.usb0/lun.0/ro
>>>>>>> echo 0 > functions/mass_storage.usb0/lun.0/nofua
>>>>>>> echo "${USBDISK}" > functions/mass_storage.usb0/lun.0/file
>>>>>>> ln -s functions/mass_storage.usb0 configs/c.1/
>>>>>>>
>>>>>>> # UAC2 device
>>>>>>> mkdir functions/uac2.usb0
>>>>>>> ln -s functions/uac2.usb0 configs/c.1
>>>>>>> ....
>>>>>>
>>>>>> As you say, could perhaps the reason be that the extra EP-IN added in
>>>>>> those patches (previously 1, now 2 with the default config you use)
>>>>>> exceeds your EP-IN max count or available fifos somehow?  You have a
>>>>>> number of functions initialized. If you change the load order of the
>>>>>> functions, do you get the error later with a different function? Just
>>>>>> guessing...
>>>>>>
>>>>>> You should be able to switch the default async EP-OUT (which
>>>>>> configures the new feedback EP-IN ) to adaptive EP-OUT (which requires
>>>>>> no feedback EP) with c_sync=8 parameter of f_uac2.
>>>>>>
>>>>>> https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c*L47__;Iw!!A4F2R9G_pg!LBySrM_zgMGV0x-zZ7nSrs54yJw1GlnpUVUVxdQE8PeszSMZ6OkFX8lSoigwRbWQzLcU$
>>>>>>
>>>>>> https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c*L830__;Iw!!A4F2R9G_pg!LBySrM_zgMGV0x-zZ7nSrs54yJw1GlnpUVUVxdQE8PeszSMZ6OkFX8lSoigwRfP5TdZC$
>>>>>>
>>>>>> https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14-rc6/source/include/uapi/linux/usb/ch9.h*L453__;Iw!!A4F2R9G_pg!LBySrM_zgMGV0x-zZ7nSrs54yJw1GlnpUVUVxdQE8PeszSMZ6OkFX8lSoigwRejzMbWO$
>>>>>>
>>>>>> Does that fix the problem?
>>>>>
>>>>> Not sure how to do that. Do you mean the module should have a parameter
>>>>> called c_sync? `modinfo` list no parameters for usb_f_uac2.
>>>>
>>>> Those are configfs params, not available in modinfo.
>>>>
>>>> I checked and the value is "adaptive"
>>>> https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14-rc7/source/drivers/usb/gadget/function/f_uac2.c*L1312__;Iw!!A4F2R9G_pg!LBySrM_zgMGV0x-zZ7nSrs54yJw1GlnpUVUVxdQE8PeszSMZ6OkFX8lSoigwRTETcbsN$
>>>
>>>
>>>> In your configfs script:
>>>
>>> Kernel shouldn't crash with any available set of configuration
>>> parameters, right? So, even if the parameter would fix the crash the
>>> series is buggy and has to be reverted in my opinion.
>>
>> Sure, no problem with reverting. I am just trying to start up some
>> troubleshooting. A resource exhaustion was mentioned here, that's why I
>> suggested a way to test the patch with the original number of endpoints
>> allocated. I do not get any crashes on my setup which uses fewer gadget
>> functions. That's why I asked what happens if the functions load order
>> is changed. I am afraid this thread is so complex that the actual
>> problem has been burried in the history.
>>
> 
> As I pointed out previously, the crash is probably because of f_uac2
> prematurely freeing feedback request before its completion.
> usb_ep_dequeue() is asynchronous. dwc2() may treat it as a synchronous
> call so you didn't get a crash.

Thanks for your hint, it greatly helps.
>>
> 
> I'm not sure how easy it is for you to obtain/test a device with dwc3,
> but it would be great to also have it as part of your testing suite. :)

Can you recommend a reasonably priced device with viable kernel updates 
for the testing? Optionally with SS which you mentioned last time? Thanks.

Best regards,

Pavel.

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
  2021-08-24  5:39                                                         ` Pavel Hofman
@ 2021-08-24 13:44                                                           ` Ferry Toth
       [not found]                                                           ` <7ffab777-0f77-f949-f70f-7bf34c504381@gmail.com>
  1 sibling, 0 replies; 73+ messages in thread
From: Ferry Toth @ 2021-08-24 13:44 UTC (permalink / raw)
  To: Pavel Hofman, Thinh Nguyen, Andy Shevchenko
  Cc: Felipe Balbi, Wesley Cheng, gregkh, robh+dt, agross,
	bjorn.andersson, frowand.list, linux-usb, linux-kernel,
	devicetree, linux-arm-msm, jackp, heikki.krogerus,
	Ruslan Bilovol

Hi

Op 24-08-2021 om 07:39 schreef Pavel Hofman:
>
>
> Dne 24. 08. 21 v 0:50 Thinh Nguyen napsal(a):
>> Pavel Hofman wrote:
>>>
>>>
>>> Dne 23. 08. 21 v 17:21 Andy Shevchenko napsal(a):
>>>> On Mon, Aug 23, 2021 at 5:59 PM Pavel Hofman
>>>> <pavel.hofman@ivitera.com> wrote:
>>>>> Dne 22. 08. 21 v 21:43 Ferry Toth napsal(a):
>>>>>> Op 19-08-2021 om 23:04 schreef Pavel Hofman:
>>>>>>> Dne 19. 08. 21 v 22:10 Ferry Toth napsal(a):
>>>>>>>> Op 19-08-2021 om 09:51 schreef Pavel Hofman:
>>>>>>>>> Dne 18. 08. 21 v 21:07 Ferry Toth napsal(a):
>>>>>>>>>> Op 18-08-2021 om 00:00 schreef Ferry Toth:
>>>>
>>>> ...
>>>>
>>>>>>>>>> So, where do we go from here?
>>>>>>>>>
>>>>>>>>> I know the patches have been tested on dwc2 (by me and 
>>>>>>>>> others).  I
>>>>>>>>> do not know if Ruslan or Jerome tested them on dwc3 but probably
>>>>>>>>> not. Ruslan has talked about RPi (my case too) and 
>>>>>>>>> BeagleboneBlack,
>>>>>>>>> both with dwc2. Perhaps the dwc2 behaves a bit differently than
>>>>>>>>> dwc3?
>>>>>>>>>
>>>>>>>>> The patches add a new EP-IN for async feedback. I am sorry I have
>>>>>>>>> not followed your long thread (it started as unrelated to 
>>>>>>>>> uac). Does
>>>>>>>>> the problem appear with f_uac1 or f_uac2? Please how have you
>>>>>>>>> reached the above problem?
>>>>>>>>
>>>>>>>> I'm sorry too. I first believed the issue was related to the patch
>>>>>>>> mentioned in the subject line.
>>>>>>>>
>>>>>>>> The problem appaers with f_uac2. I bost Edison_Arduino board in 
>>>>>>>> host
>>>>>>>> mode (there is a switch allowing to select host/device mode). When
>>>>>>>> flipping the switch to device mode udev run a script:
>>>>>>>> But as I am using configfs (excerpt follows) and just disabling 
>>>>>>>> the
>>>>>>>> last 2 line resolves the issue, I'm guessing uac2 is the issue. Or
>>>>>>>> exceeding the available resources.
>>>>>>>>
>>>>>>>> # Create directory structure
>>>>>>>> mkdir "${GADGET_BASE_DIR}"
>>>>>>>> cd "${GADGET_BASE_DIR}"
>>>>>>>> mkdir -p configs/c.1/strings/0x409
>>>>>>>> mkdir -p strings/0x409
>>>>>>>>
>>>>>>>> # Serial device
>>>>>>>> mkdir functions/gser.usb0
>>>>>>>> ln -s functions/gser.usb0 configs/c.1/
>>>>>>>> ###
>>>>>>>>
>>>>>>>> # Ethernet device
>>>>>>>> mkdir functions/eem.usb0
>>>>>>>> echo "${DEV_ETH_ADDR}" > functions/eem.usb0/dev_addr
>>>>>>>> echo "${HOST_ETH_ADDR}" > functions/eem.usb0/host_addr
>>>>>>>> ln -s functions/eem.usb0 configs/c.1/
>>>>>>>>
>>>>>>>> # Mass Storage device
>>>>>>>> mkdir functions/mass_storage.usb0
>>>>>>>> echo 1 > functions/mass_storage.usb0/stall
>>>>>>>> echo 0 > functions/mass_storage.usb0/lun.0/cdrom
>>>>>>>> echo 0 > functions/mass_storage.usb0/lun.0/ro
>>>>>>>> echo 0 > functions/mass_storage.usb0/lun.0/nofua
>>>>>>>> echo "${USBDISK}" > functions/mass_storage.usb0/lun.0/file
>>>>>>>> ln -s functions/mass_storage.usb0 configs/c.1/
>>>>>>>>
>>>>>>>> # UAC2 device
>>>>>>>> mkdir functions/uac2.usb0
>>>>>>>> ln -s functions/uac2.usb0 configs/c.1
>>>>>>>> ....
>>>>>>>
>>>>>>> As you say, could perhaps the reason be that the extra EP-IN 
>>>>>>> added in
>>>>>>> those patches (previously 1, now 2 with the default config you use)
>>>>>>> exceeds your EP-IN max count or available fifos somehow?  You 
>>>>>>> have a
>>>>>>> number of functions initialized. If you change the load order of 
>>>>>>> the
>>>>>>> functions, do you get the error later with a different function? 
>>>>>>> Just
>>>>>>> guessing...
>>>>>>>
>>>>>>> You should be able to switch the default async EP-OUT (which
>>>>>>> configures the new feedback EP-IN ) to adaptive EP-OUT (which 
>>>>>>> requires
>>>>>>> no feedback EP) with c_sync=8 parameter of f_uac2.
>>>>>>>
>>>>>>> https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c*L47__;Iw!!A4F2R9G_pg!LBySrM_zgMGV0x-zZ7nSrs54yJw1GlnpUVUVxdQE8PeszSMZ6OkFX8lSoigwRbWQzLcU$ 
>>>>>>>
>>>>>>>
>>>>>>> https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14-rc6/source/drivers/usb/gadget/function/f_uac2.c*L830__;Iw!!A4F2R9G_pg!LBySrM_zgMGV0x-zZ7nSrs54yJw1GlnpUVUVxdQE8PeszSMZ6OkFX8lSoigwRfP5TdZC$ 
>>>>>>>
>>>>>>>
>>>>>>> https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14-rc6/source/include/uapi/linux/usb/ch9.h*L453__;Iw!!A4F2R9G_pg!LBySrM_zgMGV0x-zZ7nSrs54yJw1GlnpUVUVxdQE8PeszSMZ6OkFX8lSoigwRejzMbWO$ 
>>>>>>>
>>>>>>>
>>>>>>> Does that fix the problem?
>>>>>>
>>>>>> Not sure how to do that. Do you mean the module should have a 
>>>>>> parameter
>>>>>> called c_sync? `modinfo` list no parameters for usb_f_uac2.
>>>>>
>>>>> Those are configfs params, not available in modinfo.
>>>>>
>>>>> I checked and the value is "adaptive"
>>>>> https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14-rc7/source/drivers/usb/gadget/function/f_uac2.c*L1312__;Iw!!A4F2R9G_pg!LBySrM_zgMGV0x-zZ7nSrs54yJw1GlnpUVUVxdQE8PeszSMZ6OkFX8lSoigwRTETcbsN$ 
>>>>>
>>>>
>>>>
>>>>> In your configfs script:
>>>>
>>>> Kernel shouldn't crash with any available set of configuration
>>>> parameters, right? So, even if the parameter would fix the crash the
>>>> series is buggy and has to be reverted in my opinion.
>>>
>>> Sure, no problem with reverting. I am just trying to start up some
>>> troubleshooting. A resource exhaustion was mentioned here, that's why I
>>> suggested a way to test the patch with the original number of endpoints
>>> allocated. I do not get any crashes on my setup which uses fewer gadget
>>> functions. That's why I asked what happens if the functions load order
>>> is changed. I am afraid this thread is so complex that the actual
>>> problem has been burried in the history.
>>>
>>
>> As I pointed out previously, the crash is probably because of f_uac2
>> prematurely freeing feedback request before its completion.
>> usb_ep_dequeue() is asynchronous. dwc2() may treat it as a synchronous
>> call so you didn't get a crash.
>
> Thanks for your hint, it greatly helps.
>>>
>>
>> I'm not sure how easy it is for you to obtain/test a device with dwc3,
>> but it would be great to also have it as part of your testing suite. :)
>
> Can you recommend a reasonably priced device with viable kernel 
> updates for the testing? Optionally with SS which you mentioned last 
> time? Thanks.
>
Edison-Arduino kit 2nd hand with display on ebay ~$100 :-)

> Best regards,
>
> Pavel.

^ permalink raw reply	[flat|nested] 73+ messages in thread

* Re: [PATCH v10 0/6] Re-introduce TX FIFO resize for larger EP bursting
       [not found]                                                           ` <7ffab777-0f77-f949-f70f-7bf34c504381@gmail.com>
@ 2021-08-26  1:18                                                             ` Thinh Nguyen
  2021-08-26  7:30                                                               ` Ferry Toth
  2021-08-26 13:35                                                               ` Pavel Hofman
  0 siblings, 2 replies; 73+ messages in thread
From: Thinh Nguyen @ 2021-08-26  1:18 UTC (permalink / raw)
  To: Ferry Toth, Pavel Hofman, Thinh Nguyen, Andy Shevchenko
  Cc: Felipe Balbi, Wesley Cheng, gregkh, robh+dt, agross,
	bjorn.andersson, frowand.list, linux-usb, linux-kernel,
	devicetree, linux-arm-msm, jackp, heikki.krogerus,
	Ruslan Bilovol

Ferry Toth wrote:
> Hi
> 
> Op 24-08-2021 om 07:39 schreef Pavel Hofman:
>>
>>
>> Dne 24. 08. 21 v 0:50 Thinh Nguyen napsal(a):
>>> Pavel Hofman wrote:
>>>>
>>>>
>>>> Dne 23. 08. 21 v 17:21 Andy Shevchenko napsal(a):
>>>>> On Mon, Aug 23, 2021 at 5:59 PM Pavel Hofman
>>