Enabling Device DWC3 Device Mode

Enabling Device DWC3 Device Mode

[Sid Spry]

I've a RK3399 based board (RockPro64) and am unable to find any useful information on how to ensure the USB controller is set up for device/gadget operation. 

The base armbian (Ubuntu Focal based distribution) kernel did not have dwc2/dwc3 modules compiled. I compiled the modules for kernel ~5.7 after ensuring that dual role mode was supported along with gadget configfs and am able to modprobe them, but I see no devices populate in /sys/class/udc and no dmesg errors.

I should be able to target any kernel version, but I am concerned that there may be missing device tree information but am unsure how to check that. Most people seem uninterested in using the device mode on this hardware, but it was 1 of 2 boards, based on the same part, that I found claiming to support device mode USB3.

Cheers!

RE: Enabling Device DWC3 Device Mode

[Ran Wang]

Hi Sid,

> -----Original Message-----
> From: Sid Spry <sid@aeam.us>
> Sent: Friday, September 18, 2020 7:55 AM
> To: linux-usb@vger.kernel.org
> Subject: Enabling Device DWC3 Device Mode
> 
> I've a RK3399 based board (RockPro64) and am unable to find any useful
> information on how to ensure the USB controller is set up for device/gadget
> operation.
> 
> The base armbian (Ubuntu Focal based distribution) kernel did not have
> dwc2/dwc3 modules compiled. I compiled the modules for kernel ~5.7 after
> ensuring that dual role mode was supported along with gadget configfs and
> am able to modprobe them, but I see no devices populate in /sys/class/udc
> and no dmesg errors.
> 
> I should be able to target any kernel version, but I am concerned that there
> may be missing device tree information but am unsure how to check that.
> Most people seem uninterested in using the device mode on this hardware,
> but it was 1 of 2 boards, based on the same part, that I found claiming to
> support device mode USB3.

For DWC3 device mode support, maybe you could check Layerscape platforms
as a reference. There is a doc which has details on how-to: https://www.nxp.com/docs/en/user-guide/LSDKUG_Rev20.04.pdf
search 'Gadget driver' .

One of the example in DTS is arch/arm64/boot/dts/freescale/fsl-lx2160a.dtsi
and arch/arm64/boot/dts/freescale/fsl-lx2160a-rdb.dts, and you need to manually
update it by following above doc to enable device mode.

Regards,
Ran

> Cheers!

Re: Enabling Device DWC3 Device Mode

[Sid Spry]

Thanks Ran. I took a look at the document, if you or anyone else can weigh in
I'd appreciate it. The platform devices seem to be active, see below.

On Thu, Sep 17, 2020, at 8:11 PM, Ran Wang wrote:
> Hi Sid,
> 
> > -----Original Message-----
> > From: Sid Spry <sid@aeam.us>
> > Sent: Friday, September 18, 2020 7:55 AM
> > To: linux-usb@vger.kernel.org
> > Subject: Enabling Device DWC3 Device Mode
> > 
> > I've a RK3399 based board (RockPro64) and am unable to find any useful
> > information on how to ensure the USB controller is set up for device/gadget
> > operation.
> > 
> > The base armbian (Ubuntu Focal based distribution) kernel did not have
> > dwc2/dwc3 modules compiled. I compiled the modules for kernel ~5.7 after
> > ensuring that dual role mode was supported along with gadget configfs and
> > am able to modprobe them, but I see no devices populate in /sys/class/udc
> > and no dmesg errors.
> > 
> > I should be able to target any kernel version, but I am concerned that there
> > may be missing device tree information but am unsure how to check that.
> > Most people seem uninterested in using the device mode on this hardware,
> > but it was 1 of 2 boards, based on the same part, that I found claiming to
> > support device mode USB3.
> 
> For DWC3 device mode support, maybe you could check Layerscape platforms
> as a reference. There is a doc which has details on how-to: 
> https://www.nxp.com/docs/en/user-guide/LSDKUG_Rev20.04.pdf
> search 'Gadget driver' .
> 

Thanks for the pointer. Around line 771 is what I expected to see for my board.
However, in arch/arm64/boot/dts/rockchip/rk3399-rockpro64.dtsi (I have the v2
board) the USB related entries I see are:

&usbdrd3_0 {
        status = "okay";
};

&usbdrd_dwc3_0 {
        status = "okay";
        dr_mode = "otg";
};

&usbdrd3_1 {
        status = "okay";
};

&usbdrd_dwc3_1 {
        status = "okay";
        dr_mode = "host";
};

Starting at line 767. It's a lot sparser, but the dr_mode is there. I feel I should
note that despite the lack of devices in /sys/class/udc I have nodes in
/sys/bus/platform/drivers/dwc3:

# ls /sys/bus/platform/drivers/dwc3
bind  fe800000.usb  fe900000.usb  module  uevent  unbind

> One of the example in DTS is arch/arm64/boot/dts/freescale/fsl-lx2160a.dtsi
> and arch/arm64/boot/dts/freescale/fsl-lx2160a-rdb.dts, and you need to manually
> update it by following above doc to enable device mode.
> 

The document linked is relevant beginning page 445. The above dts settings
seem appropriate and I have the necessary kernel configuration enabled. Any
relevant drivers are modules.

I'm not really sure how to diagnose it from here. Any help is appreciated.

Thanks in advance!

RE: Enabling Device DWC3 Device Mode

[Ran Wang]

Hi Sid,

> -----Original Message-----
> From: Sid Spry <sid@aeam.us>
> Sent: Friday, September 18, 2020 11:08 AM
> To: Ran Wang <ran.wang_1@nxp.com>
> Cc: linux-usb@vger.kernel.org
> Subject: Re: Enabling Device DWC3 Device Mode
> 
> Thanks Ran. I took a look at the document, if you or anyone else can weigh in
> I'd appreciate it. The platform devices seem to be active, see below.

<snip>

>
> 
> Thanks for the pointer. Around line 771 is what I expected to see for my board.
> However, in arch/arm64/boot/dts/rockchip/rk3399-rockpro64.dtsi (I have the
> v2
> board) the USB related entries I see are:
> 
> &usbdrd3_0 {
>         status = "okay";
> };
> 
> &usbdrd_dwc3_0 {
>         status = "okay";
>         dr_mode = "otg";
> };
> 
> &usbdrd3_1 {
>         status = "okay";
> };
> 
> &usbdrd_dwc3_1 {
>         status = "okay";
>         dr_mode = "host";
> };

So you want to use 'otg' or 'preripheral'? The SW flows is a little bit different (in dwc3/core.c)
If you want otg, I guess module solution would not be good. But I have no idea how to enable
mass storage gadget in build-in way, to be honest. So you could try ethernet gadget which
doesn't require passing parameters, it works on my part.

> Starting at line 767. It's a lot sparser, but the dr_mode is there. I feel I should
> note that despite the lack of devices in /sys/class/udc I have nodes in
> /sys/bus/platform/drivers/dwc3:
> 
> # ls /sys/bus/platform/drivers/dwc3
> bind  fe800000.usb  fe900000.usb  module  uevent  unbind
> 
> > One of the example in DTS is
> > arch/arm64/boot/dts/freescale/fsl-lx2160a.dtsi
> > and arch/arm64/boot/dts/freescale/fsl-lx2160a-rdb.dts, and you need to
> > manually update it by following above doc to enable device mode.
> >
> 
> The document linked is relevant beginning page 445. The above dts settings
> seem appropriate and I have the necessary kernel configuration enabled. Any
> relevant drivers are modules.
> 
> I'm not really sure how to diagnose it from here. Any help is appreciated.

Parsing property 'dr_mode' is the key to control driver logic path.

I think you could begin with drivers/usb/dwc3/core.c, see what happen in dwc3_core_init_mode()
which handling role switching (host/peripheral/otg) in calling dwc3_probe()
if you have made sure all kernel configuration had been done correctly, such as (ethernet gadget):
CONFIG_USB_DWC3=y
CONFIG_USB_DWC3_DUAL_ROLE=y
CONFIG_USB_GADGET=y
CONFIG_USB_CONFIGFS=y
CONFIG_USB_ETH=y

With dr_mode=peripheral (or otg), it should work (if above dwc3_probe() encountered no error).

> Thanks in advance!

Welcome

Regards,
Ran

Re: Enabling Device DWC3 Device Mode

[Sid Spry]

Hello list and Ran, I've some more observations. Firstly, I have to set the dwc3
mode to peripheral directly. Checking the code I was unable to find what actually
allows you to select the mode if specifying OTG.

I have devices in /sys/class/udc and they function properly via configfs when
plugged into a USB2 only port. If I plug them into a superspeed port I get
nothing in dmesg on the host side. At first, I did get a response on the host, in
the form of:

[591550.770819] usb 1-1.2: new full-speed USB device number 76 using xhci_hcd
[591550.870962] usb 1-1.2: device descriptor read/64, error -32
[591551.078865] usb 1-1.2: device descriptor read/64, error -32
[591551.286875] usb 1-1.2: new full-speed USB device number 77 using xhci_hcd
[591551.386873] usb 1-1.2: device descriptor read/64, error -32
[591551.594859] usb 1-1.2: device descriptor read/64, error -32
[591551.703160] usb 1-1-port2: attempt power cycle

However now I get no driver activity when binding the UDC in configfs or
loading or unloading the respective modules. Kernel is 5.7.15 w/ patches.

It doesn't seem like a fried the port or tripped a polyfuse, the port still works
after attempting USB3 device mode.

On Thu, Sep 17, 2020, at 10:42 PM, Ran Wang wrote:
> Hi Sid,
> 
> > -----Original Message-----
> > From: Sid Spry <sid@aeam.us>
> > Sent: Friday, September 18, 2020 11:08 AM
> > To: Ran Wang <ran.wang_1@nxp.com>
> > Cc: linux-usb@vger.kernel.org
> > Subject: Re: Enabling Device DWC3 Device Mode
> > 
> > Thanks Ran. I took a look at the document, if you or anyone else can weigh in
> > I'd appreciate it. The platform devices seem to be active, see below.
>
> So you want to use 'otg' or 'preripheral'? The SW flows is a little bit 
> different (in dwc3/core.c)
> If you want otg, I guess module solution would not be good. But I have 
> no idea how to enable
> mass storage gadget in build-in way, to be honest. So you could try 
> ethernet gadget which
> doesn't require passing parameters, it works on my part.
> 
> > Starting at line 767. It's a lot sparser, but the dr_mode is there. I feel I should
> > note that despite the lack of devices in /sys/class/udc I have nodes in
> > /sys/bus/platform/drivers/dwc3:
> > 
> > # ls /sys/bus/platform/drivers/dwc3
> > bind  fe800000.usb  fe900000.usb  module  uevent  unbind
> > 
> > > One of the example in DTS is
> > > arch/arm64/boot/dts/freescale/fsl-lx2160a.dtsi
> > > and arch/arm64/boot/dts/freescale/fsl-lx2160a-rdb.dts, and you need to
> > > manually update it by following above doc to enable device mode.
> > >
> > 
> > The document linked is relevant beginning page 445. The above dts settings
> > seem appropriate and I have the necessary kernel configuration enabled. Any
> > relevant drivers are modules.
> > 
> > I'm not really sure how to diagnose it from here. Any help is appreciated.
> 
> Parsing property 'dr_mode' is the key to control driver logic path.
> 
> I think you could begin with drivers/usb/dwc3/core.c, see what happen 
> in dwc3_core_init_mode()
> which handling role switching (host/peripheral/otg) in calling 
> dwc3_probe()
> if you have made sure all kernel configuration had been done correctly, 
> such as (ethernet gadget):
> CONFIG_USB_DWC3=y
> CONFIG_USB_DWC3_DUAL_ROLE=y
> CONFIG_USB_GADGET=y
> CONFIG_USB_CONFIGFS=y
> CONFIG_USB_ETH=y
> 
> With dr_mode=peripheral (or otg), it should work (if above dwc3_probe() 
> encountered no error).
> 

I will pursue this in more detail so I don't need to reboot to change dwc3 mode
but didn't find any relevant code path for otg -> device from userspace. I'll try
debugging later.

Sid.

RE: Enabling Device DWC3 Device Mode

[Ran Wang]

Hi Sid,

> -----Original Message-----
> From: Sid Spry <sid@aeam.us>
> Sent: Saturday, September 19, 2020 1:59 PM
> To: Ran Wang <ran.wang_1@nxp.com>
> Cc: linux-usb@vger.kernel.org
> Subject: Re: Enabling Device DWC3 Device Mode
> 
> Hello list and Ran, I've some more observations. Firstly, I have to set the dwc3
> mode to peripheral directly. Checking the code I was unable to find what
> actually allows you to select the mode if specifying OTG.

Yeah, I also think debugging on peripheral firstly would be easier than OTG.

> I have devices in /sys/class/udc and they function properly via configfs when
> plugged into a USB2 only port. If I plug them into a superspeed port I get
> nothing in dmesg on the host side. At first, I did get a response on the host, in
> the form of:

So the gadget registration for DWC3 UDC encountered no error, right?
dwc3_core_init_mode()->dwc3_gadget_init()->usb_add_gadget_udc() return 0?
If yes, it looks like SoC internal configuring has no big issue, I think.

> [591550.770819] usb 1-1.2: new full-speed USB device number 76 using
> xhci_hcd [591550.870962] usb 1-1.2: device descriptor read/64, error -32
> [591551.078865] usb 1-1.2: device descriptor read/64, error -32
> [591551.286875] usb 1-1.2: new full-speed USB device number 77 using
> xhci_hcd [591551.386873] usb 1-1.2: device descriptor read/64, error -32
> [591551.594859] usb 1-1.2: device descriptor read/64, error -32
> [591551.703160] usb 1-1-port2: attempt power cycle

For the speed issue (expect SS but got FS), you could try adding below property to dwc3 node,
On our platform this is necessary (see dwc3_gadget_set_speed()):

maximum-speed = "super-speed";

If the get descriptor read issue still happen with above change, maybe you need to check PHY status,
or using protocol-analyzer to see what happen on bus.

> However now I get no driver activity when binding the UDC in configfs or
> loading or unloading the respective modules. Kernel is 5.7.15 w/ patches.
> 
> It doesn't seem like a fried the port or tripped a polyfuse, the port still works
> after attempting USB3 device mode.

How about dr_mode="host"? Does it work? I think it would help to verify your HW is good to go (such as PHY at least).

Regards,
Ran

Re: Enabling Device DWC3 Device Mode

[Sid Spry]

Hello list and Ran,

Well, the error I saw in dmesg about an unassignable address is something
I had only seen one other time when a device was miswired. I flipped the
USB-C connector over and it started working in USB3 ports, but at USB2
speeds. I thought of this days ago but figured the port should be handling it...

On Windows the system reports a bad setup, so I think my SS descriptors
are malformed. I'm using functionfs. I'll start a separate thread on this. On
Linux the device connects as USB2 instead of reporting an error.

If I use the NCM/ECM configfs driver debugging tools in Windows show me
that the port is superspeed capable though registered as HS. Patches hit the
list to fix this only a few days ago. Hooray!

It seems I may be able to select device mode by writing to nodes in
/sys/class/typec/port0. But I am still not sure.

On Sun, Sep 20, 2020, at 8:20 PM, Ran Wang wrote:
> Hi Sid,
> 
> > -----Original Message-----
> > From: Sid Spry <sid@aeam.us>
> > Sent: Saturday, September 19, 2020 1:59 PM
> > To: Ran Wang <ran.wang_1@nxp.com>
> > Cc: linux-usb@vger.kernel.org
> > Subject: Re: Enabling Device DWC3 Device Mode
> > 
> > Hello list and Ran, I've some more observations. Firstly, I have to set the dwc3
> > mode to peripheral directly. Checking the code I was unable to find what
> > actually allows you to select the mode if specifying OTG.
> 
> Yeah, I also think debugging on peripheral firstly would be easier than OTG.
> 
> > I have devices in /sys/class/udc and they function properly via configfs when
> > plugged into a USB2 only port. If I plug them into a superspeed port I get
> > nothing in dmesg on the host side. At first, I did get a response on the host, in
> > the form of:
> 
> So the gadget registration for DWC3 UDC encountered no error, right?
> dwc3_core_init_mode()->dwc3_gadget_init()->usb_add_gadget_udc() return 0?
> If yes, it looks like SoC internal configuring has no big issue, I think.
> 
> > [591550.770819] usb 1-1.2: new full-speed USB device number 76 using
> > xhci_hcd [591550.870962] usb 1-1.2: device descriptor read/64, error -32
> > [591551.078865] usb 1-1.2: device descriptor read/64, error -32
> > [591551.286875] usb 1-1.2: new full-speed USB device number 77 using
> > xhci_hcd [591551.386873] usb 1-1.2: device descriptor read/64, error -32
> > [591551.594859] usb 1-1.2: device descriptor read/64, error -32
> > [591551.703160] usb 1-1-port2: attempt power cycle
> 
> For the speed issue (expect SS but got FS), you could try adding below 
> property to dwc3 node,
> On our platform this is necessary (see dwc3_gadget_set_speed()):
> 
> maximum-speed = "super-speed";
> 

For anyone following along I did implement this. It may be necessary. I will
see later.

> If the get descriptor read issue still happen with above change, maybe 
> you need to check PHY status,
> or using protocol-analyzer to see what happen on bus.
> 
> > However now I get no driver activity when binding the UDC in configfs or
> > loading or unloading the respective modules. Kernel is 5.7.15 w/ patches.
> > 
> > It doesn't seem like a fried the port or tripped a polyfuse, the port still works
> > after attempting USB3 device mode.
> 
> How about dr_mode="host"? Does it work? I think it would help to verify 
> your HW is good to go (such as PHY at least).
> 

It seems to work, the issue is that the hardware isn't handling the reversible
connector properly.

Cheers.

RE: Enabling Device DWC3 Device Mode

[Ran Wang]

Hi Sid,

> -----Original Message-----
> From: Sid Spry <sid@aeam.us>
> Sent: Sunday, September 27, 2020 11:23 AM
> To: Ran Wang <ran.wang_1@nxp.com>
> Cc: linux-usb@vger.kernel.org
> Subject: Re: Enabling Device DWC3 Device Mode
> 
> Hello list and Ran,
> 
> Well, the error I saw in dmesg about an unassignable address is something I
> had only seen one other time when a device was miswired. I flipped the USB-C
> connector over and it started working in USB3 ports, but at USB2 speeds. I
> thought of this days ago but figured the port should be handling it...

Speed issue might be HW related I guess. My understanding is that speed
handshaking is handled by HW automatically, does host have this issue?

Rgards,
Ran