Re: [PATCH v4 00/16] Tegra XHCI controller ELPG support

From: JC Kuo <jckuo@nvidia.com>
To: Thierry Reding <thierry.reding@gmail.com>
Cc: <gregkh@linuxfoundation.org>, <robh@kernel.org>,
	<jonathanh@nvidia.com>, <kishon@ti.com>,
	<linux-tegra@vger.kernel.org>, <linux-usb@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>, <devicetree@vger.kernel.org>,
	<nkristam@nvidia.com>
Subject: Re: [PATCH v4 00/16] Tegra XHCI controller ELPG support
Date: Tue, 17 Nov 2020 14:14:33 +0800	[thread overview]
Message-ID: <9b64c181-1c5b-f6bd-d9db-03e30b4479a5@nvidia.com> (raw)
In-Reply-To: <20201113164455.GG1408970@ulmo>

On 11/14/20 12:44 AM, Thierry Reding wrote:
> On Fri, Oct 16, 2020 at 09:07:10PM +0800, JC Kuo wrote:
>> Tegra XHCI controler can be placed in ELPG (Engine Level PowerGated)
>> state for power saving when all of the connected USB devices are in
>> suspended state. This patch series includes clk, phy and pmc changes
>> that are required for properly place controller in ELPG and bring
>> controller out of ELPG.
>>
>> JC Kuo (16):
>>   clk: tegra: Add PLLE HW power sequencer control
>>   clk: tegra: Don't enable PLLE HW sequencer at init
>>   phy: tegra: xusb: Move usb3 port init for Tegra210
>>   phy: tegra: xusb: tegra210: Do not reset UPHY PLL
>>   phy: tegra: xusb: Rearrange UPHY init on Tegra210
>>   phy: tegra: xusb: Add Tegra210 lane_iddq operation
>>   phy: tegra: xusb: Add sleepwalk and suspend/resume
>>   soc/tegra: pmc: Provide USB sleepwalk register map
>>   arm64: tegra210: XUSB PADCTL add "nvidia,pmc" prop
>>   dt-bindings: phy: tegra-xusb: Add nvidia,pmc prop
>>   phy: tegra: xusb: Add wake/sleepwalk for Tegra210
>>   phy: tegra: xusb: Tegra210 host mode VBUS control
>>   phy: tegra: xusb: Add wake/sleepwalk for Tegra186
>>   arm64: tegra210/tegra186/tegra194: XUSB PADCTL irq
>>   usb: host: xhci-tegra: Unlink power domain devices
>>   xhci: tegra: Enable ELPG for runtime/system PM
>>
>>  .../phy/nvidia,tegra124-xusb-padctl.txt       |    1 +
>>  arch/arm64/boot/dts/nvidia/tegra186.dtsi      |    1 +
>>  arch/arm64/boot/dts/nvidia/tegra194.dtsi      |    1 +
>>  arch/arm64/boot/dts/nvidia/tegra210.dtsi      |    2 +
>>  drivers/clk/tegra/clk-pll.c                   |   12 -
>>  drivers/clk/tegra/clk-tegra210.c              |   53 +-
>>  drivers/phy/tegra/xusb-tegra186.c             |  558 ++++-
>>  drivers/phy/tegra/xusb-tegra210.c             | 1889 +++++++++++++----
>>  drivers/phy/tegra/xusb.c                      |   92 +-
>>  drivers/phy/tegra/xusb.h                      |   22 +-
>>  drivers/soc/tegra/pmc.c                       |   94 +
>>  drivers/usb/host/xhci-tegra.c                 |  610 ++++--
>>  include/linux/clk/tegra.h                     |    4 +-
>>  include/linux/phy/tegra/xusb.h                |   10 +-
>>  14 files changed, 2785 insertions(+), 564 deletions(-)
> 
> I've been testing this, but I keep seeing the following oops on suspend
> on a Jetson TX1:
> 
> [  153.451108] tegra-xusb-padctl phy-usb2.0: > tegra_xusb_padctl_suspend_noirq(dev=ffff000080917000)
> [  153.460353] tegra-xusb-padctl phy-usb2.0:   driver: ffff8000114453e0 (tegra_xusb_padctl_driver)
> [  153.469245] tegra-xusb-padctl phy-usb2.0:   padctl: ffff0000829f6480
> [  153.475772] tegra-xusb-padctl phy-usb2.0:     soc: ef7bdd7fffffffff (0xef7bdd7fffffffff)
> [  153.484061] Unable to handle kernel paging request at virtual address 007bdd800000004f
> [  153.492132] Mem abort info:
> [  153.495083]   ESR = 0x96000004
> [  153.498308]   EC = 0x25: DABT (current EL), IL = 32 bits
> [  153.503771]   SET = 0, FnV = 0
> [  153.506979]   EA = 0, S1PTW = 0
> [  153.510260] Data abort info:
> [  153.513200]   ISV = 0, ISS = 0x00000004
> [  153.517181]   CM = 0, WnR = 0
> [  153.520302] [007bdd800000004f] address between user and kernel address ranges
> [  153.527600] Internal error: Oops: 96000004 [#1] PREEMPT SMP
> [  153.533231] Modules linked in: nouveau panel_simple tegra_video(C) tegra_drm drm_ttm_helper videobuf2_dma_contig ttm videobuf2_memops cec videobuf2_v4l2 videobuf2_common drm_kms_helper v4l2_fwnode videodev drm mc snd_hda_codec_hdmi cdc_ether usbnet snd_hda_tegra r8152 crct10dif_ce snd_hda_codec snd_hda_core tegra_xudc host1x lp855x_bl at24 ip_tables x_tables ipv6
> [  153.566417] CPU: 0 PID: 300 Comm: systemd-sleep Tainted: G         C        5.10.0-rc3-next-20201113-00019-g5c064d5372b0-dirty #624
> [  153.578283] Hardware name: NVIDIA Jetson TX1 Developer Kit (DT)
> [  153.584281] pstate: 40000005 (nZcv daif -PAN -UAO -TCO BTYPE=--)
> [  153.590381] pc : tegra_xusb_padctl_suspend_noirq+0x88/0x100
> [  153.596016] lr : tegra_xusb_padctl_suspend_noirq+0x80/0x100
> [  153.601632] sp : ffff8000120dbb60
> [  153.604999] x29: ffff8000120dbb60 x28: ffff000080a1df00
> [  153.610430] x27: 0000000000000002 x26: ffff8000106f8540
> [  153.615858] x25: ffff8000113ac4a4 x24: ffff80001148c198
> [  153.621277] x23: ffff800010c4538c x22: 0000000000000002
> [  153.626692] x21: ffff800010ccde80 x20: ffff0000829f6480
> [  153.632107] x19: ffff000080917000 x18: 0000000000000030
> [  153.637521] x17: 0000000000000000 x16: 0000000000000000
> [  153.642933] x15: ffff000080a1e380 x14: 74636461702d6273
> [  153.648346] x13: ffff8000113ad058 x12: 0000000000000f39
> [  153.653759] x11: 0000000000000513 x10: ffff800011405058
> [  153.659176] x9 : 00000000fffff000 x8 : ffff8000113ad058
> [  153.664590] x7 : ffff800011405058 x6 : 0000000000000000
> [  153.670002] x5 : 0000000000000000 x4 : ffff0000fe908bc0
> [  153.675414] x3 : ffff0000fe910228 x2 : 162ef67e0581e700
> [  153.680826] x1 : 162ef67e0581e700 x0 : ef7bdd7fffffffff
> [  153.686241] Call trace:
> [  153.688769]  tegra_xusb_padctl_suspend_noirq+0x88/0x100
> [  153.694077]  __device_suspend_noirq+0x68/0x1cc
> [  153.698594]  dpm_noirq_suspend_devices+0x10c/0x1d0
> [  153.703456]  dpm_suspend_noirq+0x28/0xa0
> [  153.707461]  suspend_devices_and_enter+0x234/0x4bc
> [  153.712314]  pm_suspend+0x1e4/0x270
> [  153.715868]  state_store+0x8c/0x110
> [  153.719440]  kobj_attr_store+0x1c/0x30
> [  153.723259]  sysfs_kf_write+0x4c/0x7c
> [  153.726981]  kernfs_fop_write+0x124/0x240
> [  153.731065]  vfs_write+0xe4/0x204
> [  153.734449]  ksys_write+0x6c/0x100
> [  153.737925]  __arm64_sys_write+0x20/0x30
> [  153.741931]  el0_svc_common.constprop.0+0x78/0x1a0
> [  153.746789]  do_el0_svc+0x24/0x90
> [  153.750181]  el0_sync_handler+0x254/0x260
> [  153.754251]  el0_sync+0x174/0x180
> [  153.757663] Code: aa0303e2 94000f64 f9405680 b40000e0 (f9402803)
> [  153.763826] ---[ end trace 81543a3394cb409d ]---
> 
> Note that I've added a bit of debug information there to show what's
> going on. See how tegra_xusb_padctl_suspend_noirq() is being called for
> the phy-usb2.0 device? That's one of the PHYs that's being created for
> the USB2 lanes. Sometimes I do see that padctl->soc ends up being NULL
> for that device and in that case the function just aborts early and then
> tegra_xusb_padctl_suspend_noirq() will get called again for the padctl
> device and succeed.
> 
> I can't explain what's happening here. tegra_xusb_padctl_driver never
> binds to the phy-usb2.0 device, so I don't understand how it could end
> up suspending the device with that set of dev_pm_ops. Perhaps this is
> some weird type of corruption somewhere?
> 
> Thierry
> 

Hi Thierry,
I figured out that phy-usb2.0 device is bound to tegra-xusb-padctl driver by
those two lines in tegra_xusb_setup_usb_role_switch()

	port->usb_phy.dev = &lane->pad->lanes[port->index]->dev;
	port->usb_phy.dev->driver = port->padctl->dev->driver;

Instead of assigning lane (phy) device to 'struct usb_phy', I think it should be
the port device (port->dev) since the 'usb-role-switch' capability belongs to
port device.

	port->usb_phy.dev = &port->dev;

I will submit a patch and ask for your review.

Thanks,
JC