[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Fabio Estevam]

From: Fabio Estevam <fabio.estevam@freescale.com>

When the mx6 PCI conctroller is initialized in the bootloader we see a kernel 
hang inside imx6_add_pcie_port().

Adding a 30ms delay allows the kernel to boot.

Suggested-by: David Müller <d.mueller@elsoft.ch>
Signed-off-by: Fabio Estevam <fabio.estevam@freescale.com>
---
I am happy to get feedback on how to properly fix this.

Thanks

 drivers/pci/host/pci-imx6.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c
index a568efa..669f771 100644
--- a/drivers/pci/host/pci-imx6.c
+++ b/drivers/pci/host/pci-imx6.c
@@ -507,6 +507,8 @@ static int __init imx6_add_pcie_port(struct pcie_port *pp,
 	pp->root_bus_nr = -1;
 	pp->ops = &imx6_pcie_host_ops;
 
+	usleep_range(25000, 30000);
+
 	ret = dw_pcie_host_init(pp);
 	if (ret) {
 		dev_err(&pdev->dev, "failed to initialize host\n");
-- 
1.8.3.2

[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Fabio Estevam]

From: Fabio Estevam <fabio.estevam@freescale.com>

When the mx6 PCI conctroller is initialized in the bootloader we see a kernel 
hang inside imx6_add_pcie_port().

Adding a 30ms delay allows the kernel to boot.

Suggested-by: David M?ller <d.mueller@elsoft.ch>
Signed-off-by: Fabio Estevam <fabio.estevam@freescale.com>
---
I am happy to get feedback on how to properly fix this.

Thanks

 drivers/pci/host/pci-imx6.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c
index a568efa..669f771 100644
--- a/drivers/pci/host/pci-imx6.c
+++ b/drivers/pci/host/pci-imx6.c
@@ -507,6 +507,8 @@ static int __init imx6_add_pcie_port(struct pcie_port *pp,
 	pp->root_bus_nr = -1;
 	pp->ops = &imx6_pcie_host_ops;
 
+	usleep_range(25000, 30000);
+
 	ret = dw_pcie_host_init(pp);
 	if (ret) {
 		dev_err(&pdev->dev, "failed to initialize host\n");
-- 
1.8.3.2

Re: [RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Marek Vasut]

On Tuesday, June 24, 2014 at 09:18:27 PM, Fabio Estevam wrote:
> From: Fabio Estevam <fabio.estevam@freescale.com>
> 
> When the mx6 PCI conctroller is initialized in the bootloader we see a
> kernel hang inside imx6_add_pcie_port().
> 
> Adding a 30ms delay allows the kernel to boot.
> 
> Suggested-by: David Müller <d.mueller@elsoft.ch>
> Signed-off-by: Fabio Estevam <fabio.estevam@freescale.com>
> ---
> I am happy to get feedback on how to properly fix this.

Honestly, I have no clue. I would expect Freescale to help us in solving this as 
they are the chip vendor, but they cannot help either :-(

Apologies, I really have no idea .

Best regards,
Marek Vasut

[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Marek Vasut]

On Tuesday, June 24, 2014 at 09:18:27 PM, Fabio Estevam wrote:
> From: Fabio Estevam <fabio.estevam@freescale.com>
> 
> When the mx6 PCI conctroller is initialized in the bootloader we see a
> kernel hang inside imx6_add_pcie_port().
> 
> Adding a 30ms delay allows the kernel to boot.
> 
> Suggested-by: David M?ller <d.mueller@elsoft.ch>
> Signed-off-by: Fabio Estevam <fabio.estevam@freescale.com>
> ---
> I am happy to get feedback on how to properly fix this.

Honestly, I have no clue. I would expect Freescale to help us in solving this as 
they are the chip vendor, but they cannot help either :-(

Apologies, I really have no idea .

Best regards,
Marek Vasut

Re: [RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Shawn Guo]

On Tue, Jun 24, 2014 at 04:18:27PM -0300, Fabio Estevam wrote:
> From: Fabio Estevam <fabio.estevam@freescale.com>
> 
> When the mx6 PCI conctroller is initialized in the bootloader we see a kernel 
> hang inside imx6_add_pcie_port().
> 
> Adding a 30ms delay allows the kernel to boot.

We may not want to add a random delay into the driver before we
understand the root cause of the issue.

Do you see this issue with FSL kernel?

Shawn

> 
> Suggested-by: David Müller <d.mueller@elsoft.ch>
> Signed-off-by: Fabio Estevam <fabio.estevam@freescale.com>
> ---
> I am happy to get feedback on how to properly fix this.
> 
> Thanks
> 
>  drivers/pci/host/pci-imx6.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c
> index a568efa..669f771 100644
> --- a/drivers/pci/host/pci-imx6.c
> +++ b/drivers/pci/host/pci-imx6.c
> @@ -507,6 +507,8 @@ static int __init imx6_add_pcie_port(struct pcie_port *pp,
>  	pp->root_bus_nr = -1;
>  	pp->ops = &imx6_pcie_host_ops;
>  
> +	usleep_range(25000, 30000);
> +
>  	ret = dw_pcie_host_init(pp);
>  	if (ret) {
>  		dev_err(&pdev->dev, "failed to initialize host\n");
> -- 
> 1.8.3.2
> 

[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Shawn Guo]

On Tue, Jun 24, 2014 at 04:18:27PM -0300, Fabio Estevam wrote:
> From: Fabio Estevam <fabio.estevam@freescale.com>
> 
> When the mx6 PCI conctroller is initialized in the bootloader we see a kernel 
> hang inside imx6_add_pcie_port().
> 
> Adding a 30ms delay allows the kernel to boot.

We may not want to add a random delay into the driver before we
understand the root cause of the issue.

Do you see this issue with FSL kernel?

Shawn

> 
> Suggested-by: David M?ller <d.mueller@elsoft.ch>
> Signed-off-by: Fabio Estevam <fabio.estevam@freescale.com>
> ---
> I am happy to get feedback on how to properly fix this.
> 
> Thanks
> 
>  drivers/pci/host/pci-imx6.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c
> index a568efa..669f771 100644
> --- a/drivers/pci/host/pci-imx6.c
> +++ b/drivers/pci/host/pci-imx6.c
> @@ -507,6 +507,8 @@ static int __init imx6_add_pcie_port(struct pcie_port *pp,
>  	pp->root_bus_nr = -1;
>  	pp->ops = &imx6_pcie_host_ops;
>  
> +	usleep_range(25000, 30000);
> +
>  	ret = dw_pcie_host_init(pp);
>  	if (ret) {
>  		dev_err(&pdev->dev, "failed to initialize host\n");
> -- 
> 1.8.3.2
> 

Re: [RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Fabio Estevam]

On Thu, Jun 26, 2014 at 12:12 AM, Shawn Guo <shawn.guo@freescale.com> wrote:
> On Tue, Jun 24, 2014 at 04:18:27PM -0300, Fabio Estevam wrote:
>> From: Fabio Estevam <fabio.estevam@freescale.com>
>>
>> When the mx6 PCI conctroller is initialized in the bootloader we see a kernel
>> hang inside imx6_add_pcie_port().
>>
>> Adding a 30ms delay allows the kernel to boot.
>
> We may not want to add a random delay into the driver before we
> understand the root cause of the issue.

Yes, that's why I sent this as RFC and also explained it below the ---
line that I am actually trying to get some help with this issue.

>
> Do you see this issue with FSL kernel?

Yes, it also hangs.

It is reproducible in 100% of the boots. Just need to use mainline
U-boot (which has PCI driver enabled by default).
I am using an Intel Wifi 7260 PCI card. This was also reported by
other folks in the U-boot list.

Below is the log with linux-next kernel and earlyprintk enabled:

U-Boot 2014.07-rc1-15766-g5f46552 (Jun 24 2014 - 17:41:55)

CPU:   Freescale i.MX6Q rev1.2 at 792 MHz
Reset cause: POR
Board: MX6-SabreSD
I2C:   ready
DRAM:  1 GiB
MMC:   FSL_SDHC: 0, FSL_SDHC: 1, FSL_SDHC: 2
  00:01.0     - 16c3:abcd - Bridge device
   01:00.0    - 8086:08b1 - Network controller
No panel detected: default to Hannstar-XGA
Display: Hannstar-XGA (1024x768)
In:    serial
Out:   serial
Err:   serial
PMIC:  PFUZE100 ID=0x10
Net:   FEC [PRIME]
Hit any key to stop autoboot:  0
Booting from net ...
FEC Waiting for PHY auto negotiation to complete.. done
Using FEC device
TFTP from server 192.168.0.2; our IP address is 192.168.0.8
Filename 'zImage'.
Load address: 0x12000000
Loading: #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #############################################################
         1.7 MiB/s
done
Bytes transferred = 5302272 (50e800 hex)
Using FEC device
TFTP from server 192.168.0.2; our IP address is 192.168.0.8
Filename 'imx6q-sabresd.dtb'.
Load address: 0x18000000
Loading: #######
         1.5 MiB/s
done
Bytes transferred = 33744 (83d0 hex)
Kernel image @ 0x12000000 [ 0x000000 - 0x50e800 ]
## Flattened Device Tree blob at 18000000
   Booting using the fdt blob at 0x18000000
   Using Device Tree in place at 18000000, end 1800b3cf

Starting kernel ...

Uncompressing Linux... done, booting the kernel.
Booting Linux on physical CPU 0x0
Linux version 3.16.0-rc2-next-20140625 (fabio@fabio-Latitude-E6410)
(gcc version 4.7.3 (Ubuntu/Linaro 4.7.3-1ubuntu1) ) #1440 SMP Thu
Jun 26 00:38:33 BRT 2014
CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
Machine model: Freescale i.MX6 Quad SABRE Smart Device Board
bootconsole [earlycon0] enabled
cma: Reserved 16 MiB at 4f000000
Memory policy: Data cache writealloc
PERCPU: Embedded 8 pages/cpu @be7a6000 s8896 r8192 d15680 u32768
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 260096
Kernel command line: console=ttymxc0,115200 root=/dev/nfs ip=dhcp
nfsroot=192.168.0.2:/tftpboot/rfs,v3,tcp earlyprintk
PID hash table entries: 4096 (order: 2, 16384 bytes)
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 1004880K/1048576K available (6579K kernel code, 406K rwdata,
2264K rodata, 340K init, 8331K bss, 43696K reserved, 0K highmem)
Virtual kernel memory layout:
    vector  : 0xffff0000 - 0xffff1000   (   4 kB)
    fixmap  : 0xffc00000 - 0xffe00000   (2048 kB)
    vmalloc : 0xc0800000 - 0xff000000   (1000 MB)
    lowmem  : 0x80000000 - 0xc0000000   (1024 MB)
    pkmap   : 0x7fe00000 - 0x80000000   (   2 MB)
    modules : 0x7f000000 - 0x7fe00000   (  14 MB)
      .text : 0x80008000 - 0x808aafbc   (8844 kB)
      .init : 0x808ab000 - 0x809002c0   ( 341 kB)
      .data : 0x80902000 - 0x80967a40   ( 407 kB)
       .bss : 0x80967a48 - 0x8118aa10   (8332 kB)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
Hierarchical RCU implementation.
NR_IRQS:16 nr_irqs:16 16
L2C-310 erratum 769419 enabled
L2C-310 enabling early BRESP for Cortex-A9
L2C-310 full line of zeros enabled for Cortex-A9
L2C-310 ID prefetch enabled, offset 1 lines
L2C-310 dynamic clock gating enabled, standby mode enabled
L2C-310 cache controller enabled, 16 ways, 1024 kB
L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x76070001
Switching to timer-based delay loop, resolution 15ns
sched_clock: 32 bits at 66MHz, resolution 15ns, wraps every 65075262448ns
Console: colour dummy device 80x30
Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
... MAX_LOCKDEP_SUBCLASSES:  8
... MAX_LOCK_DEPTH:          48
... MAX_LOCKDEP_KEYS:        8191
... CLASSHASH_SIZE:          4096
... MAX_LOCKDEP_ENTRIES:     32768
... MAX_LOCKDEP_CHAINS:      65536
... CHAINHASH_SIZE:          32768
 memory used by lock dependency info: 5167 kB
 per task-struct memory footprint: 1152 bytes
Calibrating delay loop (skipped), value calculated using timer
frequency.. 132.00 BogoMIPS (lpj=660000)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
CPU: Testing write buffer coherency: ok
CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
Setting up static identity map for 0x10672328 - 0x10672398
CPU1: Booted secondary processor
CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
CPU2: Booted secondary processor
CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
CPU3: Booted secondary processor
CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
Brought up 4 CPUs
SMP: Total of 4 processors activated.
CPU: All CPU(s) started in SVC mode.
devtmpfs: initialized
VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
pinctrl core: initialized pinctrl subsystem
regulator-dummy: no parameters
NET: Registered protocol family 16
DMA: preallocated 256 KiB pool for atomic coherent allocations
CPU identified as i.MX6Q, silicon rev 1.2
vdd1p1: 800 <--> 1375 mV at 1100 mV
vdd3p0: 2800 <--> 3150 mV at 3000 mV
vdd2p5: 2000 <--> 2750 mV at 2400 mV
vddarm: 725 <--> 1450 mV at 1100 mV
vddpu: 725 <--> 1450 mV at 1100 mV
vddsoc: 725 <--> 1450 mV at 1175 mV
hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
hw-breakpoint: maximum watchpoint size is 4 bytes.
imx6q-pinctrl 20e0000.iomuxc: initialized IMX pinctrl driver
mxs-dma 110000.dma-apbh: initialized
usb_otg_vbus: 5000 mV
usb_h1_vbus: 5000 mV
wm8962-supply: no parameters
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
i2c i2c-0: IMX I2C adapter registered
i2c i2c-1: IMX I2C adapter registered
i2c i2c-2: IMX I2C adapter registered
Linux video capture interface: v2.00
pps_core: LinuxPPS API ver. 1 registered
pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti
<giometti@linux.it>
PTP clock support registered
Advanced Linux Sound Architecture Driver Initialized.
cfg80211: Calling CRDA to update world regulatory domain
Switched to clocksource mxc_timer1

[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Fabio Estevam]

On Thu, Jun 26, 2014 at 12:12 AM, Shawn Guo <shawn.guo@freescale.com> wrote:
> On Tue, Jun 24, 2014 at 04:18:27PM -0300, Fabio Estevam wrote:
>> From: Fabio Estevam <fabio.estevam@freescale.com>
>>
>> When the mx6 PCI conctroller is initialized in the bootloader we see a kernel
>> hang inside imx6_add_pcie_port().
>>
>> Adding a 30ms delay allows the kernel to boot.
>
> We may not want to add a random delay into the driver before we
> understand the root cause of the issue.

Yes, that's why I sent this as RFC and also explained it below the ---
line that I am actually trying to get some help with this issue.

>
> Do you see this issue with FSL kernel?

Yes, it also hangs.

It is reproducible in 100% of the boots. Just need to use mainline
U-boot (which has PCI driver enabled by default).
I am using an Intel Wifi 7260 PCI card. This was also reported by
other folks in the U-boot list.

Below is the log with linux-next kernel and earlyprintk enabled:

U-Boot 2014.07-rc1-15766-g5f46552 (Jun 24 2014 - 17:41:55)

CPU:   Freescale i.MX6Q rev1.2 at 792 MHz
Reset cause: POR
Board: MX6-SabreSD
I2C:   ready
DRAM:  1 GiB
MMC:   FSL_SDHC: 0, FSL_SDHC: 1, FSL_SDHC: 2
  00:01.0     - 16c3:abcd - Bridge device
   01:00.0    - 8086:08b1 - Network controller
No panel detected: default to Hannstar-XGA
Display: Hannstar-XGA (1024x768)
In:    serial
Out:   serial
Err:   serial
PMIC:  PFUZE100 ID=0x10
Net:   FEC [PRIME]
Hit any key to stop autoboot:  0
Booting from net ...
FEC Waiting for PHY auto negotiation to complete.. done
Using FEC device
TFTP from server 192.168.0.2; our IP address is 192.168.0.8
Filename 'zImage'.
Load address: 0x12000000
Loading: #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #############################################################
         1.7 MiB/s
done
Bytes transferred = 5302272 (50e800 hex)
Using FEC device
TFTP from server 192.168.0.2; our IP address is 192.168.0.8
Filename 'imx6q-sabresd.dtb'.
Load address: 0x18000000
Loading: #######
         1.5 MiB/s
done
Bytes transferred = 33744 (83d0 hex)
Kernel image @ 0x12000000 [ 0x000000 - 0x50e800 ]
## Flattened Device Tree blob at 18000000
   Booting using the fdt blob at 0x18000000
   Using Device Tree in place at 18000000, end 1800b3cf

Starting kernel ...

Uncompressing Linux... done, booting the kernel.
Booting Linux on physical CPU 0x0
Linux version 3.16.0-rc2-next-20140625 (fabio at fabio-Latitude-E6410)
(gcc version 4.7.3 (Ubuntu/Linaro 4.7.3-1ubuntu1) ) #1440 SMP Thu
Jun 26 00:38:33 BRT 2014
CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
Machine model: Freescale i.MX6 Quad SABRE Smart Device Board
bootconsole [earlycon0] enabled
cma: Reserved 16 MiB at 4f000000
Memory policy: Data cache writealloc
PERCPU: Embedded 8 pages/cpu @be7a6000 s8896 r8192 d15680 u32768
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 260096
Kernel command line: console=ttymxc0,115200 root=/dev/nfs ip=dhcp
nfsroot=192.168.0.2:/tftpboot/rfs,v3,tcp earlyprintk
PID hash table entries: 4096 (order: 2, 16384 bytes)
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 1004880K/1048576K available (6579K kernel code, 406K rwdata,
2264K rodata, 340K init, 8331K bss, 43696K reserved, 0K highmem)
Virtual kernel memory layout:
    vector  : 0xffff0000 - 0xffff1000   (   4 kB)
    fixmap  : 0xffc00000 - 0xffe00000   (2048 kB)
    vmalloc : 0xc0800000 - 0xff000000   (1000 MB)
    lowmem  : 0x80000000 - 0xc0000000   (1024 MB)
    pkmap   : 0x7fe00000 - 0x80000000   (   2 MB)
    modules : 0x7f000000 - 0x7fe00000   (  14 MB)
      .text : 0x80008000 - 0x808aafbc   (8844 kB)
      .init : 0x808ab000 - 0x809002c0   ( 341 kB)
      .data : 0x80902000 - 0x80967a40   ( 407 kB)
       .bss : 0x80967a48 - 0x8118aa10   (8332 kB)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
Hierarchical RCU implementation.
NR_IRQS:16 nr_irqs:16 16
L2C-310 erratum 769419 enabled
L2C-310 enabling early BRESP for Cortex-A9
L2C-310 full line of zeros enabled for Cortex-A9
L2C-310 ID prefetch enabled, offset 1 lines
L2C-310 dynamic clock gating enabled, standby mode enabled
L2C-310 cache controller enabled, 16 ways, 1024 kB
L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x76070001
Switching to timer-based delay loop, resolution 15ns
sched_clock: 32 bits at 66MHz, resolution 15ns, wraps every 65075262448ns
Console: colour dummy device 80x30
Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
... MAX_LOCKDEP_SUBCLASSES:  8
... MAX_LOCK_DEPTH:          48
... MAX_LOCKDEP_KEYS:        8191
... CLASSHASH_SIZE:          4096
... MAX_LOCKDEP_ENTRIES:     32768
... MAX_LOCKDEP_CHAINS:      65536
... CHAINHASH_SIZE:          32768
 memory used by lock dependency info: 5167 kB
 per task-struct memory footprint: 1152 bytes
Calibrating delay loop (skipped), value calculated using timer
frequency.. 132.00 BogoMIPS (lpj=660000)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
CPU: Testing write buffer coherency: ok
CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
Setting up static identity map for 0x10672328 - 0x10672398
CPU1: Booted secondary processor
CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
CPU2: Booted secondary processor
CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
CPU3: Booted secondary processor
CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
Brought up 4 CPUs
SMP: Total of 4 processors activated.
CPU: All CPU(s) started in SVC mode.
devtmpfs: initialized
VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
pinctrl core: initialized pinctrl subsystem
regulator-dummy: no parameters
NET: Registered protocol family 16
DMA: preallocated 256 KiB pool for atomic coherent allocations
CPU identified as i.MX6Q, silicon rev 1.2
vdd1p1: 800 <--> 1375 mV at 1100 mV
vdd3p0: 2800 <--> 3150 mV at 3000 mV
vdd2p5: 2000 <--> 2750 mV at 2400 mV
vddarm: 725 <--> 1450 mV at 1100 mV
vddpu: 725 <--> 1450 mV at 1100 mV
vddsoc: 725 <--> 1450 mV at 1175 mV
hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
hw-breakpoint: maximum watchpoint size is 4 bytes.
imx6q-pinctrl 20e0000.iomuxc: initialized IMX pinctrl driver
mxs-dma 110000.dma-apbh: initialized
usb_otg_vbus: 5000 mV
usb_h1_vbus: 5000 mV
wm8962-supply: no parameters
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
i2c i2c-0: IMX I2C adapter registered
i2c i2c-1: IMX I2C adapter registered
i2c i2c-2: IMX I2C adapter registered
Linux video capture interface: v2.00
pps_core: LinuxPPS API ver. 1 registered
pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti
<giometti@linux.it>
PTP clock support registered
Advanced Linux Sound Architecture Driver Initialized.
cfg80211: Calling CRDA to update world regulatory domain
Switched to clocksource mxc_timer1

Re: [RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Shawn Guo]

On Thu, Jun 26, 2014 at 12:43:19AM -0300, Fabio Estevam wrote:
> On Thu, Jun 26, 2014 at 12:12 AM, Shawn Guo <shawn.guo@freescale.com> wrote:
> > On Tue, Jun 24, 2014 at 04:18:27PM -0300, Fabio Estevam wrote:
> >> From: Fabio Estevam <fabio.estevam@freescale.com>
> >>
> >> When the mx6 PCI conctroller is initialized in the bootloader we see a kernel
> >> hang inside imx6_add_pcie_port().
> >>
> >> Adding a 30ms delay allows the kernel to boot.
> >
> > We may not want to add a random delay into the driver before we
> > understand the root cause of the issue.
> 
> Yes, that's why I sent this as RFC and also explained it below the ---
> line that I am actually trying to get some help with this issue.
> 
> >
> > Do you see this issue with FSL kernel?
> 
> Yes, it also hangs.
> 
> It is reproducible in 100% of the boots. Just need to use mainline
> U-boot (which has PCI driver enabled by default).
> I am using an Intel Wifi 7260 PCI card. This was also reported by
> other folks in the U-boot list.

Richard,

Can you schedule some time to look at this issue?  I think it will come
to us sooner or later if any our customer enables PCIe before launching
kernel?

Shawn

> 
> Below is the log with linux-next kernel and earlyprintk enabled:
> 
> U-Boot 2014.07-rc1-15766-g5f46552 (Jun 24 2014 - 17:41:55)
> 
> CPU:   Freescale i.MX6Q rev1.2 at 792 MHz
> Reset cause: POR
> Board: MX6-SabreSD
> I2C:   ready
> DRAM:  1 GiB
> MMC:   FSL_SDHC: 0, FSL_SDHC: 1, FSL_SDHC: 2
>   00:01.0     - 16c3:abcd - Bridge device
>    01:00.0    - 8086:08b1 - Network controller
> No panel detected: default to Hannstar-XGA
> Display: Hannstar-XGA (1024x768)
> In:    serial
> Out:   serial
> Err:   serial
> PMIC:  PFUZE100 ID=0x10
> Net:   FEC [PRIME]
> Hit any key to stop autoboot:  0
> Booting from net ...
> FEC Waiting for PHY auto negotiation to complete.. done
> Using FEC device
> TFTP from server 192.168.0.2; our IP address is 192.168.0.8
> Filename 'zImage'.
> Load address: 0x12000000
> Loading: #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #############################################################
>          1.7 MiB/s
> done
> Bytes transferred = 5302272 (50e800 hex)
> Using FEC device
> TFTP from server 192.168.0.2; our IP address is 192.168.0.8
> Filename 'imx6q-sabresd.dtb'.
> Load address: 0x18000000
> Loading: #######
>          1.5 MiB/s
> done
> Bytes transferred = 33744 (83d0 hex)
> Kernel image @ 0x12000000 [ 0x000000 - 0x50e800 ]
> ## Flattened Device Tree blob at 18000000
>    Booting using the fdt blob at 0x18000000
>    Using Device Tree in place at 18000000, end 1800b3cf
> 
> Starting kernel ...
> 
> Uncompressing Linux... done, booting the kernel.
> Booting Linux on physical CPU 0x0
> Linux version 3.16.0-rc2-next-20140625 (fabio@fabio-Latitude-E6410)
> (gcc version 4.7.3 (Ubuntu/Linaro 4.7.3-1ubuntu1) ) #1440 SMP Thu
> Jun 26 00:38:33 BRT 2014
> CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
> CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
> Machine model: Freescale i.MX6 Quad SABRE Smart Device Board
> bootconsole [earlycon0] enabled
> cma: Reserved 16 MiB at 4f000000
> Memory policy: Data cache writealloc
> PERCPU: Embedded 8 pages/cpu @be7a6000 s8896 r8192 d15680 u32768
> Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 260096
> Kernel command line: console=ttymxc0,115200 root=/dev/nfs ip=dhcp
> nfsroot=192.168.0.2:/tftpboot/rfs,v3,tcp earlyprintk
> PID hash table entries: 4096 (order: 2, 16384 bytes)
> Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
> Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
> Memory: 1004880K/1048576K available (6579K kernel code, 406K rwdata,
> 2264K rodata, 340K init, 8331K bss, 43696K reserved, 0K highmem)
> Virtual kernel memory layout:
>     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
>     fixmap  : 0xffc00000 - 0xffe00000   (2048 kB)
>     vmalloc : 0xc0800000 - 0xff000000   (1000 MB)
>     lowmem  : 0x80000000 - 0xc0000000   (1024 MB)
>     pkmap   : 0x7fe00000 - 0x80000000   (   2 MB)
>     modules : 0x7f000000 - 0x7fe00000   (  14 MB)
>       .text : 0x80008000 - 0x808aafbc   (8844 kB)
>       .init : 0x808ab000 - 0x809002c0   ( 341 kB)
>       .data : 0x80902000 - 0x80967a40   ( 407 kB)
>        .bss : 0x80967a48 - 0x8118aa10   (8332 kB)
> SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
> Hierarchical RCU implementation.
> NR_IRQS:16 nr_irqs:16 16
> L2C-310 erratum 769419 enabled
> L2C-310 enabling early BRESP for Cortex-A9
> L2C-310 full line of zeros enabled for Cortex-A9
> L2C-310 ID prefetch enabled, offset 1 lines
> L2C-310 dynamic clock gating enabled, standby mode enabled
> L2C-310 cache controller enabled, 16 ways, 1024 kB
> L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x76070001
> Switching to timer-based delay loop, resolution 15ns
> sched_clock: 32 bits at 66MHz, resolution 15ns, wraps every 65075262448ns
> Console: colour dummy device 80x30
> Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
> ... MAX_LOCKDEP_SUBCLASSES:  8
> ... MAX_LOCK_DEPTH:          48
> ... MAX_LOCKDEP_KEYS:        8191
> ... CLASSHASH_SIZE:          4096
> ... MAX_LOCKDEP_ENTRIES:     32768
> ... MAX_LOCKDEP_CHAINS:      65536
> ... CHAINHASH_SIZE:          32768
>  memory used by lock dependency info: 5167 kB
>  per task-struct memory footprint: 1152 bytes
> Calibrating delay loop (skipped), value calculated using timer
> frequency.. 132.00 BogoMIPS (lpj=660000)
> pid_max: default: 32768 minimum: 301
> Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
> Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
> CPU: Testing write buffer coherency: ok
> CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
> Setting up static identity map for 0x10672328 - 0x10672398
> CPU1: Booted secondary processor
> CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
> CPU2: Booted secondary processor
> CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
> CPU3: Booted secondary processor
> CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
> Brought up 4 CPUs
> SMP: Total of 4 processors activated.
> CPU: All CPU(s) started in SVC mode.
> devtmpfs: initialized
> VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
> pinctrl core: initialized pinctrl subsystem
> regulator-dummy: no parameters
> NET: Registered protocol family 16
> DMA: preallocated 256 KiB pool for atomic coherent allocations
> CPU identified as i.MX6Q, silicon rev 1.2
> vdd1p1: 800 <--> 1375 mV at 1100 mV
> vdd3p0: 2800 <--> 3150 mV at 3000 mV
> vdd2p5: 2000 <--> 2750 mV at 2400 mV
> vddarm: 725 <--> 1450 mV at 1100 mV
> vddpu: 725 <--> 1450 mV at 1100 mV
> vddsoc: 725 <--> 1450 mV at 1175 mV
> hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
> hw-breakpoint: maximum watchpoint size is 4 bytes.
> imx6q-pinctrl 20e0000.iomuxc: initialized IMX pinctrl driver
> mxs-dma 110000.dma-apbh: initialized
> usb_otg_vbus: 5000 mV
> usb_h1_vbus: 5000 mV
> wm8962-supply: no parameters
> vgaarb: loaded
> SCSI subsystem initialized
> usbcore: registered new interface driver usbfs
> usbcore: registered new interface driver hub
> usbcore: registered new device driver usb
> i2c i2c-0: IMX I2C adapter registered
> i2c i2c-1: IMX I2C adapter registered
> i2c i2c-2: IMX I2C adapter registered
> Linux video capture interface: v2.00
> pps_core: LinuxPPS API ver. 1 registered
> pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti
> <giometti@linux.it>
> PTP clock support registered
> Advanced Linux Sound Architecture Driver Initialized.
> cfg80211: Calling CRDA to update world regulatory domain
> Switched to clocksource mxc_timer1

[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Shawn Guo]

On Thu, Jun 26, 2014 at 12:43:19AM -0300, Fabio Estevam wrote:
> On Thu, Jun 26, 2014 at 12:12 AM, Shawn Guo <shawn.guo@freescale.com> wrote:
> > On Tue, Jun 24, 2014 at 04:18:27PM -0300, Fabio Estevam wrote:
> >> From: Fabio Estevam <fabio.estevam@freescale.com>
> >>
> >> When the mx6 PCI conctroller is initialized in the bootloader we see a kernel
> >> hang inside imx6_add_pcie_port().
> >>
> >> Adding a 30ms delay allows the kernel to boot.
> >
> > We may not want to add a random delay into the driver before we
> > understand the root cause of the issue.
> 
> Yes, that's why I sent this as RFC and also explained it below the ---
> line that I am actually trying to get some help with this issue.
> 
> >
> > Do you see this issue with FSL kernel?
> 
> Yes, it also hangs.
> 
> It is reproducible in 100% of the boots. Just need to use mainline
> U-boot (which has PCI driver enabled by default).
> I am using an Intel Wifi 7260 PCI card. This was also reported by
> other folks in the U-boot list.

Richard,

Can you schedule some time to look at this issue?  I think it will come
to us sooner or later if any our customer enables PCIe before launching
kernel?

Shawn

> 
> Below is the log with linux-next kernel and earlyprintk enabled:
> 
> U-Boot 2014.07-rc1-15766-g5f46552 (Jun 24 2014 - 17:41:55)
> 
> CPU:   Freescale i.MX6Q rev1.2 at 792 MHz
> Reset cause: POR
> Board: MX6-SabreSD
> I2C:   ready
> DRAM:  1 GiB
> MMC:   FSL_SDHC: 0, FSL_SDHC: 1, FSL_SDHC: 2
>   00:01.0     - 16c3:abcd - Bridge device
>    01:00.0    - 8086:08b1 - Network controller
> No panel detected: default to Hannstar-XGA
> Display: Hannstar-XGA (1024x768)
> In:    serial
> Out:   serial
> Err:   serial
> PMIC:  PFUZE100 ID=0x10
> Net:   FEC [PRIME]
> Hit any key to stop autoboot:  0
> Booting from net ...
> FEC Waiting for PHY auto negotiation to complete.. done
> Using FEC device
> TFTP from server 192.168.0.2; our IP address is 192.168.0.8
> Filename 'zImage'.
> Load address: 0x12000000
> Loading: #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #############################################################
>          1.7 MiB/s
> done
> Bytes transferred = 5302272 (50e800 hex)
> Using FEC device
> TFTP from server 192.168.0.2; our IP address is 192.168.0.8
> Filename 'imx6q-sabresd.dtb'.
> Load address: 0x18000000
> Loading: #######
>          1.5 MiB/s
> done
> Bytes transferred = 33744 (83d0 hex)
> Kernel image @ 0x12000000 [ 0x000000 - 0x50e800 ]
> ## Flattened Device Tree blob at 18000000
>    Booting using the fdt blob at 0x18000000
>    Using Device Tree in place at 18000000, end 1800b3cf
> 
> Starting kernel ...
> 
> Uncompressing Linux... done, booting the kernel.
> Booting Linux on physical CPU 0x0
> Linux version 3.16.0-rc2-next-20140625 (fabio at fabio-Latitude-E6410)
> (gcc version 4.7.3 (Ubuntu/Linaro 4.7.3-1ubuntu1) ) #1440 SMP Thu
> Jun 26 00:38:33 BRT 2014
> CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
> CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
> Machine model: Freescale i.MX6 Quad SABRE Smart Device Board
> bootconsole [earlycon0] enabled
> cma: Reserved 16 MiB at 4f000000
> Memory policy: Data cache writealloc
> PERCPU: Embedded 8 pages/cpu @be7a6000 s8896 r8192 d15680 u32768
> Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 260096
> Kernel command line: console=ttymxc0,115200 root=/dev/nfs ip=dhcp
> nfsroot=192.168.0.2:/tftpboot/rfs,v3,tcp earlyprintk
> PID hash table entries: 4096 (order: 2, 16384 bytes)
> Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
> Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
> Memory: 1004880K/1048576K available (6579K kernel code, 406K rwdata,
> 2264K rodata, 340K init, 8331K bss, 43696K reserved, 0K highmem)
> Virtual kernel memory layout:
>     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
>     fixmap  : 0xffc00000 - 0xffe00000   (2048 kB)
>     vmalloc : 0xc0800000 - 0xff000000   (1000 MB)
>     lowmem  : 0x80000000 - 0xc0000000   (1024 MB)
>     pkmap   : 0x7fe00000 - 0x80000000   (   2 MB)
>     modules : 0x7f000000 - 0x7fe00000   (  14 MB)
>       .text : 0x80008000 - 0x808aafbc   (8844 kB)
>       .init : 0x808ab000 - 0x809002c0   ( 341 kB)
>       .data : 0x80902000 - 0x80967a40   ( 407 kB)
>        .bss : 0x80967a48 - 0x8118aa10   (8332 kB)
> SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
> Hierarchical RCU implementation.
> NR_IRQS:16 nr_irqs:16 16
> L2C-310 erratum 769419 enabled
> L2C-310 enabling early BRESP for Cortex-A9
> L2C-310 full line of zeros enabled for Cortex-A9
> L2C-310 ID prefetch enabled, offset 1 lines
> L2C-310 dynamic clock gating enabled, standby mode enabled
> L2C-310 cache controller enabled, 16 ways, 1024 kB
> L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x76070001
> Switching to timer-based delay loop, resolution 15ns
> sched_clock: 32 bits at 66MHz, resolution 15ns, wraps every 65075262448ns
> Console: colour dummy device 80x30
> Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
> ... MAX_LOCKDEP_SUBCLASSES:  8
> ... MAX_LOCK_DEPTH:          48
> ... MAX_LOCKDEP_KEYS:        8191
> ... CLASSHASH_SIZE:          4096
> ... MAX_LOCKDEP_ENTRIES:     32768
> ... MAX_LOCKDEP_CHAINS:      65536
> ... CHAINHASH_SIZE:          32768
>  memory used by lock dependency info: 5167 kB
>  per task-struct memory footprint: 1152 bytes
> Calibrating delay loop (skipped), value calculated using timer
> frequency.. 132.00 BogoMIPS (lpj=660000)
> pid_max: default: 32768 minimum: 301
> Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
> Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
> CPU: Testing write buffer coherency: ok
> CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
> Setting up static identity map for 0x10672328 - 0x10672398
> CPU1: Booted secondary processor
> CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
> CPU2: Booted secondary processor
> CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
> CPU3: Booted secondary processor
> CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
> Brought up 4 CPUs
> SMP: Total of 4 processors activated.
> CPU: All CPU(s) started in SVC mode.
> devtmpfs: initialized
> VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
> pinctrl core: initialized pinctrl subsystem
> regulator-dummy: no parameters
> NET: Registered protocol family 16
> DMA: preallocated 256 KiB pool for atomic coherent allocations
> CPU identified as i.MX6Q, silicon rev 1.2
> vdd1p1: 800 <--> 1375 mV at 1100 mV
> vdd3p0: 2800 <--> 3150 mV at 3000 mV
> vdd2p5: 2000 <--> 2750 mV at 2400 mV
> vddarm: 725 <--> 1450 mV at 1100 mV
> vddpu: 725 <--> 1450 mV at 1100 mV
> vddsoc: 725 <--> 1450 mV at 1175 mV
> hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
> hw-breakpoint: maximum watchpoint size is 4 bytes.
> imx6q-pinctrl 20e0000.iomuxc: initialized IMX pinctrl driver
> mxs-dma 110000.dma-apbh: initialized
> usb_otg_vbus: 5000 mV
> usb_h1_vbus: 5000 mV
> wm8962-supply: no parameters
> vgaarb: loaded
> SCSI subsystem initialized
> usbcore: registered new interface driver usbfs
> usbcore: registered new interface driver hub
> usbcore: registered new device driver usb
> i2c i2c-0: IMX I2C adapter registered
> i2c i2c-1: IMX I2C adapter registered
> i2c i2c-2: IMX I2C adapter registered
> Linux video capture interface: v2.00
> pps_core: LinuxPPS API ver. 1 registered
> pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti
> <giometti@linux.it>
> PTP clock support registered
> Advanced Linux Sound Architecture Driver Initialized.
> cfg80211: Calling CRDA to update world regulatory domain
> Switched to clocksource mxc_timer1

Re: [RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Juergen Borleis]

Hi Fabio,

On Thursday 26 June 2014 05:43:19 Fabio Estevam wrote:
> [...]
> Below is the log with linux-next kernel and earlyprintk enabled:

Does it also hang when you disable "earlyprintk"? A few weeks ago we had a 
similar issue with i.MX6/PCI and "earlyprintk" (at the end it was caused by a 
clock change for the UART when changing from the early to the regular 
console).

Regards,
Juergen

-- 
Pengutronix e.K.                              | Juergen Borleis             |
Industrial Linux Solutions                    | http://www.pengutronix.de/  |

[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Juergen Borleis]

Hi Fabio,

On Thursday 26 June 2014 05:43:19 Fabio Estevam wrote:
> [...]
> Below is the log with linux-next kernel and earlyprintk enabled:

Does it also hang when you disable "earlyprintk"? A few weeks ago we had a 
similar issue with i.MX6/PCI and "earlyprintk" (at the end it was caused by a 
clock change for the UART when changing from the early to the regular 
console).

Regards,
Juergen

-- 
Pengutronix e.K. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?| Juergen Borleis ? ? ? ? ? ? |
Industrial Linux Solutions               ? ? ?| http://www.pengutronix.de/  |

Re: [RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Lucas Stach]

Hi Fabio,

Am Dienstag, den 24.06.2014, 16:18 -0300 schrieb Fabio Estevam:
> From: Fabio Estevam <fabio.estevam@freescale.com>
> 
> When the mx6 PCI conctroller is initialized in the bootloader we see a kernel 
> hang inside imx6_add_pcie_port().
> 
> Adding a 30ms delay allows the kernel to boot.
> 
> Suggested-by: David Müller <d.mueller@elsoft.ch>
> Signed-off-by: Fabio Estevam <fabio.estevam@freescale.com>
> ---
> I am happy to get feedback on how to properly fix this.
> 
> Thanks
> 
>  drivers/pci/host/pci-imx6.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c
> index a568efa..669f771 100644
> --- a/drivers/pci/host/pci-imx6.c
> +++ b/drivers/pci/host/pci-imx6.c
> @@ -507,6 +507,8 @@ static int __init imx6_add_pcie_port(struct pcie_port *pp,
>  	pp->root_bus_nr = -1;
>  	pp->ops = &imx6_pcie_host_ops;
>  
> +	usleep_range(25000, 30000);
> +
>  	ret = dw_pcie_host_init(pp);
>  	if (ret) {
>  		dev_err(&pdev->dev, "failed to initialize host\n");

I would suspect the issue to be somewhere in imx6_pcie_host_init(). Can
you move the delay there (and to different positions in this function)
to narrow down where the hang happens?

Regards,
Lucas
-- 
Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |

[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Lucas Stach]

Hi Fabio,

Am Dienstag, den 24.06.2014, 16:18 -0300 schrieb Fabio Estevam:
> From: Fabio Estevam <fabio.estevam@freescale.com>
> 
> When the mx6 PCI conctroller is initialized in the bootloader we see a kernel 
> hang inside imx6_add_pcie_port().
> 
> Adding a 30ms delay allows the kernel to boot.
> 
> Suggested-by: David M?ller <d.mueller@elsoft.ch>
> Signed-off-by: Fabio Estevam <fabio.estevam@freescale.com>
> ---
> I am happy to get feedback on how to properly fix this.
> 
> Thanks
> 
>  drivers/pci/host/pci-imx6.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c
> index a568efa..669f771 100644
> --- a/drivers/pci/host/pci-imx6.c
> +++ b/drivers/pci/host/pci-imx6.c
> @@ -507,6 +507,8 @@ static int __init imx6_add_pcie_port(struct pcie_port *pp,
>  	pp->root_bus_nr = -1;
>  	pp->ops = &imx6_pcie_host_ops;
>  
> +	usleep_range(25000, 30000);
> +
>  	ret = dw_pcie_host_init(pp);
>  	if (ret) {
>  		dev_err(&pdev->dev, "failed to initialize host\n");

I would suspect the issue to be somewhere in imx6_pcie_host_init(). Can
you move the delay there (and to different positions in this function)
to narrow down where the hang happens?

Regards,
Lucas
-- 
Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |

Re: [RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Marek Vasut]

On Thursday, June 26, 2014 at 07:49:38 AM, Shawn Guo wrote:
> On Thu, Jun 26, 2014 at 12:43:19AM -0300, Fabio Estevam wrote:
> > On Thu, Jun 26, 2014 at 12:12 AM, Shawn Guo <shawn.guo@freescale.com> wrote:
> > > On Tue, Jun 24, 2014 at 04:18:27PM -0300, Fabio Estevam wrote:
> > >> From: Fabio Estevam <fabio.estevam@freescale.com>
> > >> 
> > >> When the mx6 PCI conctroller is initialized in the bootloader we see a
> > >> kernel hang inside imx6_add_pcie_port().
> > >> 
> > >> Adding a 30ms delay allows the kernel to boot.
> > > 
> > > We may not want to add a random delay into the driver before we
> > > understand the root cause of the issue.
> > 
> > Yes, that's why I sent this as RFC and also explained it below the ---
> > line that I am actually trying to get some help with this issue.
> > 
> > > Do you see this issue with FSL kernel?
> > 
> > Yes, it also hangs.
> > 
> > It is reproducible in 100% of the boots. Just need to use mainline
> > U-boot (which has PCI driver enabled by default).
> > I am using an Intel Wifi 7260 PCI card. This was also reported by
> > other folks in the U-boot list.
> 
> Richard,
> 
> Can you schedule some time to look at this issue?  I think it will come
> to us sooner or later if any our customer enables PCIe before launching
> kernel?

I have this problem on MX6DL SabreSDP as well. Apparently, this issue happens 
more often on 6DL than it happens on 6Q. I'm clueless here.

I presume FSL won't be willing to release how the PCIe block is exactly wired 
into the GPR registers, right ? Also, I don't think there's some magic register 
which allows controlling the PCIe core and PCIe PHY reset lines directly (like 
on exynos), or is there please?

Best regards,
Marek Vasut

[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Marek Vasut]

On Thursday, June 26, 2014 at 07:49:38 AM, Shawn Guo wrote:
> On Thu, Jun 26, 2014 at 12:43:19AM -0300, Fabio Estevam wrote:
> > On Thu, Jun 26, 2014 at 12:12 AM, Shawn Guo <shawn.guo@freescale.com> wrote:
> > > On Tue, Jun 24, 2014 at 04:18:27PM -0300, Fabio Estevam wrote:
> > >> From: Fabio Estevam <fabio.estevam@freescale.com>
> > >> 
> > >> When the mx6 PCI conctroller is initialized in the bootloader we see a
> > >> kernel hang inside imx6_add_pcie_port().
> > >> 
> > >> Adding a 30ms delay allows the kernel to boot.
> > > 
> > > We may not want to add a random delay into the driver before we
> > > understand the root cause of the issue.
> > 
> > Yes, that's why I sent this as RFC and also explained it below the ---
> > line that I am actually trying to get some help with this issue.
> > 
> > > Do you see this issue with FSL kernel?
> > 
> > Yes, it also hangs.
> > 
> > It is reproducible in 100% of the boots. Just need to use mainline
> > U-boot (which has PCI driver enabled by default).
> > I am using an Intel Wifi 7260 PCI card. This was also reported by
> > other folks in the U-boot list.
> 
> Richard,
> 
> Can you schedule some time to look at this issue?  I think it will come
> to us sooner or later if any our customer enables PCIe before launching
> kernel?

I have this problem on MX6DL SabreSDP as well. Apparently, this issue happens 
more often on 6DL than it happens on 6Q. I'm clueless here.

I presume FSL won't be willing to release how the PCIe block is exactly wired 
into the GPR registers, right ? Also, I don't think there's some magic register 
which allows controlling the PCIe core and PCIe PHY reset lines directly (like 
on exynos), or is there please?

Best regards,
Marek Vasut

Re: [RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Marek Vasut]

On Thursday, June 26, 2014 at 09:32:30 AM, Juergen Borleis wrote:
> Hi Fabio,
> 
> On Thursday 26 June 2014 05:43:19 Fabio Estevam wrote:
> > [...]
> 
> > Below is the log with linux-next kernel and earlyprintk enabled:
> Does it also hang when you disable "earlyprintk"? A few weeks ago we had a
> similar issue with i.MX6/PCI and "earlyprintk" (at the end it was caused by
> a clock change for the UART when changing from the early to the regular
> console).

How can the UART clock have impact on the PCIe please ?

Best regards,
Marek Vasut

[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Marek Vasut]

On Thursday, June 26, 2014 at 09:32:30 AM, Juergen Borleis wrote:
> Hi Fabio,
> 
> On Thursday 26 June 2014 05:43:19 Fabio Estevam wrote:
> > [...]
> 
> > Below is the log with linux-next kernel and earlyprintk enabled:
> Does it also hang when you disable "earlyprintk"? A few weeks ago we had a
> similar issue with i.MX6/PCI and "earlyprintk" (at the end it was caused by
> a clock change for the UART when changing from the early to the regular
> console).

How can the UART clock have impact on the PCIe please ?

Best regards,
Marek Vasut

Re: [RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Juergen Borleis]

Hi Marek,

On Thursday 26 June 2014 11:17:12 Marek Vasut wrote:
> On Thursday, June 26, 2014 at 09:32:30 AM, Juergen Borleis wrote:
> > On Thursday 26 June 2014 05:43:19 Fabio Estevam wrote:
> > > [...]
> > >
> > > Below is the log with linux-next kernel and earlyprintk enabled:
> >
> > Does it also hang when you disable "earlyprintk"? A few weeks ago we had
> > a similar issue with i.MX6/PCI and "earlyprintk" (at the end it was
> > caused by a clock change for the UART when changing from the early to the
> > regular console).
>
> How can the UART clock have impact on the PCIe please ?

Not to the PCIe hardware, it just hanged the kernel when the PCIe driver came 
up (due to some of its messages it tries to output at this moment).

jbe

-- 
Pengutronix e.K.                              | Juergen Borleis             |
Industrial Linux Solutions                    | Phone: +49-5121-206917-5128 |
Peiner Str. 6-8, 31137 Hildesheim, Germany    | Fax:   +49-5121-206917-5555 |
Amtsgericht Hildesheim, HRA 2686              | http://www.pengutronix.de/  |

[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Juergen Borleis]

Hi Marek,

On Thursday 26 June 2014 11:17:12 Marek Vasut wrote:
> On Thursday, June 26, 2014 at 09:32:30 AM, Juergen Borleis wrote:
> > On Thursday 26 June 2014 05:43:19 Fabio Estevam wrote:
> > > [...]
> > >
> > > Below is the log with linux-next kernel and earlyprintk enabled:
> >
> > Does it also hang when you disable "earlyprintk"? A few weeks ago we had
> > a similar issue with i.MX6/PCI and "earlyprintk" (at the end it was
> > caused by a clock change for the UART when changing from the early to the
> > regular console).
>
> How can the UART clock have impact on the PCIe please ?

Not to the PCIe hardware, it just hanged the kernel when the PCIe driver came 
up (due to some of its messages it tries to output at this moment).

jbe

-- 
Pengutronix e.K. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?| Juergen Borleis ? ? ? ? ? ? |
Industrial Linux Solutions               ? ? ?| Phone: +49-5121-206917-5128 |
Peiner Str. 6-8, 31137 Hildesheim, Germany ? ?| Fax: ? +49-5121-206917-5555 |
Amtsgericht Hildesheim, HRA 2686 ? ? ? ? ? ? ?| http://www.pengutronix.de/ ?|

Re: [RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Marek Vasut]

On Thursday, June 26, 2014 at 11:26:09 AM, Juergen Borleis wrote:
> Hi Marek,
> 
> On Thursday 26 June 2014 11:17:12 Marek Vasut wrote:
> > On Thursday, June 26, 2014 at 09:32:30 AM, Juergen Borleis wrote:
> > > On Thursday 26 June 2014 05:43:19 Fabio Estevam wrote:
> > > > [...]
> > > 
> > > > Below is the log with linux-next kernel and earlyprintk enabled:
> > > Does it also hang when you disable "earlyprintk"? A few weeks ago we
> > > had a similar issue with i.MX6/PCI and "earlyprintk" (at the end it
> > > was caused by a clock change for the UART when changing from the early
> > > to the regular console).
> > 
> > How can the UART clock have impact on the PCIe please ?
> 
> Not to the PCIe hardware, it just hanged the kernel when the PCIe driver
> came up (due to some of its messages it tries to output at this moment).

Another hint, can a blocking msleep() hang the driver ? The driver probes in 
fs_initcall stage, I'm not sure if it cannot hit some kind of a problem when 
using the msleep() ? It's unlikely though ...

Best regards,
Marek Vasut

[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Marek Vasut]

On Thursday, June 26, 2014 at 11:26:09 AM, Juergen Borleis wrote:
> Hi Marek,
> 
> On Thursday 26 June 2014 11:17:12 Marek Vasut wrote:
> > On Thursday, June 26, 2014 at 09:32:30 AM, Juergen Borleis wrote:
> > > On Thursday 26 June 2014 05:43:19 Fabio Estevam wrote:
> > > > [...]
> > > 
> > > > Below is the log with linux-next kernel and earlyprintk enabled:
> > > Does it also hang when you disable "earlyprintk"? A few weeks ago we
> > > had a similar issue with i.MX6/PCI and "earlyprintk" (at the end it
> > > was caused by a clock change for the UART when changing from the early
> > > to the regular console).
> > 
> > How can the UART clock have impact on the PCIe please ?
> 
> Not to the PCIe hardware, it just hanged the kernel when the PCIe driver
> came up (due to some of its messages it tries to output at this moment).

Another hint, can a blocking msleep() hang the driver ? The driver probes in 
fs_initcall stage, I'm not sure if it cannot hit some kind of a problem when 
using the msleep() ? It's unlikely though ...

Best regards,
Marek Vasut

Re: [RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Fabio Estevam]

Hi Juergen,

On Thu, Jun 26, 2014 at 4:32 AM, Juergen Borleis <jbe@pengutronix.de> wrote:
> Hi Fabio,
>
> On Thursday 26 June 2014 05:43:19 Fabio Estevam wrote:
>> [...]
>> Below is the log with linux-next kernel and earlyprintk enabled:
>
> Does it also hang when you disable "earlyprintk"? A few weeks ago we had a
> similar issue with i.MX6/PCI and "earlyprintk" (at the end it was caused by a
> clock change for the UART when changing from the early to the regular
> console).

Yes, it also hangs without earlyprintk.

Regards,

Fabio Estevam

[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Fabio Estevam]

Hi Juergen,

On Thu, Jun 26, 2014 at 4:32 AM, Juergen Borleis <jbe@pengutronix.de> wrote:
> Hi Fabio,
>
> On Thursday 26 June 2014 05:43:19 Fabio Estevam wrote:
>> [...]
>> Below is the log with linux-next kernel and earlyprintk enabled:
>
> Does it also hang when you disable "earlyprintk"? A few weeks ago we had a
> similar issue with i.MX6/PCI and "earlyprintk" (at the end it was caused by a
> clock change for the UART when changing from the early to the regular
> console).

Yes, it also hangs without earlyprintk.

Regards,

Fabio Estevam

Re: [RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Tim Harvey]

On Wed, Jun 25, 2014 at 10:49 PM, Shawn Guo <shawn.guo@freescale.com> wrote:
> On Thu, Jun 26, 2014 at 12:43:19AM -0300, Fabio Estevam wrote:
>> On Thu, Jun 26, 2014 at 12:12 AM, Shawn Guo <shawn.guo@freescale.com> wrote:
>> > On Tue, Jun 24, 2014 at 04:18:27PM -0300, Fabio Estevam wrote:
>> >> From: Fabio Estevam <fabio.estevam@freescale.com>
>> >>
>> >> When the mx6 PCI conctroller is initialized in the bootloader we see a kernel
>> >> hang inside imx6_add_pcie_port().
>> >>
>> >> Adding a 30ms delay allows the kernel to boot.
>> >
>> > We may not want to add a random delay into the driver before we
>> > understand the root cause of the issue.
>>
>> Yes, that's why I sent this as RFC and also explained it below the ---
>> line that I am actually trying to get some help with this issue.
>>
>> >
>> > Do you see this issue with FSL kernel?
>>
>> Yes, it also hangs.
>>
>> It is reproducible in 100% of the boots. Just need to use mainline
>> U-boot (which has PCI driver enabled by default).
>> I am using an Intel Wifi 7260 PCI card. This was also reported by
>> other folks in the U-boot list.
>
> Richard,
>
> Can you schedule some time to look at this issue?  I think it will come
> to us sooner or later if any our customer enables PCIe before launching
> kernel?
>
> Shawn
>

Shawn / Richard,

I am also affected by this issue on IMX6 boards that I support. If I
enable PCI in the bootloader I see similar hangs.

I have the following hardware configurations on my bench:
  1. IMX6DL + i210 (same PCI setup as Fabio's above, but DL instead of Q)
  2. IMX6Q + ath9k device
  3. IMX6DL + PLX PEX860x PCIe-to-PCIe bridge with various devices
behind the bridge, using a clock buffer from IMX6 PCIe clock
  4. IMX6Q + PLX PEX860x PCIe-to-PCIe bridge with various devices
behind the bridge, using a clock buffer from IMX6 PCIe clock
  5. IMX6Q + PLX PEX860x PCIe-to-PCIe bridge with various devices
behind the bridge using a clock generator (always on, ignoring the
PCIe clock)

For all of the above I have no PCI issues using
3.14/3.15/3.16-rc2/vendor 3.10.17_1.0.0_ga unless I enable PCI in the
bootloader. When I do so, all of the above configurations hang
somewhere around PCI init/enumeration. The same occurs with the most
recent vendor kernel 3.10.17_1.0.0_ga kernel (works when PCI is
disabled in the bootloader, hangs otherwise).

When I apply Fabio's patch above to the 3.16-rc2 kernel I find that
scenarios #4 and #5 above then work, #3 boots but the PLX bridge fails
all config cycles (0xff's), #2 boots but with no PCIe link, and #1
above still hangs. Previously, when I have dug into this particular
'hang' issue on 3.15 I found that the delay needed to be between
imx6_pcie_probe() requesting and asserting reset_gpio low, and before
setting IOMUX_GPR1:18 to power down the PCIe PHY (note here, that the
PHY is currently enabled in the bootloader when PCI is enabled there).

When I apply Fabio's patch above to the most recent vendor kernel
3.10.17_1.0.0_ga I still hang in all cases.

So while I agree there is something horribly wrong with IMX6 PCI
still, I don't think Fabio's patch is the right solution and I don't
have anything better at this point in time. I'm happy to share any
hardware with anyone that can work through this issue.

Thanks,

Tim

[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Tim Harvey]

On Wed, Jun 25, 2014 at 10:49 PM, Shawn Guo <shawn.guo@freescale.com> wrote:
> On Thu, Jun 26, 2014 at 12:43:19AM -0300, Fabio Estevam wrote:
>> On Thu, Jun 26, 2014 at 12:12 AM, Shawn Guo <shawn.guo@freescale.com> wrote:
>> > On Tue, Jun 24, 2014 at 04:18:27PM -0300, Fabio Estevam wrote:
>> >> From: Fabio Estevam <fabio.estevam@freescale.com>
>> >>
>> >> When the mx6 PCI conctroller is initialized in the bootloader we see a kernel
>> >> hang inside imx6_add_pcie_port().
>> >>
>> >> Adding a 30ms delay allows the kernel to boot.
>> >
>> > We may not want to add a random delay into the driver before we
>> > understand the root cause of the issue.
>>
>> Yes, that's why I sent this as RFC and also explained it below the ---
>> line that I am actually trying to get some help with this issue.
>>
>> >
>> > Do you see this issue with FSL kernel?
>>
>> Yes, it also hangs.
>>
>> It is reproducible in 100% of the boots. Just need to use mainline
>> U-boot (which has PCI driver enabled by default).
>> I am using an Intel Wifi 7260 PCI card. This was also reported by
>> other folks in the U-boot list.
>
> Richard,
>
> Can you schedule some time to look at this issue?  I think it will come
> to us sooner or later if any our customer enables PCIe before launching
> kernel?
>
> Shawn
>

Shawn / Richard,

I am also affected by this issue on IMX6 boards that I support. If I
enable PCI in the bootloader I see similar hangs.

I have the following hardware configurations on my bench:
  1. IMX6DL + i210 (same PCI setup as Fabio's above, but DL instead of Q)
  2. IMX6Q + ath9k device
  3. IMX6DL + PLX PEX860x PCIe-to-PCIe bridge with various devices
behind the bridge, using a clock buffer from IMX6 PCIe clock
  4. IMX6Q + PLX PEX860x PCIe-to-PCIe bridge with various devices
behind the bridge, using a clock buffer from IMX6 PCIe clock
  5. IMX6Q + PLX PEX860x PCIe-to-PCIe bridge with various devices
behind the bridge using a clock generator (always on, ignoring the
PCIe clock)

For all of the above I have no PCI issues using
3.14/3.15/3.16-rc2/vendor 3.10.17_1.0.0_ga unless I enable PCI in the
bootloader. When I do so, all of the above configurations hang
somewhere around PCI init/enumeration. The same occurs with the most
recent vendor kernel 3.10.17_1.0.0_ga kernel (works when PCI is
disabled in the bootloader, hangs otherwise).

When I apply Fabio's patch above to the 3.16-rc2 kernel I find that
scenarios #4 and #5 above then work, #3 boots but the PLX bridge fails
all config cycles (0xff's), #2 boots but with no PCIe link, and #1
above still hangs. Previously, when I have dug into this particular
'hang' issue on 3.15 I found that the delay needed to be between
imx6_pcie_probe() requesting and asserting reset_gpio low, and before
setting IOMUX_GPR1:18 to power down the PCIe PHY (note here, that the
PHY is currently enabled in the bootloader when PCI is enabled there).

When I apply Fabio's patch above to the most recent vendor kernel
3.10.17_1.0.0_ga I still hang in all cases.

So while I agree there is something horribly wrong with IMX6 PCI
still, I don't think Fabio's patch is the right solution and I don't
have anything better at this point in time. I'm happy to share any
hardware with anyone that can work through this issue.

Thanks,

Tim

RE: [RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Hong-Xing.Zhu]

SGkNCg0KDQo+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+IEZyb206IFRpbSBIYXJ2ZXkg
W21haWx0bzp0aGFydmV5QGdhdGV3b3Jrcy5jb21dDQo+IFNlbnQ6IEZyaWRheSwgSnVuZSAyNywg
MjAxNCA4OjI5IEFNDQo+IFRvOiBHdW8gU2hhd24tUjY1MDczOyBaaHUgUmljaGFyZC1SNjUwMzcN
Cj4gQ2M6IEZhYmlvIEVzdGV2YW07IEJqb3JuIEhlbGdhYXM7IE1hcmVrIFZhxaF1dDsgRGF2aWQg
TcO8bGxlciAoRUxTT0ZUIEFHKTsNCj4gU2FzY2hhIEhhdWVyOyBsaW51eC1hcm0ta2VybmVsQGxp
c3RzLmluZnJhZGVhZC5vcmc7IGxpbnV4LXBjaUB2Z2VyLmtlcm5lbC5vcmc7DQo+IEVzdGV2YW0g
RmFiaW8tUjQ5NDk2DQo+IFN1YmplY3Q6IFJlOiBbUkZDXSBQQ0k6IHBjaS1pbXg2OiBBZGQgZGVs
YXkgdG8gd29ya2Fyb3VuZCBrZXJuZWwgaGFuZw0KPiANCj4gT24gV2VkLCBKdW4gMjUsIDIwMTQg
YXQgMTA6NDkgUE0sIFNoYXduIEd1byA8c2hhd24uZ3VvQGZyZWVzY2FsZS5jb20+IHdyb3RlOg0K
PiA+IE9uIFRodSwgSnVuIDI2LCAyMDE0IGF0IDEyOjQzOjE5QU0gLTAzMDAsIEZhYmlvIEVzdGV2
YW0gd3JvdGU6DQo+ID4+IE9uIFRodSwgSnVuIDI2LCAyMDE0IGF0IDEyOjEyIEFNLCBTaGF3biBH
dW8gPHNoYXduLmd1b0BmcmVlc2NhbGUuY29tPiB3cm90ZToNCj4gPj4gPiBPbiBUdWUsIEp1biAy
NCwgMjAxNCBhdCAwNDoxODoyN1BNIC0wMzAwLCBGYWJpbyBFc3RldmFtIHdyb3RlOg0KPiA+PiA+
PiBGcm9tOiBGYWJpbyBFc3RldmFtIDxmYWJpby5lc3RldmFtQGZyZWVzY2FsZS5jb20+DQo+ID4+
ID4+DQo+ID4+ID4+IFdoZW4gdGhlIG14NiBQQ0kgY29uY3Ryb2xsZXIgaXMgaW5pdGlhbGl6ZWQg
aW4gdGhlIGJvb3Rsb2FkZXIgd2UNCj4gPj4gPj4gc2VlIGEga2VybmVsIGhhbmcgaW5zaWRlIGlt
eDZfYWRkX3BjaWVfcG9ydCgpLg0KPiA+PiA+Pg0KPiA+PiA+PiBBZGRpbmcgYSAzMG1zIGRlbGF5
IGFsbG93cyB0aGUga2VybmVsIHRvIGJvb3QuDQo+ID4+ID4NCj4gPj4gPiBXZSBtYXkgbm90IHdh
bnQgdG8gYWRkIGEgcmFuZG9tIGRlbGF5IGludG8gdGhlIGRyaXZlciBiZWZvcmUgd2UNCj4gPj4g
PiB1bmRlcnN0YW5kIHRoZSByb290IGNhdXNlIG9mIHRoZSBpc3N1ZS4NCj4gPj4NCj4gPj4gWWVz
LCB0aGF0J3Mgd2h5IEkgc2VudCB0aGlzIGFzIFJGQyBhbmQgYWxzbyBleHBsYWluZWQgaXQgYmVs
b3cgdGhlDQo+ID4+IC0tLSBsaW5lIHRoYXQgSSBhbSBhY3R1YWxseSB0cnlpbmcgdG8gZ2V0IHNv
bWUgaGVscCB3aXRoIHRoaXMgaXNzdWUuDQo+ID4+DQo+ID4+ID4NCj4gPj4gPiBEbyB5b3Ugc2Vl
IHRoaXMgaXNzdWUgd2l0aCBGU0wga2VybmVsPw0KPiA+Pg0KPiA+PiBZZXMsIGl0IGFsc28gaGFu
Z3MuDQo+ID4+DQo+ID4+IEl0IGlzIHJlcHJvZHVjaWJsZSBpbiAxMDAlIG9mIHRoZSBib290cy4g
SnVzdCBuZWVkIHRvIHVzZSBtYWlubGluZQ0KPiA+PiBVLWJvb3QgKHdoaWNoIGhhcyBQQ0kgZHJp
dmVyIGVuYWJsZWQgYnkgZGVmYXVsdCkuDQo+ID4+IEkgYW0gdXNpbmcgYW4gSW50ZWwgV2lmaSA3
MjYwIFBDSSBjYXJkLiBUaGlzIHdhcyBhbHNvIHJlcG9ydGVkIGJ5DQo+ID4+IG90aGVyIGZvbGtz
IGluIHRoZSBVLWJvb3QgbGlzdC4NCj4gPg0KPiA+IFJpY2hhcmQsDQo+ID4NCj4gPiBDYW4geW91
IHNjaGVkdWxlIHNvbWUgdGltZSB0byBsb29rIGF0IHRoaXMgaXNzdWU/ICBJIHRoaW5rIGl0IHdp
bGwNCj4gPiBjb21lIHRvIHVzIHNvb25lciBvciBsYXRlciBpZiBhbnkgb3VyIGN1c3RvbWVyIGVu
YWJsZXMgUENJZSBiZWZvcmUNCj4gPiBsYXVuY2hpbmcga2VybmVsPw0KPiA+DQo+ID4gU2hhd24N
Cj4gPg0KPiANCltSaWNoYXJkXSBJIGRpZCB0aGUgdGVzdHMgcmVmZXIgdG8gdGhpcyB1c2UtY2Fz
ZSwgZW5hYmxlIGlteDYgcGNpZSBvbiBib3RoIHUtYm9vdCBhbmQga2VybmVsLA0KYW5kIEkgZW5j
b3VudGVyIHRoZSBzeXN0ZW0gaGFuZyB0b28uDQoNCkhlcmUgYXJlIHRoZSBsYXRlc3QgdXBkYXRl
cyBhbmQgc29tZSBjbHVlcyBmcm9tIG15IHNpZGU6DQotIEFib3V0IDIwMHVzIGRlbGF5IGlzIHJl
cXVpcmVkIGFmdGVyIHRoZSBsdHNzbV9lbiBpcyBzZXQgdG8gYmUgJzEnIGF0IG15IHNpZGUuDQpP
dGhlcndpc2UsIHN5c3RlbSB3b3VsZCBiZSBoYW5nIHdoZW4gZHJpdmVyIGFjY2VzcyB0aGUgcGNp
ZV9waHlfZGVidWdfcjEgdG8gY2hlY2sNCnRoZSBsaW5rIGlzIHVwIG9yIG5vdC4NCg0KLSBBZnRl
ciB0aGF0LCBzeXN0ZW0gaGFuZyB3aGVuIHJjIHRyeWluZyB0byBhY2Nlc3MgdGhlIGNmZyBzcGFj
ZSBvZiBlcCBkZXZpY2UsDQpBbHRob3VnaCB0aGUgcGNpZSBsaW5rIGhhZCBiZWVuIHNldHVwIGFu
ZCBJIGRvbid0IGhhdmUga25vdyB0aGUgcm9vdCBjYXVzZSB5ZXQuDQoNCi0gVGhlIGtlcm5lbCBj
YW4gYm9vdCB1cCBzdWNjZXNzZnVsbHksIHdoZW4gSSBtYXNrIHRoZSBjZmcgcmVhZC93cml0ZSBj
YWxsLWJhY2sgaW4gcGNpZS1kZXNpZ253YXJlLmMNCkRyaXZlciBmb3IgZGVidWcgcHVycG9zZS4N
Cg0KTm90ZToNClRoZSBzZXF1ZW5jZSBvZiB0aGUgcGNpZSBpbml0aWFsaXphdGlvbiBzaG91bGQg
YmUgYWRqdXN0ZWQgcmVmZXIgdG8gdGhlIG5ld2x5DQogZGlzY292ZXJlZCBidWcocGNpZSBsaW5r
IG1heWJlIHJhcmVseSByYW5kb20gZG93biBhZnRlciB0aGUgc3lzdGVtIHdhcm0tcmVzZXQpLg0K
KiBSZWZfc3NwX2VuKGJpdDE2IG9mIGdwcjEgcmVnaXN0ZXIpIHNob3VsZCBiZSBzZXQgYWZ0ZXIg
dGhlIHBjaWUgb3RoZXJzIGNsa3MgYXJlIGVuYWJsZS4NCkVuYWJsZSBwY2llIHJlbGF0ZWQgY2xr
cyAtLT4gZGVsYXkgZm9yIGFib3V0IH4xMHVzIHdhaXRpbmcgZm9yIHRoZSBjbGtzIHN0YWJsZS0t
PnNldCByZWZfc3NwX2VuKGJpdDE2IG9mIGdwcjEgcmVnaXN0ZXIpLg0KDQpEZWJ1ZyBpcyBzdGls
bCBvbi1nb2luZy4NCg0KPiBTaGF3biAvIFJpY2hhcmQsDQo+IA0KPiBJIGFtIGFsc28gYWZmZWN0
ZWQgYnkgdGhpcyBpc3N1ZSBvbiBJTVg2IGJvYXJkcyB0aGF0IEkgc3VwcG9ydC4gSWYgSSBlbmFi
bGUNCj4gUENJIGluIHRoZSBib290bG9hZGVyIEkgc2VlIHNpbWlsYXIgaGFuZ3MuDQo+IA0KPiBJ
IGhhdmUgdGhlIGZvbGxvd2luZyBoYXJkd2FyZSBjb25maWd1cmF0aW9ucyBvbiBteSBiZW5jaDoN
Cj4gICAxLiBJTVg2REwgKyBpMjEwIChzYW1lIFBDSSBzZXR1cCBhcyBGYWJpbydzIGFib3ZlLCBi
dXQgREwgaW5zdGVhZCBvZiBRKQ0KPiAgIDIuIElNWDZRICsgYXRoOWsgZGV2aWNlDQo+ICAgMy4g
SU1YNkRMICsgUExYIFBFWDg2MHggUENJZS10by1QQ0llIGJyaWRnZSB3aXRoIHZhcmlvdXMgZGV2
aWNlcyBiZWhpbmQgdGhlDQo+IGJyaWRnZSwgdXNpbmcgYSBjbG9jayBidWZmZXIgZnJvbSBJTVg2
IFBDSWUgY2xvY2sNCj4gICA0LiBJTVg2USArIFBMWCBQRVg4NjB4IFBDSWUtdG8tUENJZSBicmlk
Z2Ugd2l0aCB2YXJpb3VzIGRldmljZXMgYmVoaW5kIHRoZQ0KPiBicmlkZ2UsIHVzaW5nIGEgY2xv
Y2sgYnVmZmVyIGZyb20gSU1YNiBQQ0llIGNsb2NrDQo+ICAgNS4gSU1YNlEgKyBQTFggUEVYODYw
eCBQQ0llLXRvLVBDSWUgYnJpZGdlIHdpdGggdmFyaW91cyBkZXZpY2VzIGJlaGluZCB0aGUNCj4g
YnJpZGdlIHVzaW5nIGEgY2xvY2sgZ2VuZXJhdG9yIChhbHdheXMgb24sIGlnbm9yaW5nIHRoZSBQ
Q0llIGNsb2NrKQ0KPiANCj4gRm9yIGFsbCBvZiB0aGUgYWJvdmUgSSBoYXZlIG5vIFBDSSBpc3N1
ZXMgdXNpbmcgMy4xNC8zLjE1LzMuMTYtcmMyL3ZlbmRvcg0KPiAzLjEwLjE3XzEuMC4wX2dhIHVu
bGVzcyBJIGVuYWJsZSBQQ0kgaW4gdGhlIGJvb3Rsb2FkZXIuIFdoZW4gSSBkbyBzbywgYWxsIG9m
DQo+IHRoZSBhYm92ZSBjb25maWd1cmF0aW9ucyBoYW5nIHNvbWV3aGVyZSBhcm91bmQgUENJIGlu
aXQvZW51bWVyYXRpb24uIFRoZSBzYW1lDQo+IG9jY3VycyB3aXRoIHRoZSBtb3N0IHJlY2VudCB2
ZW5kb3Iga2VybmVsIDMuMTAuMTdfMS4wLjBfZ2Ega2VybmVsICh3b3JrcyB3aGVuDQo+IFBDSSBp
cyBkaXNhYmxlZCBpbiB0aGUgYm9vdGxvYWRlciwgaGFuZ3Mgb3RoZXJ3aXNlKS4NCj4gDQo+IFdo
ZW4gSSBhcHBseSBGYWJpbydzIHBhdGNoIGFib3ZlIHRvIHRoZSAzLjE2LXJjMiBrZXJuZWwgSSBm
aW5kIHRoYXQgc2NlbmFyaW9zDQo+ICM0IGFuZCAjNSBhYm92ZSB0aGVuIHdvcmssICMzIGJvb3Rz
IGJ1dCB0aGUgUExYIGJyaWRnZSBmYWlscyBhbGwgY29uZmlnIGN5Y2xlcw0KPiAoMHhmZidzKSwg
IzIgYm9vdHMgYnV0IHdpdGggbm8gUENJZSBsaW5rLCBhbmQgIzEgYWJvdmUgc3RpbGwgaGFuZ3Mu
IFByZXZpb3VzbHksDQo+IHdoZW4gSSBoYXZlIGR1ZyBpbnRvIHRoaXMgcGFydGljdWxhciAnaGFu
ZycgaXNzdWUgb24gMy4xNSBJIGZvdW5kIHRoYXQgdGhlDQo+IGRlbGF5IG5lZWRlZCB0byBiZSBi
ZXR3ZWVuDQo+IGlteDZfcGNpZV9wcm9iZSgpIHJlcXVlc3RpbmcgYW5kIGFzc2VydGluZyByZXNl
dF9ncGlvIGxvdywgYW5kIGJlZm9yZSBzZXR0aW5nDQo+IElPTVVYX0dQUjE6MTggdG8gcG93ZXIg
ZG93biB0aGUgUENJZSBQSFkgKG5vdGUgaGVyZSwgdGhhdCB0aGUgUEhZIGlzIGN1cnJlbnRseQ0K
PiBlbmFibGVkIGluIHRoZSBib290bG9hZGVyIHdoZW4gUENJIGlzIGVuYWJsZWQgdGhlcmUpLg0K
PiANCj4gV2hlbiBJIGFwcGx5IEZhYmlvJ3MgcGF0Y2ggYWJvdmUgdG8gdGhlIG1vc3QgcmVjZW50
IHZlbmRvciBrZXJuZWwNCj4gMy4xMC4xN18xLjAuMF9nYSBJIHN0aWxsIGhhbmcgaW4gYWxsIGNh
c2VzLg0KPiANCj4gU28gd2hpbGUgSSBhZ3JlZSB0aGVyZSBpcyBzb21ldGhpbmcgaG9ycmlibHkg
d3Jvbmcgd2l0aCBJTVg2IFBDSSBzdGlsbCwgSQ0KPiBkb24ndCB0aGluayBGYWJpbydzIHBhdGNo
IGlzIHRoZSByaWdodCBzb2x1dGlvbiBhbmQgSSBkb24ndCBoYXZlIGFueXRoaW5nDQo+IGJldHRl
ciBhdCB0aGlzIHBvaW50IGluIHRpbWUuIEknbSBoYXBweSB0byBzaGFyZSBhbnkgaGFyZHdhcmUg
d2l0aCBhbnlvbmUgdGhhdA0KPiBjYW4gd29yayB0aHJvdWdoIHRoaXMgaXNzdWUuDQo+IA0KPiBU
aGFua3MsDQo+IA0KPiBUaW0NCg0KQmVzdCBSZWdhcmRzDQpSaWNoYXJkIFpodQ0KDQo=

[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Hong-Xing.Zhu at freescale.com]

Hi


> -----Original Message-----
> From: Tim Harvey [mailto:tharvey at gateworks.com]
> Sent: Friday, June 27, 2014 8:29 AM
> To: Guo Shawn-R65073; Zhu Richard-R65037
> Cc: Fabio Estevam; Bjorn Helgaas; Marek Va?ut; David M?ller (ELSOFT AG);
> Sascha Hauer; linux-arm-kernel at lists.infradead.org; linux-pci at vger.kernel.org;
> Estevam Fabio-R49496
> Subject: Re: [RFC] PCI: pci-imx6: Add delay to workaround kernel hang
> 
> On Wed, Jun 25, 2014 at 10:49 PM, Shawn Guo <shawn.guo@freescale.com> wrote:
> > On Thu, Jun 26, 2014 at 12:43:19AM -0300, Fabio Estevam wrote:
> >> On Thu, Jun 26, 2014 at 12:12 AM, Shawn Guo <shawn.guo@freescale.com> wrote:
> >> > On Tue, Jun 24, 2014 at 04:18:27PM -0300, Fabio Estevam wrote:
> >> >> From: Fabio Estevam <fabio.estevam@freescale.com>
> >> >>
> >> >> When the mx6 PCI conctroller is initialized in the bootloader we
> >> >> see a kernel hang inside imx6_add_pcie_port().
> >> >>
> >> >> Adding a 30ms delay allows the kernel to boot.
> >> >
> >> > We may not want to add a random delay into the driver before we
> >> > understand the root cause of the issue.
> >>
> >> Yes, that's why I sent this as RFC and also explained it below the
> >> --- line that I am actually trying to get some help with this issue.
> >>
> >> >
> >> > Do you see this issue with FSL kernel?
> >>
> >> Yes, it also hangs.
> >>
> >> It is reproducible in 100% of the boots. Just need to use mainline
> >> U-boot (which has PCI driver enabled by default).
> >> I am using an Intel Wifi 7260 PCI card. This was also reported by
> >> other folks in the U-boot list.
> >
> > Richard,
> >
> > Can you schedule some time to look at this issue?  I think it will
> > come to us sooner or later if any our customer enables PCIe before
> > launching kernel?
> >
> > Shawn
> >
> 
[Richard] I did the tests refer to this use-case, enable imx6 pcie on both u-boot and kernel,
and I encounter the system hang too.

Here are the latest updates and some clues from my side:
- About 200us delay is required after the ltssm_en is set to be '1' at my side.
Otherwise, system would be hang when driver access the pcie_phy_debug_r1 to check
the link is up or not.

- After that, system hang when rc trying to access the cfg space of ep device,
Although the pcie link had been setup and I don't have know the root cause yet.

- The kernel can boot up successfully, when I mask the cfg read/write call-back in pcie-designware.c
Driver for debug purpose.

Note:
The sequence of the pcie initialization should be adjusted refer to the newly
 discovered bug(pcie link maybe rarely random down after the system warm-reset).
* Ref_ssp_en(bit16 of gpr1 register) should be set after the pcie others clks are enable.
Enable pcie related clks --> delay for about ~10us waiting for the clks stable-->set ref_ssp_en(bit16 of gpr1 register).

Debug is still on-going.

> Shawn / Richard,
> 
> I am also affected by this issue on IMX6 boards that I support. If I enable
> PCI in the bootloader I see similar hangs.
> 
> I have the following hardware configurations on my bench:
>   1. IMX6DL + i210 (same PCI setup as Fabio's above, but DL instead of Q)
>   2. IMX6Q + ath9k device
>   3. IMX6DL + PLX PEX860x PCIe-to-PCIe bridge with various devices behind the
> bridge, using a clock buffer from IMX6 PCIe clock
>   4. IMX6Q + PLX PEX860x PCIe-to-PCIe bridge with various devices behind the
> bridge, using a clock buffer from IMX6 PCIe clock
>   5. IMX6Q + PLX PEX860x PCIe-to-PCIe bridge with various devices behind the
> bridge using a clock generator (always on, ignoring the PCIe clock)
> 
> For all of the above I have no PCI issues using 3.14/3.15/3.16-rc2/vendor
> 3.10.17_1.0.0_ga unless I enable PCI in the bootloader. When I do so, all of
> the above configurations hang somewhere around PCI init/enumeration. The same
> occurs with the most recent vendor kernel 3.10.17_1.0.0_ga kernel (works when
> PCI is disabled in the bootloader, hangs otherwise).
> 
> When I apply Fabio's patch above to the 3.16-rc2 kernel I find that scenarios
> #4 and #5 above then work, #3 boots but the PLX bridge fails all config cycles
> (0xff's), #2 boots but with no PCIe link, and #1 above still hangs. Previously,
> when I have dug into this particular 'hang' issue on 3.15 I found that the
> delay needed to be between
> imx6_pcie_probe() requesting and asserting reset_gpio low, and before setting
> IOMUX_GPR1:18 to power down the PCIe PHY (note here, that the PHY is currently
> enabled in the bootloader when PCI is enabled there).
> 
> When I apply Fabio's patch above to the most recent vendor kernel
> 3.10.17_1.0.0_ga I still hang in all cases.
> 
> So while I agree there is something horribly wrong with IMX6 PCI still, I
> don't think Fabio's patch is the right solution and I don't have anything
> better at this point in time. I'm happy to share any hardware with anyone that
> can work through this issue.
> 
> Thanks,
> 
> Tim

Best Regards
Richard Zhu

Re: [RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Tim Harvey]

On Fri, Jun 27, 2014 at 3:06 AM, Hong-Xing.Zhu@freescale.com
<Hong-Xing.Zhu@freescale.com> wrote:
> [Richard] I did the tests refer to this use-case, enable imx6 pcie on both u-boot and kernel,
> and I encounter the system hang too.
>
> Here are the latest updates and some clues from my side:
> - About 200us delay is required after the ltssm_en is set to be '1' at my side.
> Otherwise, system would be hang when driver access the pcie_phy_debug_r1 to check
> the link is up or not.
>
> - After that, system hang when rc trying to access the cfg space of ep device,
> Although the pcie link had been setup and I don't have know the root cause yet.
>
> - The kernel can boot up successfully, when I mask the cfg read/write call-back in pcie-designware.c
> Driver for debug purpose.
>
> Note:
> The sequence of the pcie initialization should be adjusted refer to the newly
>  discovered bug(pcie link maybe rarely random down after the system warm-reset).
> * Ref_ssp_en(bit16 of gpr1 register) should be set after the pcie others clks are enable.
> Enable pcie related clks --> delay for about ~10us waiting for the clks stable-->set ref_ssp_en(bit16 of gpr1 register).
>
> Debug is still on-going.
>

Any update on this? Are you or others at Freescale actively working on this?

Regards,

Tim

[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Tim Harvey]

On Fri, Jun 27, 2014 at 3:06 AM, Hong-Xing.Zhu at freescale.com
<Hong-Xing.Zhu@freescale.com> wrote:
> [Richard] I did the tests refer to this use-case, enable imx6 pcie on both u-boot and kernel,
> and I encounter the system hang too.
>
> Here are the latest updates and some clues from my side:
> - About 200us delay is required after the ltssm_en is set to be '1' at my side.
> Otherwise, system would be hang when driver access the pcie_phy_debug_r1 to check
> the link is up or not.
>
> - After that, system hang when rc trying to access the cfg space of ep device,
> Although the pcie link had been setup and I don't have know the root cause yet.
>
> - The kernel can boot up successfully, when I mask the cfg read/write call-back in pcie-designware.c
> Driver for debug purpose.
>
> Note:
> The sequence of the pcie initialization should be adjusted refer to the newly
>  discovered bug(pcie link maybe rarely random down after the system warm-reset).
> * Ref_ssp_en(bit16 of gpr1 register) should be set after the pcie others clks are enable.
> Enable pcie related clks --> delay for about ~10us waiting for the clks stable-->set ref_ssp_en(bit16 of gpr1 register).
>
> Debug is still on-going.
>

Any update on this? Are you or others at Freescale actively working on this?

Regards,

Tim

Re: [RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Uwe Kleine-König]

Hello,

On Tue, Jun 24, 2014 at 04:18:27PM -0300, Fabio Estevam wrote:
> From: Fabio Estevam <fabio.estevam@freescale.com>
> 
> When the mx6 PCI conctroller is initialized in the bootloader we see a kernel 
> hang inside imx6_add_pcie_port().
> 
> Adding a 30ms delay allows the kernel to boot.
Just my thought on how to debug that: I'd try to bisect the pci init
routine in the boot loader. I.e. first only do the first half of the
initialisation in U-Boot. Depending on Linux being able to boot or not
initialize more or less on the next run.

Maybe there is a single register write that makes Linux fail?!

Best regards
Uwe

-- 
Pengutronix e.K.                           | Uwe Kleine-König            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |

[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Uwe Kleine-König]

Hello,

On Tue, Jun 24, 2014 at 04:18:27PM -0300, Fabio Estevam wrote:
> From: Fabio Estevam <fabio.estevam@freescale.com>
> 
> When the mx6 PCI conctroller is initialized in the bootloader we see a kernel 
> hang inside imx6_add_pcie_port().
> 
> Adding a 30ms delay allows the kernel to boot.
Just my thought on how to debug that: I'd try to bisect the pci init
routine in the boot loader. I.e. first only do the first half of the
initialisation in U-Boot. Depending on Linux being able to boot or not
initialize more or less on the next run.

Maybe there is a single register write that makes Linux fail?!

Best regards
Uwe

-- 
Pengutronix e.K.                           | Uwe Kleine-K?nig            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |

Re: [RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Marek Vasut]

On Thursday, July 17, 2014 at 08:51:48 AM, Uwe Kleine-König wrote:
> Hello,
> 
> On Tue, Jun 24, 2014 at 04:18:27PM -0300, Fabio Estevam wrote:
> > From: Fabio Estevam <fabio.estevam@freescale.com>
> > 
> > When the mx6 PCI conctroller is initialized in the bootloader we see a
> > kernel hang inside imx6_add_pcie_port().
> > 
> > Adding a 30ms delay allows the kernel to boot.
> 
> Just my thought on how to debug that: I'd try to bisect the pci init
> routine in the boot loader. I.e. first only do the first half of the
> initialisation in U-Boot. Depending on Linux being able to boot or not
> initialize more or less on the next run.
> 
> Maybe there is a single register write that makes Linux fail?!

I am still hell-bent on thinking that the missing PCIe block reset is what makes 
the Linux fail. Missing block reset is always a problem. Or do we now have a 
mean to reset the PCIe block and it's PHY from software?

Best regards,
Marek Vasut

[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Marek Vasut]

On Thursday, July 17, 2014 at 08:51:48 AM, Uwe Kleine-K?nig wrote:
> Hello,
> 
> On Tue, Jun 24, 2014 at 04:18:27PM -0300, Fabio Estevam wrote:
> > From: Fabio Estevam <fabio.estevam@freescale.com>
> > 
> > When the mx6 PCI conctroller is initialized in the bootloader we see a
> > kernel hang inside imx6_add_pcie_port().
> > 
> > Adding a 30ms delay allows the kernel to boot.
> 
> Just my thought on how to debug that: I'd try to bisect the pci init
> routine in the boot loader. I.e. first only do the first half of the
> initialisation in U-Boot. Depending on Linux being able to boot or not
> initialize more or less on the next run.
> 
> Maybe there is a single register write that makes Linux fail?!

I am still hell-bent on thinking that the missing PCIe block reset is what makes 
the Linux fail. Missing block reset is always a problem. Or do we now have a 
mean to reset the PCIe block and it's PHY from software?

Best regards,
Marek Vasut

Re: [RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Shawn Guo]

On Thu, Jul 17, 2014 at 10:23:10AM +0200, Marek Vasut wrote:
> On Thursday, July 17, 2014 at 08:51:48 AM, Uwe Kleine-König wrote:
> > Hello,
> > 
> > On Tue, Jun 24, 2014 at 04:18:27PM -0300, Fabio Estevam wrote:
> > > From: Fabio Estevam <fabio.estevam@freescale.com>
> > > 
> > > When the mx6 PCI conctroller is initialized in the bootloader we see a
> > > kernel hang inside imx6_add_pcie_port().
> > > 
> > > Adding a 30ms delay allows the kernel to boot.
> > 
> > Just my thought on how to debug that: I'd try to bisect the pci init
> > routine in the boot loader. I.e. first only do the first half of the
> > initialisation in U-Boot. Depending on Linux being able to boot or not
> > initialize more or less on the next run.
> > 
> > Maybe there is a single register write that makes Linux fail?!
> 
> I am still hell-bent on thinking that the missing PCIe block reset is what makes 
> the Linux fail. Missing block reset is always a problem.

Indeed.  We're missing a hardware reset for PCIe on i.MX6Q and i.MX6DL.
Such reset is available on i.MX6SX, so there is no this problem for
i.MX6SX PCIe.

> Or do we now have a 
> mean to reset the PCIe block and it's PHY from software?

Richard is trying to find a SW workaround for it, but we're not really
sure if it's possible.

Shawn

[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Shawn Guo]

On Thu, Jul 17, 2014 at 10:23:10AM +0200, Marek Vasut wrote:
> On Thursday, July 17, 2014 at 08:51:48 AM, Uwe Kleine-K?nig wrote:
> > Hello,
> > 
> > On Tue, Jun 24, 2014 at 04:18:27PM -0300, Fabio Estevam wrote:
> > > From: Fabio Estevam <fabio.estevam@freescale.com>
> > > 
> > > When the mx6 PCI conctroller is initialized in the bootloader we see a
> > > kernel hang inside imx6_add_pcie_port().
> > > 
> > > Adding a 30ms delay allows the kernel to boot.
> > 
> > Just my thought on how to debug that: I'd try to bisect the pci init
> > routine in the boot loader. I.e. first only do the first half of the
> > initialisation in U-Boot. Depending on Linux being able to boot or not
> > initialize more or less on the next run.
> > 
> > Maybe there is a single register write that makes Linux fail?!
> 
> I am still hell-bent on thinking that the missing PCIe block reset is what makes 
> the Linux fail. Missing block reset is always a problem.

Indeed.  We're missing a hardware reset for PCIe on i.MX6Q and i.MX6DL.
Such reset is available on i.MX6SX, so there is no this problem for
i.MX6SX PCIe.

> Or do we now have a 
> mean to reset the PCIe block and it's PHY from software?

Richard is trying to find a SW workaround for it, but we're not really
sure if it's possible.

Shawn

Re: [RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Lucas Stach]

Hi Tim,

Am Donnerstag, den 26.06.2014, 17:29 -0700 schrieb Tim Harvey:
[...]
> Shawn / Richard,
> 
> I am also affected by this issue on IMX6 boards that I support. If I
> enable PCI in the bootloader I see similar hangs.
> 
> I have the following hardware configurations on my bench:
>   1. IMX6DL + i210 (same PCI setup as Fabio's above, but DL instead of Q)
>   2. IMX6Q + ath9k device
>   3. IMX6DL + PLX PEX860x PCIe-to-PCIe bridge with various devices
> behind the bridge, using a clock buffer from IMX6 PCIe clock
>   4. IMX6Q + PLX PEX860x PCIe-to-PCIe bridge with various devices
> behind the bridge, using a clock buffer from IMX6 PCIe clock
>   5. IMX6Q + PLX PEX860x PCIe-to-PCIe bridge with various devices
> behind the bridge using a clock generator (always on, ignoring the
> PCIe clock)
> 
> For all of the above I have no PCI issues using
> 3.14/3.15/3.16-rc2/vendor 3.10.17_1.0.0_ga unless I enable PCI in the
> bootloader. When I do so, all of the above configurations hang
> somewhere around PCI init/enumeration. The same occurs with the most
> recent vendor kernel 3.10.17_1.0.0_ga kernel (works when PCI is
> disabled in the bootloader, hangs otherwise).
> 
> When I apply Fabio's patch above to the 3.16-rc2 kernel I find that
> scenarios #4 and #5 above then work, #3 boots but the PLX bridge fails
> all config cycles (0xff's), #2 boots but with no PCIe link, and #1
> above still hangs. Previously, when I have dug into this particular
> 'hang' issue on 3.15 I found that the delay needed to be between
> imx6_pcie_probe() requesting and asserting reset_gpio low, and before
> setting IOMUX_GPR1:18 to power down the PCIe PHY (note here, that the
> PHY is currently enabled in the bootloader when PCI is enabled there).
> 
> When I apply Fabio's patch above to the most recent vendor kernel
> 3.10.17_1.0.0_ga I still hang in all cases.
> 
> So while I agree there is something horribly wrong with IMX6 PCI
> still, I don't think Fabio's patch is the right solution and I don't
> have anything better at this point in time. I'm happy to share any
> hardware with anyone that can work through this issue.
> 
Can you try if the attached patch makes any difference?

Regards,
Lucas
------------------------------>8---------------------------------
>From 0e2fc443c760290166f6371d50813e6c30a678da Mon Sep 17 00:00:00 2001
From: Lucas Stach <l.stach@pengutronix.de>
Date: Thu, 17 Jul 2014 18:54:44 +0200
Subject: [PATCH] try forcing LTSSM into detect state

---
 drivers/pci/host/pci-imx6.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c
index a568efaa331c..afa450c54aba 100644
--- a/drivers/pci/host/pci-imx6.c
+++ b/drivers/pci/host/pci-imx6.c
@@ -215,6 +215,8 @@ static int imx6_pcie_assert_core_reset(struct pcie_port *pp)
 {
 	struct imx6_pcie *imx6_pcie = to_imx6_pcie(pp);
 
+	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR12,
+			IMX6Q_GPR12_PCIE_CTL_2, 0 << 10);
 	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
 			IMX6Q_GPR1_PCIE_TEST_PD, 1 << 18);
 	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
-- 
2.0.1


-- 
Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |

[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Lucas Stach]

Hi Tim,

Am Donnerstag, den 26.06.2014, 17:29 -0700 schrieb Tim Harvey:
[...]
> Shawn / Richard,
> 
> I am also affected by this issue on IMX6 boards that I support. If I
> enable PCI in the bootloader I see similar hangs.
> 
> I have the following hardware configurations on my bench:
>   1. IMX6DL + i210 (same PCI setup as Fabio's above, but DL instead of Q)
>   2. IMX6Q + ath9k device
>   3. IMX6DL + PLX PEX860x PCIe-to-PCIe bridge with various devices
> behind the bridge, using a clock buffer from IMX6 PCIe clock
>   4. IMX6Q + PLX PEX860x PCIe-to-PCIe bridge with various devices
> behind the bridge, using a clock buffer from IMX6 PCIe clock
>   5. IMX6Q + PLX PEX860x PCIe-to-PCIe bridge with various devices
> behind the bridge using a clock generator (always on, ignoring the
> PCIe clock)
> 
> For all of the above I have no PCI issues using
> 3.14/3.15/3.16-rc2/vendor 3.10.17_1.0.0_ga unless I enable PCI in the
> bootloader. When I do so, all of the above configurations hang
> somewhere around PCI init/enumeration. The same occurs with the most
> recent vendor kernel 3.10.17_1.0.0_ga kernel (works when PCI is
> disabled in the bootloader, hangs otherwise).
> 
> When I apply Fabio's patch above to the 3.16-rc2 kernel I find that
> scenarios #4 and #5 above then work, #3 boots but the PLX bridge fails
> all config cycles (0xff's), #2 boots but with no PCIe link, and #1
> above still hangs. Previously, when I have dug into this particular
> 'hang' issue on 3.15 I found that the delay needed to be between
> imx6_pcie_probe() requesting and asserting reset_gpio low, and before
> setting IOMUX_GPR1:18 to power down the PCIe PHY (note here, that the
> PHY is currently enabled in the bootloader when PCI is enabled there).
> 
> When I apply Fabio's patch above to the most recent vendor kernel
> 3.10.17_1.0.0_ga I still hang in all cases.
> 
> So while I agree there is something horribly wrong with IMX6 PCI
> still, I don't think Fabio's patch is the right solution and I don't
> have anything better at this point in time. I'm happy to share any
> hardware with anyone that can work through this issue.
> 
Can you try if the attached patch makes any difference?

Regards,
Lucas
------------------------------>8---------------------------------
>From 0e2fc443c760290166f6371d50813e6c30a678da Mon Sep 17 00:00:00 2001
From: Lucas Stach <l.stach@pengutronix.de>
Date: Thu, 17 Jul 2014 18:54:44 +0200
Subject: [PATCH] try forcing LTSSM into detect state

---
 drivers/pci/host/pci-imx6.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c
index a568efaa331c..afa450c54aba 100644
--- a/drivers/pci/host/pci-imx6.c
+++ b/drivers/pci/host/pci-imx6.c
@@ -215,6 +215,8 @@ static int imx6_pcie_assert_core_reset(struct pcie_port *pp)
 {
 	struct imx6_pcie *imx6_pcie = to_imx6_pcie(pp);
 
+	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR12,
+			IMX6Q_GPR12_PCIE_CTL_2, 0 << 10);
 	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
 			IMX6Q_GPR1_PCIE_TEST_PD, 1 << 18);
 	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
-- 
2.0.1


-- 
Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |

Re: [RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Marek Vasut]

On Friday, July 18, 2014 at 11:26:39 AM, Lucas Stach wrote:
> Hi Tim,
> 
> Am Donnerstag, den 26.06.2014, 17:29 -0700 schrieb Tim Harvey:
> [...]
> 
> > Shawn / Richard,
> > 
> > I am also affected by this issue on IMX6 boards that I support. If I
> > enable PCI in the bootloader I see similar hangs.
> > 
> > I have the following hardware configurations on my bench:
> >   1. IMX6DL + i210 (same PCI setup as Fabio's above, but DL instead of Q)
> >   2. IMX6Q + ath9k device
> >   3. IMX6DL + PLX PEX860x PCIe-to-PCIe bridge with various devices
> > 
> > behind the bridge, using a clock buffer from IMX6 PCIe clock
> > 
> >   4. IMX6Q + PLX PEX860x PCIe-to-PCIe bridge with various devices
> > 
> > behind the bridge, using a clock buffer from IMX6 PCIe clock
> > 
> >   5. IMX6Q + PLX PEX860x PCIe-to-PCIe bridge with various devices
> > 
> > behind the bridge using a clock generator (always on, ignoring the
> > PCIe clock)
> > 
> > For all of the above I have no PCI issues using
> > 3.14/3.15/3.16-rc2/vendor 3.10.17_1.0.0_ga unless I enable PCI in the
> > bootloader. When I do so, all of the above configurations hang
> > somewhere around PCI init/enumeration. The same occurs with the most
> > recent vendor kernel 3.10.17_1.0.0_ga kernel (works when PCI is
> > disabled in the bootloader, hangs otherwise).
> > 
> > When I apply Fabio's patch above to the 3.16-rc2 kernel I find that
> > scenarios #4 and #5 above then work, #3 boots but the PLX bridge fails
> > all config cycles (0xff's), #2 boots but with no PCIe link, and #1
> > above still hangs. Previously, when I have dug into this particular
> > 'hang' issue on 3.15 I found that the delay needed to be between
> > imx6_pcie_probe() requesting and asserting reset_gpio low, and before
> > setting IOMUX_GPR1:18 to power down the PCIe PHY (note here, that the
> > PHY is currently enabled in the bootloader when PCI is enabled there).
> > 
> > When I apply Fabio's patch above to the most recent vendor kernel
> > 3.10.17_1.0.0_ga I still hang in all cases.
> > 
> > So while I agree there is something horribly wrong with IMX6 PCI
> > still, I don't think Fabio's patch is the right solution and I don't
> > have anything better at this point in time. I'm happy to share any
> > hardware with anyone that can work through this issue.
> 
> Can you try if the attached patch makes any difference?
> 
> Regards,
> Lucas
> ------------------------------>8---------------------------------
> From 0e2fc443c760290166f6371d50813e6c30a678da Mon Sep 17 00:00:00 2001
> From: Lucas Stach <l.stach@pengutronix.de>
> Date: Thu, 17 Jul 2014 18:54:44 +0200
> Subject: [PATCH] try forcing LTSSM into detect state
> 
> ---
>  drivers/pci/host/pci-imx6.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c
> index a568efaa331c..afa450c54aba 100644
> --- a/drivers/pci/host/pci-imx6.c
> +++ b/drivers/pci/host/pci-imx6.c
> @@ -215,6 +215,8 @@ static int imx6_pcie_assert_core_reset(struct pcie_port
> *pp) {
>  	struct imx6_pcie *imx6_pcie = to_imx6_pcie(pp);
> 
> +	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR12,
> +			IMX6Q_GPR12_PCIE_CTL_2, 0 << 10);
>  	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
>  			IMX6Q_GPR1_PCIE_TEST_PD, 1 << 18);
>  	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,

Tried this, among many other things, but please let someone else also re-verify 
this. What I observed is that the LTSSM cannot leave the POLLING phase .

[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Marek Vasut]

On Friday, July 18, 2014 at 11:26:39 AM, Lucas Stach wrote:
> Hi Tim,
> 
> Am Donnerstag, den 26.06.2014, 17:29 -0700 schrieb Tim Harvey:
> [...]
> 
> > Shawn / Richard,
> > 
> > I am also affected by this issue on IMX6 boards that I support. If I
> > enable PCI in the bootloader I see similar hangs.
> > 
> > I have the following hardware configurations on my bench:
> >   1. IMX6DL + i210 (same PCI setup as Fabio's above, but DL instead of Q)
> >   2. IMX6Q + ath9k device
> >   3. IMX6DL + PLX PEX860x PCIe-to-PCIe bridge with various devices
> > 
> > behind the bridge, using a clock buffer from IMX6 PCIe clock
> > 
> >   4. IMX6Q + PLX PEX860x PCIe-to-PCIe bridge with various devices
> > 
> > behind the bridge, using a clock buffer from IMX6 PCIe clock
> > 
> >   5. IMX6Q + PLX PEX860x PCIe-to-PCIe bridge with various devices
> > 
> > behind the bridge using a clock generator (always on, ignoring the
> > PCIe clock)
> > 
> > For all of the above I have no PCI issues using
> > 3.14/3.15/3.16-rc2/vendor 3.10.17_1.0.0_ga unless I enable PCI in the
> > bootloader. When I do so, all of the above configurations hang
> > somewhere around PCI init/enumeration. The same occurs with the most
> > recent vendor kernel 3.10.17_1.0.0_ga kernel (works when PCI is
> > disabled in the bootloader, hangs otherwise).
> > 
> > When I apply Fabio's patch above to the 3.16-rc2 kernel I find that
> > scenarios #4 and #5 above then work, #3 boots but the PLX bridge fails
> > all config cycles (0xff's), #2 boots but with no PCIe link, and #1
> > above still hangs. Previously, when I have dug into this particular
> > 'hang' issue on 3.15 I found that the delay needed to be between
> > imx6_pcie_probe() requesting and asserting reset_gpio low, and before
> > setting IOMUX_GPR1:18 to power down the PCIe PHY (note here, that the
> > PHY is currently enabled in the bootloader when PCI is enabled there).
> > 
> > When I apply Fabio's patch above to the most recent vendor kernel
> > 3.10.17_1.0.0_ga I still hang in all cases.
> > 
> > So while I agree there is something horribly wrong with IMX6 PCI
> > still, I don't think Fabio's patch is the right solution and I don't
> > have anything better at this point in time. I'm happy to share any
> > hardware with anyone that can work through this issue.
> 
> Can you try if the attached patch makes any difference?
> 
> Regards,
> Lucas
> ------------------------------>8---------------------------------
> From 0e2fc443c760290166f6371d50813e6c30a678da Mon Sep 17 00:00:00 2001
> From: Lucas Stach <l.stach@pengutronix.de>
> Date: Thu, 17 Jul 2014 18:54:44 +0200
> Subject: [PATCH] try forcing LTSSM into detect state
> 
> ---
>  drivers/pci/host/pci-imx6.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c
> index a568efaa331c..afa450c54aba 100644
> --- a/drivers/pci/host/pci-imx6.c
> +++ b/drivers/pci/host/pci-imx6.c
> @@ -215,6 +215,8 @@ static int imx6_pcie_assert_core_reset(struct pcie_port
> *pp) {
>  	struct imx6_pcie *imx6_pcie = to_imx6_pcie(pp);
> 
> +	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR12,
> +			IMX6Q_GPR12_PCIE_CTL_2, 0 << 10);
>  	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
>  			IMX6Q_GPR1_PCIE_TEST_PD, 1 << 18);
>  	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,

Tried this, among many other things, but please let someone else also re-verify 
this. What I observed is that the LTSSM cannot leave the POLLING phase .

Re: [RFC] PCI: pci-imx6: Add delay to workaround kernel hang

["David Müller (ELSOFT AG)"]

Hello

Lucas Stach wrote:
> @@ -215,6 +215,8 @@ static int imx6_pcie_assert_core_reset(struct pcie_port *pp)
>  {
>  	struct imx6_pcie *imx6_pcie = to_imx6_pcie(pp);
>  
> +	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR12,
> +			IMX6Q_GPR12_PCIE_CTL_2, 0 << 10);
>  	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
>  			IMX6Q_GPR1_PCIE_TEST_PD, 1 << 18);
>  	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
> 

Sorry, this doesn't seem to fix the problem (at least not with my
hardware).

Dave

[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

["David Müller (ELSOFT AG)"]

Hello

Lucas Stach wrote:
> @@ -215,6 +215,8 @@ static int imx6_pcie_assert_core_reset(struct pcie_port *pp)
>  {
>  	struct imx6_pcie *imx6_pcie = to_imx6_pcie(pp);
>  
> +	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR12,
> +			IMX6Q_GPR12_PCIE_CTL_2, 0 << 10);
>  	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
>  			IMX6Q_GPR1_PCIE_TEST_PD, 1 << 18);
>  	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
> 

Sorry, this doesn't seem to fix the problem (at least not with my
hardware).

Dave

Re: [RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Marek Vasut]

On Thursday, July 17, 2014 at 05:27:09 PM, Shawn Guo wrote:
> On Thu, Jul 17, 2014 at 10:23:10AM +0200, Marek Vasut wrote:
> > On Thursday, July 17, 2014 at 08:51:48 AM, Uwe Kleine-König wrote:
> > > Hello,
> > > 
> > > On Tue, Jun 24, 2014 at 04:18:27PM -0300, Fabio Estevam wrote:
> > > > From: Fabio Estevam <fabio.estevam@freescale.com>
> > > > 
> > > > When the mx6 PCI conctroller is initialized in the bootloader we see
> > > > a kernel hang inside imx6_add_pcie_port().
> > > > 
> > > > Adding a 30ms delay allows the kernel to boot.
> > > 
> > > Just my thought on how to debug that: I'd try to bisect the pci init
> > > routine in the boot loader. I.e. first only do the first half of the
> > > initialisation in U-Boot. Depending on Linux being able to boot or not
> > > initialize more or less on the next run.
> > > 
> > > Maybe there is a single register write that makes Linux fail?!
> > 
> > I am still hell-bent on thinking that the missing PCIe block reset is
> > what makes the Linux fail. Missing block reset is always a problem.
> 
> Indeed.  We're missing a hardware reset for PCIe on i.MX6Q and i.MX6DL.
> Such reset is available on i.MX6SX, so there is no this problem for
> i.MX6SX PCIe.
> 
> > Or do we now have a
> > mean to reset the PCIe block and it's PHY from software?
> 
> Richard is trying to find a SW workaround for it, but we're not really
> sure if it's possible.

I hate to ask this, but does that mean all but MX6SLX are irrepairably broken 
and will never have a reliable PCIe implementation ever?

Best regards,
Marek Vasut

[RFC] PCI: pci-imx6: Add delay to workaround kernel hang

[Marek Vasut]

On Thursday, July 17, 2014 at 05:27:09 PM, Shawn Guo wrote:
> On Thu, Jul 17, 2014 at 10:23:10AM +0200, Marek Vasut wrote:
> > On Thursday, July 17, 2014 at 08:51:48 AM, Uwe Kleine-K?nig wrote:
> > > Hello,
> > > 
> > > On Tue, Jun 24, 2014 at 04:18:27PM -0300, Fabio Estevam wrote:
> > > > From: Fabio Estevam <fabio.estevam@freescale.com>
> > > > 
> > > > When the mx6 PCI conctroller is initialized in the bootloader we see
> > > > a kernel hang inside imx6_add_pcie_port().
> > > > 
> > > > Adding a 30ms delay allows the kernel to boot.
> > > 
> > > Just my thought on how to debug that: I'd try to bisect the pci init
> > > routine in the boot loader. I.e. first only do the first half of the
> > > initialisation in U-Boot. Depending on Linux being able to boot or not
> > > initialize more or less on the next run.
> > > 
> > > Maybe there is a single register write that makes Linux fail?!
> > 
> > I am still hell-bent on thinking that the missing PCIe block reset is
> > what makes the Linux fail. Missing block reset is always a problem.
> 
> Indeed.  We're missing a hardware reset for PCIe on i.MX6Q and i.MX6DL.
> Such reset is available on i.MX6SX, so there is no this problem for
> i.MX6SX PCIe.
> 
> > Or do we now have a
> > mean to reset the PCIe block and it's PHY from software?
> 
> Richard is trying to find a SW workaround for it, but we're not really
> sure if it's possible.

I hate to ask this, but does that mean all but MX6SLX are irrepairably broken 
and will never have a reliable PCIe implementation ever?

Best regards,
Marek Vasut