Re: [PATCH] arm64: dts: rockchip: Fix rk3328-roc-cc sdmmcio-regulator

[Adam Van Ymeren <adam@vany.ca>]

On 2020-02-10 8:37 a.m., Robin Murphy wrote:
> On 09/02/2020 1:07 am, Adam Van Ymeren wrote:
>> Got it working. Robin you were right, it just needed enable-active-high;
>> added to the regulator, updated patch at the end of this message.
>
> Weird... after my initial excitement I went back and looked more
> closely at the binding and gpiolib-of code, only to realise that that
> property is only supposed to apply to specific 'enable' GPIOs (of
> which there are none here since it's always-on), and that the 'state'
> GPIOs were a separate thing and I'd jumped to the wrong conclusion. So
> now I'm doubly surprised that it actually makes a difference :/

Yeah you're right, it doesn't make a difference, it was just
coincidence.  It just seems to be really flaky, it's booting
successfully about 20-30% of the time, I kept drawing conclusions based
on one or two successful boots, but from now on I'll try 10 boots
before  With no changes and just unplugging/replugging the board, the
kernel only finds mmcblk1 20-30% of the time.  What's weird though is
that the first stage idbloader and u-boot so far have worked 100% of the
time on the same sdcard.


As I said early in the thread, if I remove vcc_sdio/vqmmc from the
device tree, then the kernel finds mmcblk1 every time I've attempted it
so far, although I haven't done extensive tests to see if i/o is
stable.  So there seems to be _something_ about how the kernel is
interacting with the regulator and/or the mmc host, otherwise I see no
explanation for why the kernel would fail where u-boot had succeeded.


I logged writes to register 0x428, and voltage changes in
gpio-regulator, with the default device tree when asking for 3.3V it
sets 0x428 to 0x20000, which is as expected, I verified the output of
the voltage regulator with a multimeter and it does appear to be 3.3V. 
I notice however that early in the gpio setup it first writes 0x20002 to
0x428 before later writing 0x20000 what initializing the mmc driver


[    0.545007] ADAMVY regmap 428 <= 20002
[    0.549348] dwmmc_rockchip ff500000.dwmmc: IDMAC supports 32-bit
address mode.
[    0.550012] dwmmc_rockchip ff500000.dwmmc: Using internal DMA controller.
[    0.550615] dwmmc_rockchip ff500000.dwmmc: Version ID is 270a
[    0.551154] dwmmc_rockchip ff500000.dwmmc: DW MMC controller at irq
29,32 bit host data width,256 deep fifo
[    0.552527] ADAMVY: gpio_regulator_set_voltage vcc_sdio: uv: 3300000
state: 0
[    0.553158] ADAMVY regmap 428 <= 20000
[    0.566133] mmc_host mmc1: Bus speed (slot 0) = 400000Hz (slot req
400000Hz, actual 400000HZ div = 0)


My best guess is there's some sort of race condition?  Maybe a capacitor
is still charging and holding the voltage low just long enough that the
kernel gives up?  Or maybe the kernel is just not waiting long enough
for the mmc host to finish initializing.  In about 50% of my failed
boots I see this debug messages about mmc1 mixed into the kernel panic
like so.

[    0.662428] VFS: Cannot open root device "mmcblk1p5" or
unknown-block(0,0): error -6
[    0.663107] Please append a correct "root=" boot option; here are the
available partitions:
[    0.663847] Kernel panic - not syncing: VFS: Unable to mount root fs
on unknown-block(0,0)
[    0.663872] mmc1: starting CMD6 arg 00fffff0 flags 000000b5
[    0.664578] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.5.0 #37
[    0.665061] mmc1:     blksz 64 blocks 1 flags 00000200 tsac 100 ms nsac 0
[    0.665572] Hardware name: Firefly roc-rk3328-cc (DT)
[    0.666600] Call trace:
[    0.666832]  dump_backtrace+0x0/0x1b4
[    0.667161]  show_stack+0x14/0x1c
[    0.667461]  dump_stack+0xb4/0x100
[    0.667768]  panic+0x120/0x2f4
[    0.668046]  mount_block_root+0x2a0/0x328
[    0.668401]  mount_root+0x7c/0x88
[    0.668457] mmc1: req done (CMD6): 0: 00000900 00000000 00000000 00000000
[    0.668700]  prepare_namespace+0x154/0x164
[    0.669288] mmc1:     64 bytes transferred: 0
[    0.669642]  kernel_init_freeable+0x1d0/0x254
[    0.670040] mmc1: clock 25000000Hz busmode 2 powermode 2 cs 0 Vdd 21
width 1 timing 0
[    0.670404]  kernel_init+0x10/0xf8
[    0.671131] mmc_host mmc1: Bus speed (slot 0) = 25000000Hz (slot req
25000000Hz, actual 25000000HZ div = 0)
[    0.671378]  ret_from_fork+0x10/0x18
[    0.672230] mmc1: starting CMD55 arg af530000 flags 00000095

suggesting that the kernel is still doing stuff with mmc1 after it's
already given up on finding the root device.


Here's dmesg from a successful boot | grep mmc


$ grep mmc success.txt
[    0.000000] Kernel command line: earlycon=uart8250,mmio32,0xff130000
console=ttyS2,1500000 rw root=/dev/mmcblk1p5 init=/sbin/init
kgdboc=ttyS2,1500000 loglevel=7 dyndbg="module dw_mmc +p ; module
dw_mmc_rockchip +p ; module mmc_core +p"
[    0.123714] ADAMVY gpio fixed regulator quirks device:
sdmmc-regulator active_low: 1
[    0.485215] dwmmc_rockchip ff500000.dwmmc: IDMAC supports 32-bit
address mode.
[    0.485879] dwmmc_rockchip ff500000.dwmmc: Using internal DMA controller.
[    0.486482] dwmmc_rockchip ff500000.dwmmc: Version ID is 270a
[    0.487054] dwmmc_rockchip ff500000.dwmmc: DW MMC controller at irq
29,32 bit host data width,256 deep fifo
[    0.489495] dwmmc_rockchip ff520000.dwmmc: IDMAC supports 32-bit
address mode.
[    0.490153] dwmmc_rockchip ff520000.dwmmc: Using internal DMA controller.
[    0.490758] dwmmc_rockchip ff520000.dwmmc: Version ID is 270a
[    0.491313] dwmmc_rockchip ff520000.dwmmc: DW MMC controller at irq
30,32 bit host data width,256 deep fifo
[    0.492952] mmc_host mmc0: card is non-removable.
[    0.506161] mmc_host mmc0: Bus speed (slot 0) = 400000Hz (slot req
400000Hz, actual 400000HZ div = 0)
[    0.559169] dwmmc_rockchip ff500000.dwmmc: IDMAC supports 32-bit
address mode.
[    0.559834] dwmmc_rockchip ff500000.dwmmc: Using internal DMA controller.
[    0.560437] dwmmc_rockchip ff500000.dwmmc: Version ID is 270a
[    0.560980] dwmmc_rockchip ff500000.dwmmc: DW MMC controller at irq
29,32 bit host data width,256 deep fifo
[    0.575968] mmc_host mmc1: Bus speed (slot 0) = 400000Hz (slot req
400000Hz, actual 400000HZ div = 0)
[    0.650647] mmc_host mmc1: Bus speed (slot 0) = 25000000Hz (slot req
25000000Hz, actual 25000000HZ div = 0)
[    0.657873] mmc1: new SD card at address af53
[    0.659091] mmcblk1: mmc1:af53 SU02G 1.84 GiB
[    0.677231]  mmcblk1: p1 p2 p3 p4 p5
[    0.695708] EXT4-fs (mmcblk1p5): warning: mounting unchecked fs,
running e2fsck is recommended
[    0.700796] EXT4-fs (mmcblk1p5): mounted filesystem without journal.
Opts: (null)
[    1.029332] dwmmc_rockchip ff520000.dwmmc: Busy; trying anyway
[    1.529858] mmc_host mmc0: Timeout sending command (cmd 0x202000 arg
0x0 status 0x80202000)
[    1.544526] mmc_host mmc0: Bus speed (slot 0) = 300000Hz (slot req
300000Hz, actual 300000HZ div = 0)
[    2.067046] dwmmc_rockchip ff520000.dwmmc: Busy; trying anyway
[    2.567565] mmc_host mmc0: Timeout sending command (cmd 0x202000 arg
0x0 status 0x80202000)
[    2.582199] mmc_host mmc0: Bus speed (slot 0) = 200000Hz (slot req
200000Hz, actual 200000HZ div = 0)
[    3.107353] dwmmc_rockchip ff520000.dwmmc: Busy; trying anyway
[    3.607889] mmc_host mmc0: Timeout sending command (cmd 0x202000 arg
0x0 status 0x80202000)
[    3.622620] mmc_host mmc0: Bus speed (slot 0) = 100000Hz (slot req
100000Hz, actual 100000HZ div = 0)
[    4.155300] dwmmc_rockchip ff520000.dwmmc: Busy; trying anyway
[    4.655839] mmc_host mmc0: Timeout sending command (cmd 0x202000 arg
0x0 status 0x80202000)

and for a failed boot

$ grep mmc fail.txt
[    0.000000] Kernel command line: earlycon=uart8250,mmio32,0xff130000
console=ttyS2,1500000 rw root=/dev/mmcblk1p5 init=/sbin/init
kgdboc=ttyS2,1500000 loglevel=7 dyndbg="module dw_mmc +p ; module
dw_mmc_rockchip +p ; module mmc_core +p"
[    0.123799] ADAMVY gpio fixed regulator quirks device:
sdmmc-regulator active_low: 1
[    0.485295] dwmmc_rockchip ff500000.dwmmc: IDMAC supports 32-bit
address mode.
[    0.485955] dwmmc_rockchip ff500000.dwmmc: Using internal DMA controller.
[    0.486560] dwmmc_rockchip ff500000.dwmmc: Version ID is 270a
[    0.487128] dwmmc_rockchip ff500000.dwmmc: DW MMC controller at irq
29,32 bit host data width,256 deep fifo
[    0.489584] dwmmc_rockchip ff520000.dwmmc: IDMAC supports 32-bit
address mode.
[    0.490243] dwmmc_rockchip ff520000.dwmmc: Using internal DMA controller.
[    0.490849] dwmmc_rockchip ff520000.dwmmc: Version ID is 270a
[    0.491400] dwmmc_rockchip ff520000.dwmmc: DW MMC controller at irq
30,32 bit host data width,256 deep fifo
[    0.493043] mmc_host mmc0: card is non-removable.
[    0.506248] mmc_host mmc0: Bus speed (slot 0) = 400000Hz (slot req
400000Hz, actual 400000HZ div = 0)
[    0.559249] dwmmc_rockchip ff500000.dwmmc: IDMAC supports 32-bit
address mode.
[    0.559960] dwmmc_rockchip ff500000.dwmmc: Using internal DMA controller.
[    0.560564] dwmmc_rockchip ff500000.dwmmc: Version ID is 270a
[    0.561108] dwmmc_rockchip ff500000.dwmmc: DW MMC controller at irq
29,32 bit host data width,256 deep fifo
[    1.063521] dwmmc_rockchip ff520000.dwmmc: Busy; trying anyway
[    1.564049] mmc_host mmc0: Timeout sending command (cmd 0x202000 arg
0x0 status 0x80202000)
[    1.564925] mmc_host mmc1: Bus speed (slot 0) = 400000Hz (slot req
400000Hz, actual 400000HZ div = 0)
[    1.579671] mmc_host mmc0: Bus speed (slot 0) = 300000Hz (slot req
300000Hz, actual 300000HZ div = 0)
[    2.109484] dwmmc_rockchip ff520000.dwmmc: Busy; trying anyway
[    2.610015] mmc_host mmc0: Timeout sending command (cmd 0x202000 arg
0x0 status 0x80202000)
[    2.612411] VFS: Cannot open root device "mmcblk1p5" or
unknown-block(0,0): error -6


>
>> I went back over my config with a fine tooth comb and found a few
>> rockchip drivers I was missing including PINCTRL_RK805 which seems
>> related but I honestly can't figure out how.  I will try to narrow down
>> which specific driver was missing.  It was odd because it would find the
>> mmc host but just fail notice that a card was present.  It seemed to set
>> the register correctly and identify the mmc host, but failed to notice
>> any card present.
>
> PINCTRL_RK805 should only be for the couple of GPIO pins on the PMIC
> itself which IIRC the reference design uses for the ethernet LEDs, so
> I wouldn't expect it to be relevant.
Indeed.  I was drawing connections that weren't there due to the flakiness.
>
> Those symptoms make sense for the voltages being backwards (or just
> stuck at 1.8V) though - when the I/O domain is configured to expect
> 3.3V, 1.8V is too low to register as a logic high, so Linux will
> always think that something is inserted in the slot, but not talking back.

Hmm.  I tried adding broken-cd, but I couldn't tell any significant
difference in the rate of successful boots.

I have an emmc module coming so hopefully I can have something reliable
to boot from and can iterate faster without having to physically move an
sdcard back and forth every time.

> Robin.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel