All of lore.kernel.org
 help / color / mirror / Atom feed
* RZ/G2M Hangs when booting some SD cards
@ 2022-04-11 13:07 Adam Ford
  2022-04-11 19:11 ` Wolfram Sang
  0 siblings, 1 reply; 9+ messages in thread
From: Adam Ford @ 2022-04-11 13:07 UTC (permalink / raw)
  To: linux-mmc, Wolfram Sang, Ulf Hansson, Geert Uytterhoeven

I have a small collection of microSD cards that appear to hang when
booting.  It affects Renesas' 4.19 CIP kernel as well as 5.18.0-rc2
with the following:

[    2.717601] mmc1: tuning execution failed: -5
[    2.721979] mmc1: error -5 whilst initialising SD card
[    2.839001] mmc1: tuning execution failed: -5
[    2.843378] mmc1: error -5 whilst initialising SD card

I can use these same cards on a different platform like NXP IXM8M, and
they enumerate and mount just fine on a Linux machine, and my boot
vFAT partition mounts just fine on a Windows machine, so it seems
limited to just Renesas on Linux.

I did some testing, and I found that U-Boot on the same board can also
properly initialize the microSD card and it emuerates at sdr104.

Manufacturer ID: 1d
OEM: 4144
Name: USDBus Speed: 199999992
Mode: UHS SDR104 (208MHz)
Rd Block Len: 512
SD version 3.0
High Capacity: Yes
Capacity: 29.8 GiB
Bus Width: 4-bit
Erase Group Size: 512 Bytes
=>

This lead me to believe that U-Boot is initializing the micro SD card
differently than Linux, so I investigated the drivers in both to
compare what's happening differently.

The drivers between U-Boot [1] and Linux [2] appear very similar to
each other, but I noticed that U-Boot does something that Linux does
not do.

In U-Boot, renesas_sdhi_select_tuning checks a series of bits in
priv->smcmp and clears them.

renesas_sdhi_select_tuning()
<snip>

for (i = 0; i < priv->tap_num * 2; i++) {
    if (!(taps & BIT(i))) {
        taps &= ~BIT(i % priv->tap_num);
        taps &= ~BIT((i % priv->tap_num) + priv->tap_num);
    }
    if (!(priv->smpcmp & BIT(i))) {
        priv->smpcmp &= ~BIT(i % priv->tap_num);
        priv->smpcmp &= ~BIT((i % priv->tap_num) + priv->tap_num);
    }
}

Later in the code, it checks to see if various bits in priv->smcmp are set

for (i = 0; i < priv->tap_num * 2; i++) {
    if (priv->smpcmp & BIT(i))
        ntap++;
    else {
        if (ntap > match_cnt) {
            tap_start = i - ntap;
            tap_end = i - 1;
            match_cnt = ntap;
        }

<snip>

In Linux, the driver is split into multiple parts, so it's a little
harder to follow, but it appears to do something similar in
renesas_sdhi_select_tuning where it checks bits and clears various
bits in priv-smpcmp:

for (i = 0; i < taps_size; i++) {
    int offset = priv->tap_num * (i < priv->tap_num ? 1 : -1);

    if (!test_bit(i, priv->taps))
        clear_bit(i + offset, priv->taps);

    if (!test_bit(i, priv->smpcmp))
        clear_bit(i + offset, priv->smpcmp);
}

So far, these look fairly similar.

However,  a bit later, the code that checks the smcmp in U-Boot
appears dramatically different in Linux, and I don't understand the
tuning phase nor do  I have the sd card spec.

In U-BOot there is a comment (with some corresponding code) that states:

/*
* Find the longest consecutive run of successful probes.  If that
* is more than RENESAS_SDHI_MAX_TAP probes long then use the
* center index as the tap.
*/

This chunk of code appears drastically different from the Linux side,
but maybe it's ok.

Then later, U-Boot has a for loop which either increases the ntap if
priv->smpcmp & BIT(it) is set or does some math on adjusting
tap_start, end and match_cnt.  It's covered by a comment that reads:

/*
* If all of the TAP is OK, the sampling clock position is selected by
* identifying the change point of data.
*/

I don't understand the logic on the Linux side, but when we get down a
bit further, we end with

if (tap_cnt >= min_tap_row)
    priv->tap_set = (tap_start + tap_end) / 2 % priv->tap_num;
else
    return -EIO;

Which I believe yields the failure  of

"error -5 whilst initialising SD card" that I mentioned above.

I was thinking about taking the math from the U-Boot and porting it to
Linux to see if the issue goes away, but before I did that, I was
hoping someone might have some insights to see if that's the right
direction to go.

thanks

adam

[1] - https://source.denx.de/u-boot/u-boot/-/blob/master/drivers/mmc/renesas-sdhi.c
[2] - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/mmc/host/renesas_sdhi_core.c?h=v5.18-rc2

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: RZ/G2M Hangs when booting some SD cards
  2022-04-11 13:07 RZ/G2M Hangs when booting some SD cards Adam Ford
@ 2022-04-11 19:11 ` Wolfram Sang
  2022-04-11 19:38   ` Adam Ford
  0 siblings, 1 reply; 9+ messages in thread
From: Wolfram Sang @ 2022-04-11 19:11 UTC (permalink / raw)
  To: Adam Ford; +Cc: linux-mmc, Ulf Hansson, Geert Uytterhoeven

[-- Attachment #1: Type: text/plain, Size: 654 bytes --]

Hi,

> I was thinking about taking the math from the U-Boot and porting it to
> Linux to see if the issue goes away, but before I did that, I was
> hoping someone might have some insights to see if that's the right
> direction to go.

Thank you very much for your detailed report!

I have one SanDisk card which sometimes fails to probe on one specific
board, so for me it was really hard to debug the issue because it was
hardly reproducible. Can you tell me which cards fail and if they do
that realiably? Also which Renesas HW do you use?

So much for a first response. I will have a closer look at your findings
tomorrow.

Thanks again,

   Wolfram


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: RZ/G2M Hangs when booting some SD cards
  2022-04-11 19:11 ` Wolfram Sang
@ 2022-04-11 19:38   ` Adam Ford
  2022-04-12  9:04     ` Wolfram Sang
  0 siblings, 1 reply; 9+ messages in thread
From: Adam Ford @ 2022-04-11 19:38 UTC (permalink / raw)
  To: Wolfram Sang, Adam Ford, linux-mmc, Ulf Hansson, Geert Uytterhoeven

On Mon, Apr 11, 2022 at 2:11 PM Wolfram Sang
<wsa+renesas@sang-engineering.com> wrote:
>
> Hi,
>
> > I was thinking about taking the math from the U-Boot and porting it to
> > Linux to see if the issue goes away, but before I did that, I was
> > hoping someone might have some insights to see if that's the right
> > direction to go.
>
> Thank you very much for your detailed report!
>
> I have one SanDisk card which sometimes fails to probe on one specific
> board, so for me it was really hard to debug the issue because it was
> hardly reproducible. Can you tell me which cards fail and if they do
> that realiably? Also which Renesas HW do you use?
>

I have cards that fail 100% on my RZ//G2M.  I have an N and H as well,
but I haven't tested them yet.

Unfortunately, a colleague of mine tested positive for covid-19, which
prompted several of us to get tests, and I received a positive test,
so I cannot go back to the office for at least 5 days.  (I am feeling
fine today)
Because I cannot go into the office I can't get the exact part number,
but the card I have is a Patriot card, but a colleague of mine
reported another brand was failing the same way.

I was given permission to ship one out if you're interested but I have
to wait until next week to test anything or ship anything. Please
don't feel obligated to accept the card,but I know it can be easier to
have actual hardware.

> So much for a first response. I will have a closer look at your findings

Thanks for responding.  I'll try to get card info / part number from
my colleague.

adam

> tomorrow.
>
> Thanks again,
>
>    Wolfram
>

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: RZ/G2M Hangs when booting some SD cards
  2022-04-11 19:38   ` Adam Ford
@ 2022-04-12  9:04     ` Wolfram Sang
  2022-04-12 15:01       ` Wolfram Sang
  0 siblings, 1 reply; 9+ messages in thread
From: Wolfram Sang @ 2022-04-12  9:04 UTC (permalink / raw)
  To: Adam Ford; +Cc: linux-mmc, Ulf Hansson, Geert Uytterhoeven

[-- Attachment #1: Type: text/plain, Size: 1356 bytes --]

Hi Adam,

> I have cards that fail 100% on my RZ//G2M.  I have an N and H as well,
> but I haven't tested them yet.

That would be helpful to do. The board which fails for me is Salvator-X
with an M3-W SoC, so that matches your G2M. The other SoCs work fine for
me.

> Unfortunately, a colleague of mine tested positive for covid-19, which
> prompted several of us to get tests, and I received a positive test,
> so I cannot go back to the office for at least 5 days.  (I am feeling
> fine today)

Oh, get well and negative soon then!

> I was given permission to ship one out if you're interested but I have
> to wait until next week to test anything or ship anything. Please
> don't feel obligated to accept the card,but I know it can be easier to
> have actual hardware.

That would be awesome if you could send me a card which regularly fails.
Thank you! I'll try to reproduce the issue again today but if it fails,
then I'll wait for the card to arrive.

> Thanks for responding.  I'll try to get card info / part number from
> my colleague.

Great, thanks. It will be interesting to know. My experience with cards
is that they may differ from one batch to the other, though. So, it is
not guaranteed that I'll see the same issues when I purchase card with
the same name.

Anyhow, I'll report back with results from today.

All the best,

   Wolfram


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: RZ/G2M Hangs when booting some SD cards
  2022-04-12  9:04     ` Wolfram Sang
@ 2022-04-12 15:01       ` Wolfram Sang
  2022-04-12 21:38         ` Adam Ford
  2022-04-19 12:50         ` Adam Ford
  0 siblings, 2 replies; 9+ messages in thread
From: Wolfram Sang @ 2022-04-12 15:01 UTC (permalink / raw)
  To: Adam Ford, linux-mmc, Ulf Hansson, Geert Uytterhoeven

[-- Attachment #1: Type: text/plain, Size: 233 bytes --]


> Anyhow, I'll report back with results from today.

So, no news here. When the board is "cold" (hasn't been used for some
days), probing the SD card occasionally fails, but after some reboots it
just works. I don't see a pattern.


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: RZ/G2M Hangs when booting some SD cards
  2022-04-12 15:01       ` Wolfram Sang
@ 2022-04-12 21:38         ` Adam Ford
  2022-04-19 12:50         ` Adam Ford
  1 sibling, 0 replies; 9+ messages in thread
From: Adam Ford @ 2022-04-12 21:38 UTC (permalink / raw)
  To: Wolfram Sang, Adam Ford, linux-mmc, Ulf Hansson, Geert Uytterhoeven

On Tue, Apr 12, 2022 at 10:01 AM Wolfram Sang
<wsa+renesas@sang-engineering.com> wrote:
>
>
> > Anyhow, I'll report back with results from today.
>
> So, no news here. When the board is "cold" (hasn't been used for some
> days), probing the SD card occasionally fails, but after some reboots it
> just works. I don't see a pattern.

Thanks for looking into it.  I should be back in my office next week,
and I am going to add a bunch of debug code to U-Boot to get an idea
of that's going on in the tuning to see what the various registers
look like.  I'll then add some similar debugging to see how the same
registers compare in Linux.

Hopefully that will give us some ideas of what might be happening
differently between them.

adam
>

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: RZ/G2M Hangs when booting some SD cards
  2022-04-12 15:01       ` Wolfram Sang
  2022-04-12 21:38         ` Adam Ford
@ 2022-04-19 12:50         ` Adam Ford
  2023-05-31  6:51           ` Wolfram Sang
  1 sibling, 1 reply; 9+ messages in thread
From: Adam Ford @ 2022-04-19 12:50 UTC (permalink / raw)
  To: Wolfram Sang, Adam Ford, linux-mmc, Ulf Hansson, Geert Uytterhoeven

On Tue, Apr 12, 2022 at 10:01 AM Wolfram Sang
<wsa+renesas@sang-engineering.com> wrote:
>
>
> > Anyhow, I'll report back with results from today.
>
> So, no news here. When the board is "cold" (hasn't been used for some
> days), probing the SD card occasionally fails, but after some reboots it
> just works. I don't see a pattern.

From what I have been able to determine, the U-Boot code explicitly
disables DMA during the tuning phase:

<snip>
/* Force PIO for the tuning */
caps = priv->caps;
priv->caps &= ~TMIO_SD_CAP_DMA_INTERNAL;

ret = mmc_send_tuning(mmc, opcode, NULL);

priv->caps = caps;

if (ret == 0)
taps |= BIT(i);
<snip>

When doing this, the value of taps is non-zero.  For this card, it is fefe
In U-Boot smpcmp is also the save value.

However, Linux does not disable the DMA, and the value of taps is zero
and the tuning ultimately fails.

I did some experiments to attempt to disable it, but when I do, it
usually ends up hanging or a kernel panic.

Does anyone have a suggestion on how I can properly attempt to disable
the DMA and use PIO during the tuning phase to see if this fixes the
micro SD card initialization issue I'm seeing.

If I manually set taps  to match priv->smpcmp when taps=0, the microSD
card enumerates correctly.

thanks

adam

>

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: RZ/G2M Hangs when booting some SD cards
  2022-04-19 12:50         ` Adam Ford
@ 2023-05-31  6:51           ` Wolfram Sang
  2023-05-31 10:23             ` Adam Ford
  0 siblings, 1 reply; 9+ messages in thread
From: Wolfram Sang @ 2023-05-31  6:51 UTC (permalink / raw)
  To: Adam Ford; +Cc: linux-mmc, Ulf Hansson, Geert Uytterhoeven

[-- Attachment #1: Type: text/plain, Size: 807 bytes --]

Hi Adam,

I am very sorry that it took so long until I am now able to respond.
First, too many other duties with higher priority came in. Second, the
card didn't fail 100% for me after all which made debugging still
challenging. But it failed often enough, so I at least have a pointer
now.

> From what I have been able to determine, the U-Boot code explicitly
> disables DMA during the tuning phase:

...

> However, Linux does not disable the DMA, and the value of taps is zero
> and the tuning ultimately fails.

I disabled DMA while tuning but it did not help. It finally turns out
that adding a delay after each tuning command seems to help. U-Boot does
this but Marek (the driver author for U-Boot) cannot recall why he added
the delay. I'll send a patch in some minutes.

All the best,

   Wolfram


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: RZ/G2M Hangs when booting some SD cards
  2023-05-31  6:51           ` Wolfram Sang
@ 2023-05-31 10:23             ` Adam Ford
  0 siblings, 0 replies; 9+ messages in thread
From: Adam Ford @ 2023-05-31 10:23 UTC (permalink / raw)
  To: Wolfram Sang, Adam Ford, linux-mmc, Ulf Hansson, Geert Uytterhoeven

On Wed, May 31, 2023 at 1:51 AM Wolfram Sang
<wsa+renesas@sang-engineering.com> wrote:
>
> Hi Adam,
>
> I am very sorry that it took so long until I am now able to respond.
> First, too many other duties with higher priority came in. Second, the
> card didn't fail 100% for me after all which made debugging still
> challenging. But it failed often enough, so I at least have a pointer
> now.

I totally understand that it's an open source effort, and we have have
duties to perform. I do appreciate the help.

>
> > From what I have been able to determine, the U-Boot code explicitly
> > disables DMA during the tuning phase:
>
> ...
>
> > However, Linux does not disable the DMA, and the value of taps is zero
> > and the tuning ultimately fails.
>
> I disabled DMA while tuning but it did not help. It finally turns out
> that adding a delay after each tuning command seems to help. U-Boot does
> this but Marek (the driver author for U-Boot) cannot recall why he added
> the delay. I'll send a patch in some minutes.

I'll give them a try today.

adam
>
> All the best,
>
>    Wolfram
>

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2023-05-31 10:24 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-04-11 13:07 RZ/G2M Hangs when booting some SD cards Adam Ford
2022-04-11 19:11 ` Wolfram Sang
2022-04-11 19:38   ` Adam Ford
2022-04-12  9:04     ` Wolfram Sang
2022-04-12 15:01       ` Wolfram Sang
2022-04-12 21:38         ` Adam Ford
2022-04-19 12:50         ` Adam Ford
2023-05-31  6:51           ` Wolfram Sang
2023-05-31 10:23             ` Adam Ford

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.