Regression: dmaengine: imx28 with emmc

Regression: dmaengine: imx28 with emmc

[Bruno Thomsen]

Hi

I am getting a kernel oops[1] during boot on imx28 with emmc flash right
around rootfs mounting. Using git bisect I found the cause to be the
following commit.

Regression: ceeeb99cd821 ("dmaengine: mxs: rename custom flag")

Reverting the 2 changes in drivers/dma/mxs-dma.c fixes the oops,
but I am not sure that is the right solution as I don't have the full
mxs-dma + mtd/mmc overview.

I did see that the patch isn't a simple rename but also a bit define
change.
From: DMA_CTRL_ACK = (1 << 1) = BIT(1)
To: MXS_DMA_CTRL_WAIT4END = BIT(31)

Bruno

[1] One of the kernel oops during git bisect (dirty part is just a dts).

[    2.326402] 8<--- cut here ---
[    2.329575] Unable to handle kernel NULL pointer dereference at
virtual address 00000252
[    2.337988] pgd = (ptrval)
[    2.340793] [00000252] *pgd=00000000
[    2.344483] Internal error: Oops: 5 [#1] ARM
[    2.348790] Modules linked in:
[    2.351904] CPU: 0 PID: 21 Comm: kworker/0:1 Not tainted
5.2.0-10687-g5fe7b600a116-dirty #6
[    2.360283] Hardware name: Freescale MXS (Device Tree)
[    2.365480] Workqueue: events_freezable mmc_rescan
[    2.370324] PC is at mmc_wait_for_cmd+0x30/0xa4
[    2.374893] LR is at mmc_wait_for_cmd+0x30/0xa4
[    2.379453] pc : [<c0389608>]    lr : [<c0389608>]    psr: 40000013
[    2.385748] sp : cf5d9d20  ip : cf5d9d9c  fp : cf73ca64
[    2.391000] r10: 00000001  r9 : cf68f2b0  r8 : 00200000
[    2.396254] r7 : 00000000  r6 : ffffff92  r5 : c0701068  r4 : cf5d9db8
[    2.402810] r3 : 00000000  r2 : fffffffc  r1 : 00000000  r0 : cf5d9d20
[    2.409368] Flags: nZcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[    2.416537] Control: 0005317f  Table: 40004000  DAC: 00000053
[    2.422315] Process kworker/0:1 (pid: 21, stack limit = 0x(ptrval))
[    2.428611] Stack: (0xcf5d9d20 to 0xcf5da000)
[    2.433016] 9d20: 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[    2.441237] 9d40: 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[    2.449460] 9d60: 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[    2.457680] 9d80: 00000000 00000000 00000000 00000000 00000000
00000000 00000000 3a80845c
[    2.465903] 9da0: 0000000a c0701068 ffffff92 00200000 0000000a
c0392b0c 00000008 000001aa
[    2.474125] 9dc0: 00000000 00000000 00000000 00000000 000002f5
00000000 00000000 00000000
[    2.482347] 9de0: 00000000 00000000 00000000 3a80845c ffffff92
00200000 00000000 c0391a28
[    2.490569] 9e00: 00000000 cf68f000 cf73c800 cf73ca64 00200000
00200000 0000000a 00000000
[    2.498795] 9e20: c0701068 c03937b0 cf5d9e3c 3a80845c cf5d9e54
c030466c c0634ef0 c06484f0
[    2.507021] 9e40: cf4f4680 cf5d9e54 c0701068 c03048c0 ffffff92
ffffff3f cf5d9e50 3a80845c
[    2.515244] 9e60: c0760fc0 cf68f338 00000001 cf68f000 cf68f000
c0760fc0 c0525400 c0701068
[    2.523466] 9e80: 00000000 c0394460 00000008 000001aa ffffff3f
00000000 00000000 00000000
[    2.531689] 9ea0: ffffff3f 3a80845c 00000000 cf68f338 00000001
cf68f000 ffffff92 c0760fc0
[    2.539915] 9ec0: c0525400 c0525410 c0760fb0 c038b47c 00000000
c0708ca4 cf68f338 cf68f338
[    2.548139] 9ee0: cf5aa540 c0708ca4 cfde0600 cf5d9f1c c075a9f9
c075b414 c0701068 c0035298
[    2.556363] 9f00: 00000001 00000000 c0035200 c04eb0c8 00000000
00000000 c00356e8 c0dceb08
[    2.564588] 9f20: c0bbe570 00000000 c06467a0 3a80845c c0708ca4
cf5aa540 c0708ca4 cf5aa554
[    2.572811] 9f40: c0708cd8 ffffe000 c070ff60 00000008 c0708ca4
c0035644 60000053 cf5cb244
[    2.581035] 9f60: ffffe000 cf5b7280 cf5a3fc0 00000000 cf5d8000
cf5aa540 c0035618 cf49be90
[    2.589260] 9f80: cf5b72b8 c003aae8 20000053 cf5a3fc0 c003a9c8
00000000 00000000 00000000
[    2.597483] 9fa0: 00000000 00000000 00000000 c00090b0 00000000
00000000 00000000 00000000
[    2.605703] 9fc0: 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[    2.613924] 9fe0: 00000000 00000000 00000000 00000000 00000013
00000000 00000000 00000000
[    2.622176] [<c0389608>] (mmc_wait_for_cmd) from [<c0392b0c>]
(mmc_send_if_cond+0x64/0xb0)
[    2.630507] [<c0392b0c>] (mmc_send_if_cond) from [<c0391a28>]
(mmc_sd_get_cid+0x40/0x164)
[    2.638750] [<c0391a28>] (mmc_sd_get_cid) from [<c03937b0>]
(mmc_sdio_init_card+0x28c/0xcb0)
[    2.647249] [<c03937b0>] (mmc_sdio_init_card) from [<c0394460>]
(mmc_attach_sdio+0x90/0x350)
[    2.655744] [<c0394460>] (mmc_attach_sdio) from [<c038b47c>]
(mmc_rescan+0x370/0x448)
[    2.663646] [<c038b47c>] (mmc_rescan) from [<c0035298>]
(process_one_work+0x220/0x5a0)
[    2.671629] [<c0035298>] (process_one_work) from [<c0035644>]
(worker_thread+0x2c/0x50c)
[    2.679783] [<c0035644>] (worker_thread) from [<c003aae8>]
(kthread+0x120/0x134)
[    2.687232] [<c003aae8>] (kthread) from [<c00090b0>]
(ret_from_fork+0x14/0x24)
[    2.694488] Exception stack(0xcf5d9fb0 to 0xcf5d9ff8)
[    2.699581] 9fa0:                                     00000000
00000000 00000000 00000000
[    2.707803] 9fc0: 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[    2.716018] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[    2.722683] Code: e3a0207c e3a01000 e58d307c eb050cad (e5d632c0)
[    2.729008] ---[ end trace bc2ac96ca83a59e7 ]---

Re: Regression: dmaengine: imx28 with emmc

[Sascha Hauer]

Hi Bruno,

On Tue, Oct 08, 2019 at 10:03:16AM +0200, Bruno Thomsen wrote:
> Hi
> 
> I am getting a kernel oops[1] during boot on imx28 with emmc flash right
> around rootfs mounting. Using git bisect I found the cause to be the
> following commit.
> 
> Regression: ceeeb99cd821 ("dmaengine: mxs: rename custom flag")
> 
> Reverting the 2 changes in drivers/dma/mxs-dma.c fixes the oops,
> but I am not sure that is the right solution as I don't have the full
> mxs-dma + mtd/mmc overview.
> 
> I did see that the patch isn't a simple rename but also a bit define
> change.
> From: DMA_CTRL_ACK = (1 << 1) = BIT(1)
> To: MXS_DMA_CTRL_WAIT4END = BIT(31)
> 

Damn, I wasn't aware the DMA driver has other users than the GPMI Nand.
Please try the attached patch, it should fix it for MMC/SD. It seems
however, that I2C and AUART and SPI are also affected. Are you able to
test any of these?

Sascha

---------------------------8<---------------------------

From 3f7a1097099c9e57e31a86503edc479f9964bc95 Mon Sep 17 00:00:00 2001
From: Sascha Hauer <s.hauer@pengutronix.de>
Date: Mon, 14 Oct 2019 16:07:31 +0200
Subject: [PATCH] mmc: mxs: fix flags passed to dmaengine_prep_slave_sg

Since ceeeb99cd821 we no longer abuse the DMA_CTRL_ACK flag for custom
driver use and introduced the MXS_DMA_CTRL_WAIT4END instead. We have not
changed all users to this flag though. This patch fixes it for the
mxs-mmc driver.

Fixes: ceeeb99cd821 ("dmaengine: mxs: rename custom flag")
Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
---
 drivers/mmc/host/mxs-mmc.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/mmc/host/mxs-mmc.c b/drivers/mmc/host/mxs-mmc.c
index 78e7e350655c..4031217d21c3 100644
--- a/drivers/mmc/host/mxs-mmc.c
+++ b/drivers/mmc/host/mxs-mmc.c
@@ -17,6 +17,7 @@
 #include <linux/interrupt.h>
 #include <linux/dma-mapping.h>
 #include <linux/dmaengine.h>
+#include <linux/dma/mxs-dma.h>
 #include <linux/highmem.h>
 #include <linux/clk.h>
 #include <linux/err.h>
@@ -266,7 +267,7 @@ static void mxs_mmc_bc(struct mxs_mmc_host *host)
 	ssp->ssp_pio_words[2] = cmd1;
 	ssp->dma_dir = DMA_NONE;
 	ssp->slave_dirn = DMA_TRANS_NONE;
-	desc = mxs_mmc_prep_dma(host, DMA_CTRL_ACK);
+	desc = mxs_mmc_prep_dma(host, MXS_DMA_CTRL_WAIT4END);
 	if (!desc)
 		goto out;
 
@@ -311,7 +312,7 @@ static void mxs_mmc_ac(struct mxs_mmc_host *host)
 	ssp->ssp_pio_words[2] = cmd1;
 	ssp->dma_dir = DMA_NONE;
 	ssp->slave_dirn = DMA_TRANS_NONE;
-	desc = mxs_mmc_prep_dma(host, DMA_CTRL_ACK);
+	desc = mxs_mmc_prep_dma(host, MXS_DMA_CTRL_WAIT4END);
 	if (!desc)
 		goto out;
 
@@ -441,7 +442,7 @@ static void mxs_mmc_adtc(struct mxs_mmc_host *host)
 	host->data = data;
 	ssp->dma_dir = dma_data_dir;
 	ssp->slave_dirn = slave_dirn;
-	desc = mxs_mmc_prep_dma(host, DMA_PREP_INTERRUPT | DMA_CTRL_ACK);
+	desc = mxs_mmc_prep_dma(host, DMA_PREP_INTERRUPT | MXS_DMA_CTRL_WAIT4END);
 	if (!desc)
 		goto out;
 
-- 
2.23.0


-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: Regression: dmaengine: imx28 with emmc

[Bruno Thomsen]

Hi Sascha,

Den man. 14. okt. 2019 kl. 16.13 skrev Sascha Hauer <s.hauer@pengutronix.de>:
>
> > I am getting a kernel oops[1] during boot on imx28 with emmc flash right
> > around rootfs mounting. Using git bisect I found the cause to be the
> > following commit.
> >
> > Regression: ceeeb99cd821 ("dmaengine: mxs: rename custom flag")
>
> Damn, I wasn't aware the DMA driver has other users than the GPMI Nand.
> Please try the attached patch, it should fix it for MMC/SD. It seems
> however, that I2C and AUART and SPI are also affected. Are you able to
> test any of these?

Thanks for looking into the reported issue.

I have tested your patch and it works. Rootfs is now successfully
mounted during boot and it seems to work in user-land as well.

Yes, our hardware uses I2C, AUARTs and SPI as well so I can test more patches.
I have not seen them produce kernel oops or errors, so maybe they
fallback to non-DMA mode.

> Subject: [PATCH] mmc: mxs: fix flags passed to dmaengine_prep_slave_sg
>
> Fixes: ceeeb99cd821 ("dmaengine: mxs: rename custom flag")
> Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>

Reported-by: Bruno Thomsen <bruno.thomsen@gmail.com>
Tested-by: Bruno Thomsen <bruno.thomsen@gmail.com>

/Bruno

Re: Regression: dmaengine: imx28 with emmc

[Fabio Estevam]

Hi Sascha,

On Mon, Oct 14, 2019 at 11:14 AM Sascha Hauer <s.hauer@pengutronix.de> wrote:

> From 3f7a1097099c9e57e31a86503edc479f9964bc95 Mon Sep 17 00:00:00 2001
> From: Sascha Hauer <s.hauer@pengutronix.de>
> Date: Mon, 14 Oct 2019 16:07:31 +0200
> Subject: [PATCH] mmc: mxs: fix flags passed to dmaengine_prep_slave_sg
>
> Since ceeeb99cd821 we no longer abuse the DMA_CTRL_ACK flag for custom
> driver use and introduced the MXS_DMA_CTRL_WAIT4END instead. We have not
> changed all users to this flag though. This patch fixes it for the
> mxs-mmc driver.
>
> Fixes: ceeeb99cd821 ("dmaengine: mxs: rename custom flag")

Please add:

Cc: stable@vger.kernel.org

so that it can be backported to 5.3.x

> Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>

This fixes the rootfs mounting issue from SD card on a imx28-evk running 5.3.6:

Tested-by: Fabio Estevam <festevam@gmail.com>

Thanks!

Re: Regression: dmaengine: imx28 with emmc

[Vinod Koul]

On 14-10-19, 16:13, Sascha Hauer wrote:
> Hi Bruno,
> 
> On Tue, Oct 08, 2019 at 10:03:16AM +0200, Bruno Thomsen wrote:
> > Hi
> > 
> > I am getting a kernel oops[1] during boot on imx28 with emmc flash right
> > around rootfs mounting. Using git bisect I found the cause to be the
> > following commit.
> > 
> > Regression: ceeeb99cd821 ("dmaengine: mxs: rename custom flag")
> > 
> > Reverting the 2 changes in drivers/dma/mxs-dma.c fixes the oops,
> > but I am not sure that is the right solution as I don't have the full
> > mxs-dma + mtd/mmc overview.
> > 
> > I did see that the patch isn't a simple rename but also a bit define
> > change.
> > From: DMA_CTRL_ACK = (1 << 1) = BIT(1)
> > To: MXS_DMA_CTRL_WAIT4END = BIT(31)
> > 
> 
> Damn, I wasn't aware the DMA driver has other users than the GPMI Nand.
> Please try the attached patch, it should fix it for MMC/SD. It seems
> however, that I2C and AUART and SPI are also affected. Are you able to
> test any of these?
> 
> Sascha
> 
> ---------------------------8<---------------------------
> 
> >From 3f7a1097099c9e57e31a86503edc479f9964bc95 Mon Sep 17 00:00:00 2001
> From: Sascha Hauer <s.hauer@pengutronix.de>
> Date: Mon, 14 Oct 2019 16:07:31 +0200
> Subject: [PATCH] mmc: mxs: fix flags passed to dmaengine_prep_slave_sg
> 
> Since ceeeb99cd821 we no longer abuse the DMA_CTRL_ACK flag for custom
> driver use and introduced the MXS_DMA_CTRL_WAIT4END instead. We have not
> changed all users to this flag though. This patch fixes it for the
> mxs-mmc driver.

lgtm, this should be submitted to mmc folks

> 
> Fixes: ceeeb99cd821 ("dmaengine: mxs: rename custom flag")
> Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
> ---
>  drivers/mmc/host/mxs-mmc.c | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/mmc/host/mxs-mmc.c b/drivers/mmc/host/mxs-mmc.c
> index 78e7e350655c..4031217d21c3 100644
> --- a/drivers/mmc/host/mxs-mmc.c
> +++ b/drivers/mmc/host/mxs-mmc.c
> @@ -17,6 +17,7 @@
>  #include <linux/interrupt.h>
>  #include <linux/dma-mapping.h>
>  #include <linux/dmaengine.h>
> +#include <linux/dma/mxs-dma.h>
>  #include <linux/highmem.h>
>  #include <linux/clk.h>
>  #include <linux/err.h>
> @@ -266,7 +267,7 @@ static void mxs_mmc_bc(struct mxs_mmc_host *host)
>  	ssp->ssp_pio_words[2] = cmd1;
>  	ssp->dma_dir = DMA_NONE;
>  	ssp->slave_dirn = DMA_TRANS_NONE;
> -	desc = mxs_mmc_prep_dma(host, DMA_CTRL_ACK);
> +	desc = mxs_mmc_prep_dma(host, MXS_DMA_CTRL_WAIT4END);
>  	if (!desc)
>  		goto out;
>  
> @@ -311,7 +312,7 @@ static void mxs_mmc_ac(struct mxs_mmc_host *host)
>  	ssp->ssp_pio_words[2] = cmd1;
>  	ssp->dma_dir = DMA_NONE;
>  	ssp->slave_dirn = DMA_TRANS_NONE;
> -	desc = mxs_mmc_prep_dma(host, DMA_CTRL_ACK);
> +	desc = mxs_mmc_prep_dma(host, MXS_DMA_CTRL_WAIT4END);
>  	if (!desc)
>  		goto out;
>  
> @@ -441,7 +442,7 @@ static void mxs_mmc_adtc(struct mxs_mmc_host *host)
>  	host->data = data;
>  	ssp->dma_dir = dma_data_dir;
>  	ssp->slave_dirn = slave_dirn;
> -	desc = mxs_mmc_prep_dma(host, DMA_PREP_INTERRUPT | DMA_CTRL_ACK);
> +	desc = mxs_mmc_prep_dma(host, DMA_PREP_INTERRUPT | MXS_DMA_CTRL_WAIT4END);
>  	if (!desc)
>  		goto out;
>  
> -- 
> 2.23.0
> 
> 
> -- 
> Pengutronix e.K.                           |                             |
> Industrial Linux Solutions                 | http://www.pengutronix.de/  |
> Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
> Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

-- 
~Vinod

Re: Regression: dmaengine: imx28 with emmc

[Sascha Hauer]

On Mon, Oct 14, 2019 at 04:13:44PM +0200, Sascha Hauer wrote:
> Hi Bruno,
> 
> On Tue, Oct 08, 2019 at 10:03:16AM +0200, Bruno Thomsen wrote:
> > Hi
> > 
> > I am getting a kernel oops[1] during boot on imx28 with emmc flash right
> > around rootfs mounting. Using git bisect I found the cause to be the
> > following commit.
> > 
> > Regression: ceeeb99cd821 ("dmaengine: mxs: rename custom flag")
> > 
> > Reverting the 2 changes in drivers/dma/mxs-dma.c fixes the oops,
> > but I am not sure that is the right solution as I don't have the full
> > mxs-dma + mtd/mmc overview.
> > 
> > I did see that the patch isn't a simple rename but also a bit define
> > change.
> > From: DMA_CTRL_ACK = (1 << 1) = BIT(1)
> > To: MXS_DMA_CTRL_WAIT4END = BIT(31)
> > 
> 
> Damn, I wasn't aware the DMA driver has other users than the GPMI Nand.
> Please try the attached patch, it should fix it for MMC/SD. It seems
> however, that I2C and AUART and SPI are also affected. Are you able to
> test any of these?

I just sent out the patch again for inclusion with the collected
tested-by tags.

Sascha

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |