* CAAM RNG trouble
@ 2020-12-14 19:00 Lucas Stach
2021-03-02 17:33 ` Horia Geantă
0 siblings, 1 reply; 10+ messages in thread
From: Lucas Stach @ 2020-12-14 19:00 UTC (permalink / raw)
To: Horia Geantă, Aymen Sghaier, Alexandru Porosanu; +Cc: linux-crypto, kernel
Hi all,
I've been looking into a CAAM RNG issue for a while, where I could need
some input from people knowing the CAAM hardware better than I do.
Basically the issue is that on some i.MX6 units the RNG functionality
sometimes fails with this error:
caam_jr 2101000.jr0: 20003c5b: CCB: desc idx 60: RNG: Hardware error.
I can tell that it is related to the entropy delay. On all failing
units the RNG4 gets instantiated with the default entropy delay of
3200. If I dial up the delay to 3600 or 4000 the RNG works reliably. As
a negative test I changed the initial delay to 400. With this change
all units are able to successfully instantiate the RNG handles at an
entropy delay of 2000 or 2400, but then reliably fail at getting random
data with the error shown above. I guess the issue is related to
prediction resistance on the handles, which causes the PRNG to be re-
seeded from the TRNG fairly often.
Now I don't have a good idea on how to arrive at a reliably working
entropy delay setting, as apparently the simple "are we able to
instantiate the handle" check is not enough to actually guarantee a
working RNG setup. Any suggestions?
Regards,
Lucas
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: CAAM RNG trouble
2020-12-14 19:00 CAAM RNG trouble Lucas Stach
@ 2021-03-02 17:33 ` Horia Geantă
2021-03-03 0:37 ` Robert Hancock
0 siblings, 1 reply; 10+ messages in thread
From: Horia Geantă @ 2021-03-02 17:33 UTC (permalink / raw)
To: Lucas Stach, Aymen Sghaier, Alexandru Porosanu, Robert Hancock
Cc: linux-crypto, kernel
On 12/14/2020 9:00 PM, Lucas Stach wrote:
> Hi all,
>
> I've been looking into a CAAM RNG issue for a while, where I could need
> some input from people knowing the CAAM hardware better than I do.
> Basically the issue is that on some i.MX6 units the RNG functionality
> sometimes fails with this error:
> caam_jr 2101000.jr0: 20003c5b: CCB: desc idx 60: RNG: Hardware error.
>
> I can tell that it is related to the entropy delay. On all failing
> units the RNG4 gets instantiated with the default entropy delay of
> 3200. If I dial up the delay to 3600 or 4000 the RNG works reliably. As
> a negative test I changed the initial delay to 400. With this change
> all units are able to successfully instantiate the RNG handles at an
> entropy delay of 2000 or 2400, but then reliably fail at getting random
> data with the error shown above. I guess the issue is related to
> prediction resistance on the handles, which causes the PRNG to be re-
> seeded from the TRNG fairly often.
>
> Now I don't have a good idea on how to arrive at a reliably working
> entropy delay setting, as apparently the simple "are we able to
> instantiate the handle" check is not enough to actually guarantee a
> working RNG setup. Any suggestions?
>
The successful instantiation of the RNG state handle(s) means that
the HW self-tests passed, but this doesn't mean RNG will work flawlessly.
A properly configured RNG should have a certain (very low) failure rate.
The logic in the caam rng driver is not checking this rate, since it's running
only once with a given configuration.
OTOH properly checking the RNG configuration would take some time, so it would
be better to run it offline. The "characterization" should also account for
temperature, voltage and process (fixed for a given SoC).
From this perspective, the caam rng driver should be updated to statically
configure the RNG with these offline-determined parameters.
Ideally we'd be able to use a single set of parameters to cover all SoCs
that have the same IP (RNG4 TRNG).
Unfortunately we're not there yet.
The situation became more visible after changing the caam rng driver to reseed
the PRNG before every request (practically making the PRNG function like a TRNG,
a hwrng framework requirement), since the HW self-tests are now running more
often then before.
Some questions that would give me more details about the exact issue you
and Robert are facing:
1. What SoC exactly are you running on?
2. How fast and how often is the RNG hardware error occurring?
Does this happen at boot time, only when stressing /dev/hwrng etc.?
3. Try dumping some of the RNG registers using below patch:
-- >8 --
Subject: [PATCH] crypto: caam - rng debugging
Dump RNG registers at hwrng.init time and in case descriptor returns
RNG HW error.
Signed-off-by: Horia Geantă <horia.geanta@nxp.com>
---
drivers/crypto/caam/caamrng.c | 9 ++++++++-
drivers/crypto/caam/ctrl.c | 29 +++++++++++++++++++++++++++++
drivers/crypto/caam/ctrl.h | 2 ++
drivers/crypto/caam/regs.h | 5 ++++-
4 files changed, 43 insertions(+), 2 deletions(-)
diff --git a/drivers/crypto/caam/caamrng.c b/drivers/crypto/caam/caamrng.c
index 77d048dfe5d0..fc2192183696 100644
--- a/drivers/crypto/caam/caamrng.c
+++ b/drivers/crypto/caam/caamrng.c
@@ -16,6 +16,7 @@
#include "compat.h"
+#include "ctrl.h"
#include "regs.h"
#include "intern.h"
#include "desc_constr.h"
@@ -57,9 +58,12 @@ static void caam_rng_done(struct device *jrdev, u32 *desc, u32 err,
{
struct caam_rng_job_ctx *jctx = context;
- if (err)
+ if (err) {
*jctx->err = caam_jr_strstatus(jrdev, err);
+ caam_dump_rng_regs(jrdev);
+ }
+
complete(jctx->done);
}
@@ -199,6 +203,9 @@ static int caam_init(struct hwrng *rng)
return err;
}
+ dev_dbg(ctx->jrdev, "CAAM RNG - register status at hwrng.init time\n");
+ caam_dump_rng_regs(ctx->jrdev);
+
/*
* Fill async buffer to have early randomness data for
* hw_random
diff --git a/drivers/crypto/caam/ctrl.c b/drivers/crypto/caam/ctrl.c
index ca0361b2dbb0..52db32b599aa 100644
--- a/drivers/crypto/caam/ctrl.c
+++ b/drivers/crypto/caam/ctrl.c
@@ -27,6 +27,35 @@ EXPORT_SYMBOL(caam_dpaa2);
#include "qi.h"
#endif
+void caam_dump_rng_regs(struct device *jrdev)
+{
+ struct device *ctrldev = jrdev->parent;
+ struct caam_drv_private *ctrlpriv = dev_get_drvdata(ctrldev);
+ struct caam_ctrl __iomem *ctrl;
+ struct rng4tst __iomem *r4tst;
+ u32 rtmctl;
+
+ dev_dbg(jrdev, "RNG register dump:\n");
+
+ ctrl = (struct caam_ctrl __iomem *)ctrlpriv->ctrl;
+ r4tst = &ctrl->r4tst[0];
+
+ dev_dbg(jrdev, "\trdsta = 0x%08x\n", rd_reg32(&r4tst->rdsta));
+
+ rtmctl = rd_reg32(&r4tst->rtmctl);
+ dev_dbg(jrdev, "\trtmctl = 0x%08x\n", rtmctl);
+ dev_dbg(jrdev, "\trtstatus = 0x%08x\n", rd_reg32(&r4tst->rtstatus));
+
+ /* Group of registers that can be read only when RTMCTL[PRGM]=1 */
+ clrsetbits_32(&r4tst->rtmctl, 0, RTMCTL_PRGM | RTMCTL_ACC);
+ dev_dbg(jrdev, "\trtscmisc = 0x%08x\n", rd_reg32(&r4tst->rtscmisc));
+ dev_dbg(jrdev, "\trtfrqmin = 0x%08x\n", rd_reg32(&r4tst->rtfrqmin));
+ dev_dbg(jrdev, "\trtfrqmax = 0x%08x\n", rd_reg32(&r4tst->rtfrqmax));
+ clrsetbits_32(&r4tst->rtmctl, RTMCTL_PRGM | RTMCTL_ACC, RTMCTL_ERR);
+
+}
+EXPORT_SYMBOL(caam_dump_rng_regs);
+
/*
* Descriptor to instantiate RNG State Handle 0 in normal mode and
* load the JDKEK, TDKEK and TDSK registers
diff --git a/drivers/crypto/caam/ctrl.h b/drivers/crypto/caam/ctrl.h
index f3ecd67922a7..806f4563990c 100644
--- a/drivers/crypto/caam/ctrl.h
+++ b/drivers/crypto/caam/ctrl.h
@@ -11,4 +11,6 @@
/* Prototypes for backend-level services exposed to APIs */
extern bool caam_dpaa2;
+void caam_dump_rng_regs(struct device *ctrldev);
+
#endif /* CTRL_H */
diff --git a/drivers/crypto/caam/regs.h b/drivers/crypto/caam/regs.h
index af61f3a2c0d4..dfc25a458a55 100644
--- a/drivers/crypto/caam/regs.h
+++ b/drivers/crypto/caam/regs.h
@@ -493,6 +493,7 @@ struct rngtst {
/* RNG4 TRNG test registers */
struct rng4tst {
#define RTMCTL_ACC BIT(5) /* TRNG access mode */
+#define RTMCTL_ERR BIT(12) /* TRNG error */
#define RTMCTL_PRGM BIT(16) /* 1 -> program mode, 0 -> run mode */
#define RTMCTL_SAMP_MODE_VON_NEUMANN_ES_SC 0 /* use von Neumann data in
both entropy shifter and
@@ -526,7 +527,9 @@ struct rng4tst {
u32 rtfrqmax; /* PRGM=1: freq. count max. limit register */
u32 rtfrqcnt; /* PRGM=0: freq. count register */
};
- u32 rsvd1[40];
+ u32 rsvd[7];
+ u32 rtstatus; /* TRNG status register */
+ u32 rsvd1[32];
#define RDSTA_SKVT 0x80000000
#define RDSTA_SKVN 0x40000000
#define RDSTA_PR0 BIT(4)
--
2.17.1
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: CAAM RNG trouble
2021-03-02 17:33 ` Horia Geantă
@ 2021-03-03 0:37 ` Robert Hancock
2021-11-12 15:27 ` [RFC PATCH] crypto: caam - restore retry count after HW RNG failure Michal Vokáč
0 siblings, 1 reply; 10+ messages in thread
From: Robert Hancock @ 2021-03-03 0:37 UTC (permalink / raw)
To: l.stach, alexandru.porosanu, aymen.sghaier, horia.geanta
Cc: linux-crypto, kernel
On Tue, 2021-03-02 at 19:33 +0200, Horia Geantă wrote:
> On 12/14/2020 9:00 PM, Lucas Stach wrote:
> > Hi all,
> >
> > I've been looking into a CAAM RNG issue for a while, where I could need
> > some input from people knowing the CAAM hardware better than I do.
> > Basically the issue is that on some i.MX6 units the RNG functionality
> > sometimes fails with this error:
> > caam_jr 2101000.jr0: 20003c5b: CCB: desc idx 60: RNG: Hardware error.
> >
> > I can tell that it is related to the entropy delay. On all failing
> > units the RNG4 gets instantiated with the default entropy delay of
> > 3200. If I dial up the delay to 3600 or 4000 the RNG works reliably. As
> > a negative test I changed the initial delay to 400. With this change
> > all units are able to successfully instantiate the RNG handles at an
> > entropy delay of 2000 or 2400, but then reliably fail at getting random
> > data with the error shown above. I guess the issue is related to
> > prediction resistance on the handles, which causes the PRNG to be re-
> > seeded from the TRNG fairly often.
> >
> > Now I don't have a good idea on how to arrive at a reliably working
> > entropy delay setting, as apparently the simple "are we able to
> > instantiate the handle" check is not enough to actually guarantee a
> > working RNG setup. Any suggestions?
> >
> The successful instantiation of the RNG state handle(s) means that
> the HW self-tests passed, but this doesn't mean RNG will work flawlessly.
>
> A properly configured RNG should have a certain (very low) failure rate.
> The logic in the caam rng driver is not checking this rate, since it's
> running
> only once with a given configuration.
> OTOH properly checking the RNG configuration would take some time, so it
> would
> be better to run it offline. The "characterization" should also account for
> temperature, voltage and process (fixed for a given SoC).
>
> From this perspective, the caam rng driver should be updated to statically
> configure the RNG with these offline-determined parameters.
> Ideally we'd be able to use a single set of parameters to cover all SoCs
> that have the same IP (RNG4 TRNG).
> Unfortunately we're not there yet.
>
> The situation became more visible after changing the caam rng driver to
> reseed
> the PRNG before every request (practically making the PRNG function like a
> TRNG,
> a hwrng framework requirement), since the HW self-tests are now running more
> often then before.
>
> Some questions that would give me more details about the exact issue you
> and Robert are facing:
>
> 1. What SoC exactly are you running on?
>
> 2. How fast and how often is the RNG hardware error occurring?
> Does this happen at boot time, only when stressing /dev/hwrng etc.?
We are using an iMX6D. In our case, it seems this is occurring relatively
rarely - I have only seen this occur on a few boots. When it has happened, it
started reporting errors at boot and regularly thereafter - probably as a
result of accesses being made by the rngd daemon.
>
> 3. Try dumping some of the RNG registers using below patch:
>
> -- >8 --
>
> Subject: [PATCH] crypto: caam - rng debugging
>
> Dump RNG registers at hwrng.init time and in case descriptor returns
> RNG HW error.
>
> Signed-off-by: Horia Geantă <horia.geanta@nxp.com>
> ---
> drivers/crypto/caam/caamrng.c | 9 ++++++++-
> drivers/crypto/caam/ctrl.c | 29 +++++++++++++++++++++++++++++
> drivers/crypto/caam/ctrl.h | 2 ++
> drivers/crypto/caam/regs.h | 5 ++++-
> 4 files changed, 43 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/crypto/caam/caamrng.c b/drivers/crypto/caam/caamrng.c
> index 77d048dfe5d0..fc2192183696 100644
> --- a/drivers/crypto/caam/caamrng.c
> +++ b/drivers/crypto/caam/caamrng.c
> @@ -16,6 +16,7 @@
>
> #include "compat.h"
>
> +#include "ctrl.h"
> #include "regs.h"
> #include "intern.h"
> #include "desc_constr.h"
> @@ -57,9 +58,12 @@ static void caam_rng_done(struct device *jrdev, u32 *desc,
> u32 err,
> {
> struct caam_rng_job_ctx *jctx = context;
>
> - if (err)
> + if (err) {
> *jctx->err = caam_jr_strstatus(jrdev, err);
>
> + caam_dump_rng_regs(jrdev);
> + }
> +
> complete(jctx->done);
> }
>
> @@ -199,6 +203,9 @@ static int caam_init(struct hwrng *rng)
> return err;
> }
>
> + dev_dbg(ctx->jrdev, "CAAM RNG - register status at hwrng.init time\n");
> + caam_dump_rng_regs(ctx->jrdev);
> +
> /*
> * Fill async buffer to have early randomness data for
> * hw_random
> diff --git a/drivers/crypto/caam/ctrl.c b/drivers/crypto/caam/ctrl.c
> index ca0361b2dbb0..52db32b599aa 100644
> --- a/drivers/crypto/caam/ctrl.c
> +++ b/drivers/crypto/caam/ctrl.c
> @@ -27,6 +27,35 @@ EXPORT_SYMBOL(caam_dpaa2);
> #include "qi.h"
> #endif
>
> +void caam_dump_rng_regs(struct device *jrdev)
> +{
> + struct device *ctrldev = jrdev->parent;
> + struct caam_drv_private *ctrlpriv = dev_get_drvdata(ctrldev);
> + struct caam_ctrl __iomem *ctrl;
> + struct rng4tst __iomem *r4tst;
> + u32 rtmctl;
> +
> + dev_dbg(jrdev, "RNG register dump:\n");
> +
> + ctrl = (struct caam_ctrl __iomem *)ctrlpriv->ctrl;
> + r4tst = &ctrl->r4tst[0];
> +
> + dev_dbg(jrdev, "\trdsta = 0x%08x\n", rd_reg32(&r4tst->rdsta));
> +
> + rtmctl = rd_reg32(&r4tst->rtmctl);
> + dev_dbg(jrdev, "\trtmctl = 0x%08x\n", rtmctl);
> + dev_dbg(jrdev, "\trtstatus = 0x%08x\n", rd_reg32(&r4tst->rtstatus));
> +
> + /* Group of registers that can be read only when RTMCTL[PRGM]=1 */
> + clrsetbits_32(&r4tst->rtmctl, 0, RTMCTL_PRGM | RTMCTL_ACC);
> + dev_dbg(jrdev, "\trtscmisc = 0x%08x\n", rd_reg32(&r4tst->rtscmisc));
> + dev_dbg(jrdev, "\trtfrqmin = 0x%08x\n", rd_reg32(&r4tst->rtfrqmin));
> + dev_dbg(jrdev, "\trtfrqmax = 0x%08x\n", rd_reg32(&r4tst->rtfrqmax));
> + clrsetbits_32(&r4tst->rtmctl, RTMCTL_PRGM | RTMCTL_ACC, RTMCTL_ERR);
> +
> +}
> +EXPORT_SYMBOL(caam_dump_rng_regs);
> +
> /*
> * Descriptor to instantiate RNG State Handle 0 in normal mode and
> * load the JDKEK, TDKEK and TDSK registers
> diff --git a/drivers/crypto/caam/ctrl.h b/drivers/crypto/caam/ctrl.h
> index f3ecd67922a7..806f4563990c 100644
> --- a/drivers/crypto/caam/ctrl.h
> +++ b/drivers/crypto/caam/ctrl.h
> @@ -11,4 +11,6 @@
> /* Prototypes for backend-level services exposed to APIs */
> extern bool caam_dpaa2;
>
> +void caam_dump_rng_regs(struct device *ctrldev);
> +
> #endif /* CTRL_H */
> diff --git a/drivers/crypto/caam/regs.h b/drivers/crypto/caam/regs.h
> index af61f3a2c0d4..dfc25a458a55 100644
> --- a/drivers/crypto/caam/regs.h
> +++ b/drivers/crypto/caam/regs.h
> @@ -493,6 +493,7 @@ struct rngtst {
> /* RNG4 TRNG test registers */
> struct rng4tst {
> #define RTMCTL_ACC BIT(5) /* TRNG access mode */
> +#define RTMCTL_ERR BIT(12) /* TRNG error */
> #define RTMCTL_PRGM BIT(16) /* 1 -> program mode, 0 -> run mode */
> #define RTMCTL_SAMP_MODE_VON_NEUMANN_ES_SC 0 /* use von Neumann data in
> both entropy shifter and
> @@ -526,7 +527,9 @@ struct rng4tst {
> u32 rtfrqmax; /* PRGM=1: freq. count max. limit register */
> u32 rtfrqcnt; /* PRGM=0: freq. count register */
> };
> - u32 rsvd1[40];
> + u32 rsvd[7];
> + u32 rtstatus; /* TRNG status register */
> + u32 rsvd1[32];
> #define RDSTA_SKVT 0x80000000
> #define RDSTA_SKVN 0x40000000
> #define RDSTA_PR0 BIT(4)
--
Robert Hancock
Senior Hardware Designer, Calian Advanced Technologies
www.calian.com
^ permalink raw reply [flat|nested] 10+ messages in thread
* [RFC PATCH] crypto: caam - restore retry count after HW RNG failure
2021-03-03 0:37 ` Robert Hancock
@ 2021-11-12 15:27 ` Michal Vokáč
[not found] ` <DU2PR04MB863088F218774551FA254EB795539@DU2PR04MB8630.eurprd04.prod.outlook.com>
0 siblings, 1 reply; 10+ messages in thread
From: Michal Vokáč @ 2021-11-12 15:27 UTC (permalink / raw)
To: Horia Geantă, Pankaj Gupta, Herbert Xu
Cc: David S. Miller, linux-crypto, linux-kernel, l.stach,
robert.hancock, Petr Benes, petrben, Michal Vokáč
From: Petr Benes <petr.benes@ysoft.com>
Each time TRNG generates entropy, statistical tests are run.
If they fail, RETRY_COUNT value is decremented. Once it
reaches 0, HW RNG returns an error, and needs to be reset.
RETRY_COUNT could be programmed in RTSCMISC register and is
set to 1 by default. Hence, we are left without hwrng after
the first error, which could happen even under normal
conditions.
Cc: petrben@gmail.com
Signed-off-by: Petr Benes <petr.benes@ysoft.com>
Signed-off-by: Michal Vokáč <michal.vokac@ysoft.com>
---
Hi,
we are also experiencing this issue:
caam_jr 2101000.jr0: 20003c5b: CCB: desc idx 60: RNG: Hardware error.
It is happening on both i.MX6S and i.MX6DL SoCs we use.
On Solo I can reproduce it really fast. Sometimes it happens right
after the board is NFS booted, sometimes I need to stress the HWRNG
for a while (generate few hundred KBs of random data). On some
DualLite SoCs it is happening at least once a day.
We are using the v5.10 LTS branch but I can confirm that this is
happening on all kernels since v5.7 to the latest linux-next.
We also tried to increase the RTSDCTL_ENT_DLY_MIN delay as suggested
in this thread [1]. It helped and the issue never occurred since then
but we are looking for more universal and permanent solution suitable
for upstream, hence we came up with this patch.
Any comments will be appreciated.
Thanks, Michal
[1] https://lkml.org/lkml/2021/8/30/296
drivers/crypto/caam/caamrng.c | 42 ++++++++++++++++++++++++++++++++---
drivers/crypto/caam/ctrl.c | 13 +++++++++++
drivers/crypto/caam/ctrl.h | 2 ++
3 files changed, 54 insertions(+), 3 deletions(-)
diff --git a/drivers/crypto/caam/caamrng.c b/drivers/crypto/caam/caamrng.c
index 77d048dfe5d0..2be5584ae591 100644
--- a/drivers/crypto/caam/caamrng.c
+++ b/drivers/crypto/caam/caamrng.c
@@ -21,6 +21,7 @@
#include "desc_constr.h"
#include "jr.h"
#include "error.h"
+#include "ctrl.h"
#define CAAM_RNG_MAX_FIFO_STORE_SIZE 16
@@ -113,6 +114,35 @@ static int caam_rng_read_one(struct device *jrdev,
return err ?: (ret ?: len);
}
+static void caam_rng_retry_reset(struct caam_rng_ctx *context)
+{
+ struct device *ctrldev = context->ctrldev;
+ struct caam_drv_private *ctrlpriv = dev_get_drvdata(ctrldev);
+ struct caam_ctrl __iomem *ctrl;
+ struct rng4tst __iomem *r4tst;
+ u32 __iomem *rtstatus;
+ u32 retry_count;
+
+ ctrl = (struct caam_ctrl __iomem *)ctrlpriv->ctrl;
+ r4tst = &ctrl->r4tst[0];
+
+ /*
+ * There is unfortunately no member for RTSTATUS register in
+ * struct rng4tst and the structure doesn't look stable
+ */
+ rtstatus = (u32 *)((char *)&ctrl->r4tst[0] + 0x3C);
+ retry_count = (rd_reg32(rtstatus) >> 16) & 0xf;
+ dev_dbg(ctrldev, "CAAM RNG retry count %d\n", retry_count);
+ if (retry_count == 0) {
+ dev_err(ctrldev, "CAAM RNG resetting retry count to 1\n");
+ clrsetbits_32(&r4tst->rtmctl, 0, RTMCTL_PRGM | RTMCTL_ACC);
+ wr_reg32(&r4tst->rtscmisc, (rd_reg32(&r4tst->rtscmisc) & 0x7f) | (1 << 16));
+ clrsetbits_32(&r4tst->rtmctl, RTMCTL_PRGM | RTMCTL_ACC,
+ RTMCTL_SAMP_MODE_RAW_ES_SC);
+ caam_reinstantiate_rng(ctrldev);
+ }
+}
+
static void caam_rng_fill_async(struct caam_rng_ctx *ctx)
{
struct scatterlist sg[1];
@@ -129,8 +159,10 @@ static void caam_rng_fill_async(struct caam_rng_ctx *ctx)
sg[0].length,
ctx->desc_async,
&done);
- if (len < 0)
+ if (len < 0) {
+ caam_rng_retry_reset(ctx);
return;
+ }
kfifo_dma_in_finish(&ctx->fifo, len);
}
@@ -145,13 +177,17 @@ static void caam_rng_worker(struct work_struct *work)
static int caam_read(struct hwrng *rng, void *dst, size_t max, bool wait)
{
struct caam_rng_ctx *ctx = to_caam_rng_ctx(rng);
- int out;
+ int out, ret;
if (wait) {
struct completion done;
- return caam_rng_read_one(ctx->jrdev, dst, max,
+ ret = caam_rng_read_one(ctx->jrdev, dst, max,
ctx->desc_sync, &done);
+ if (ret < 0)
+ caam_rng_retry_reset(ctx);
+
+ return ret;
}
out = kfifo_out(&ctx->fifo, dst, max);
diff --git a/drivers/crypto/caam/ctrl.c b/drivers/crypto/caam/ctrl.c
index ca0361b2dbb0..e421f8d1982b 100644
--- a/drivers/crypto/caam/ctrl.c
+++ b/drivers/crypto/caam/ctrl.c
@@ -339,6 +339,19 @@ static int instantiate_rng(struct device *ctrldev, int state_handle_mask,
return devm_add_action_or_reset(ctrldev, devm_deinstantiate_rng, ctrldev);
}
+/*
+ * caam_reinstantiate_rng - reinstantiates RNG. Intended for a case when RNG falls into
+ * HW error condition. That happens if TRNG fails statistical
+ * check and RTY_CNT value set in RTSCMISC decrements to zero.
+ * It is exported to caamrng.c
+ * @ctrldev - pointer to device
+ */
+
+int caam_reinstantiate_rng(struct device *ctrldev)
+{
+ return instantiate_rng(ctrldev, 0, 0);
+}
+
/*
* kick_trng - sets the various parameters for enabling the initialization
* of the RNG4 block in CAAM
diff --git a/drivers/crypto/caam/ctrl.h b/drivers/crypto/caam/ctrl.h
index f3ecd67922a7..26ff5a49a865 100644
--- a/drivers/crypto/caam/ctrl.h
+++ b/drivers/crypto/caam/ctrl.h
@@ -8,6 +8,8 @@
#ifndef CTRL_H
#define CTRL_H
+int caam_reinstantiate_rng(struct device *ctrldev);
+
/* Prototypes for backend-level services exposed to APIs */
extern bool caam_dpaa2;
--
2.25.1
^ permalink raw reply related [flat|nested] 10+ messages in thread
* RE: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG failure
[not found] ` <DU2PR04MB863088F218774551FA254EB795539@DU2PR04MB8630.eurprd04.prod.outlook.com>
@ 2022-01-13 7:23 ` Gaurav Jain
2022-01-17 14:08 ` Petr Benes
2022-01-20 9:59 ` Petr Benes
0 siblings, 2 replies; 10+ messages in thread
From: Gaurav Jain @ 2022-01-13 7:23 UTC (permalink / raw)
To: Horia Geanta, Pankaj Gupta, "Herbert Xu",
"Michal Vokáč"
Cc: "David S. Miller", "Petr Benes",
petrben, l.stach, robert.hancock, linux-crypto, linux-kernel,
Varun Sethi, Vabhav Sharma
Hello Michal
>
> -----Original Message-----
> From: Michal Vokáč <michal.vokac@ysoft.com>
> Sent: Friday, November 12, 2021 8:57 PM
> To: Horia Geanta <horia.geanta@nxp.com>; Pankaj Gupta
> <pankaj.gupta@nxp.com>; Herbert Xu <herbert@gondor.apana.org.au>
> Cc: David S. Miller <davem@davemloft.net>; linux-crypto@vger.kernel.org;
> linux-kernel@vger.kernel.org; l.stach@pengutronix.de;
> robert.hancock@calian.com; Petr Benes <petr.benes@ysoft.com>;
> petrben@gmail.com; Michal Vokáč <michal.vokac@ysoft.com>
> Subject: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG
> failure
>
> Caution: EXT Email
>
> From: Petr Benes <petr.benes@ysoft.com>
>
> Each time TRNG generates entropy, statistical tests are run.
> If they fail, RETRY_COUNT value is decremented. Once it reaches 0, HW RNG
> returns an error, and needs to be reset.
The least-significant 16 bits of the RTSTATUS register reflect the result of each of statistical tests.
Can you dump RTSTATUS to see which test has failed? Please dump RTMCTL as well.
Have you tried to build this patch for armv8? For me it is failing.
Regards
Gaurav Jain
> RETRY_COUNT could be programmed in RTSCMISC register and is set to 1 by
> default. Hence, we are left without hwrng after the first error, which could
> happen even under normal conditions.
>
> Cc: petrben@gmail.com
> Signed-off-by: Petr Benes <petr.benes@ysoft.com>
> Signed-off-by: Michal Vokáč <michal.vokac@ysoft.com>
> ---
> Hi,
> we are also experiencing this issue:
>
> caam_jr 2101000.jr0: 20003c5b: CCB: desc idx 60: RNG: Hardware error.
>
> It is happening on both i.MX6S and i.MX6DL SoCs we use.
> On Solo I can reproduce it really fast. Sometimes it happens right after the board
> is NFS booted, sometimes I need to stress the HWRNG for a while (generate few
> hundred KBs of random data). On some DualLite SoCs it is happening at least
> once a day.
>
> We are using the v5.10 LTS branch but I can confirm that this is happening on all
> kernels since v5.7 to the latest linux-next.
>
> We also tried to increase the RTSDCTL_ENT_DLY_MIN delay as suggested in this
> thread [1]. It helped and the issue never occurred since then but we are looking
> for more universal and permanent solution suitable for upstream, hence we
> came up with this patch.
>
> Any comments will be appreciated.
> Thanks, Michal
>
> [1]
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flkml.org
> %2Flkml%2F2021%2F8%2F30%2F296&data=04%7C01%7Cpankaj.gupta%40
> nxp.com%7C5d6bf460b260415aeda008d9a5f0ffab%7C686ea1d3bc2b4c6fa92cd
> 99c5c301635%7C0%7C0%7C637723276775699794%7CUnknown%7CTWFpbGZs
> b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
> %3D%7C1000&sdata=VxN9MqDFbISptW3OX9XTtZ%2FwQTsEbF6dETxXB%2
> BHrywg%3D&reserved=0
>
> drivers/crypto/caam/caamrng.c | 42 ++++++++++++++++++++++++++++++++---
> drivers/crypto/caam/ctrl.c | 13 +++++++++++
> drivers/crypto/caam/ctrl.h | 2 ++
> 3 files changed, 54 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/crypto/caam/caamrng.c b/drivers/crypto/caam/caamrng.c
> index 77d048dfe5d0..2be5584ae591 100644
> --- a/drivers/crypto/caam/caamrng.c
> +++ b/drivers/crypto/caam/caamrng.c
> @@ -21,6 +21,7 @@
> #include "desc_constr.h"
> #include "jr.h"
> #include "error.h"
> +#include "ctrl.h"
>
> #define CAAM_RNG_MAX_FIFO_STORE_SIZE 16
>
> @@ -113,6 +114,35 @@ static int caam_rng_read_one(struct device *jrdev,
> return err ?: (ret ?: len);
> }
>
> +static void caam_rng_retry_reset(struct caam_rng_ctx *context) {
> + struct device *ctrldev = context->ctrldev;
> + struct caam_drv_private *ctrlpriv = dev_get_drvdata(ctrldev);
> + struct caam_ctrl __iomem *ctrl;
> + struct rng4tst __iomem *r4tst;
> + u32 __iomem *rtstatus;
> + u32 retry_count;
> +
> + ctrl = (struct caam_ctrl __iomem *)ctrlpriv->ctrl;
> + r4tst = &ctrl->r4tst[0];
> +
> + /*
> + * There is unfortunately no member for RTSTATUS register in
> + * struct rng4tst and the structure doesn't look stable
> + */
> + rtstatus = (u32 *)((char *)&ctrl->r4tst[0] + 0x3C);
> + retry_count = (rd_reg32(rtstatus) >> 16) & 0xf;
> + dev_dbg(ctrldev, "CAAM RNG retry count %d\n", retry_count);
> + if (retry_count == 0) {
> + dev_err(ctrldev, "CAAM RNG resetting retry count to 1\n");
> + clrsetbits_32(&r4tst->rtmctl, 0, RTMCTL_PRGM | RTMCTL_ACC);
> + wr_reg32(&r4tst->rtscmisc, (rd_reg32(&r4tst->rtscmisc) & 0x7f) | (1 <<
> 16));
> + clrsetbits_32(&r4tst->rtmctl, RTMCTL_PRGM | RTMCTL_ACC,
> + RTMCTL_SAMP_MODE_RAW_ES_SC);
> + caam_reinstantiate_rng(ctrldev);
> + }
> +}
> +
> static void caam_rng_fill_async(struct caam_rng_ctx *ctx) {
> struct scatterlist sg[1];
> @@ -129,8 +159,10 @@ static void caam_rng_fill_async(struct caam_rng_ctx
> *ctx)
> sg[0].length,
> ctx->desc_async,
> &done);
> - if (len < 0)
> + if (len < 0) {
> + caam_rng_retry_reset(ctx);
> return;
> + }
>
> kfifo_dma_in_finish(&ctx->fifo, len); } @@ -145,13 +177,17 @@ static void
> caam_rng_worker(struct work_struct *work) static int caam_read(struct hwrng
> *rng, void *dst, size_t max, bool wait) {
> struct caam_rng_ctx *ctx = to_caam_rng_ctx(rng);
> - int out;
> + int out, ret;
>
> if (wait) {
> struct completion done;
>
> - return caam_rng_read_one(ctx->jrdev, dst, max,
> + ret = caam_rng_read_one(ctx->jrdev, dst, max,
> ctx->desc_sync, &done);
> + if (ret < 0)
> + caam_rng_retry_reset(ctx);
> +
> + return ret;
> }
>
> out = kfifo_out(&ctx->fifo, dst, max); diff --git a/drivers/crypto/caam/ctrl.c
> b/drivers/crypto/caam/ctrl.c index ca0361b2dbb0..e421f8d1982b 100644
> --- a/drivers/crypto/caam/ctrl.c
> +++ b/drivers/crypto/caam/ctrl.c
> @@ -339,6 +339,19 @@ static int instantiate_rng(struct device *ctrldev, int
> state_handle_mask,
> return devm_add_action_or_reset(ctrldev, devm_deinstantiate_rng,
> ctrldev); }
>
> +/*
> + * caam_reinstantiate_rng - reinstantiates RNG. Intended for a case when RNG
> falls into
> + * HW error condition. That happens if TRNG fails statistical
> + * check and RTY_CNT value set in RTSCMISC decrements to zero.
> + * It is exported to caamrng.c
> + * @ctrldev - pointer to device
> + */
> +
> +int caam_reinstantiate_rng(struct device *ctrldev) {
> + return instantiate_rng(ctrldev, 0, 0); }
> +
> /*
> * kick_trng - sets the various parameters for enabling the initialization
> * of the RNG4 block in CAAM
> diff --git a/drivers/crypto/caam/ctrl.h b/drivers/crypto/caam/ctrl.h index
> f3ecd67922a7..26ff5a49a865 100644
> --- a/drivers/crypto/caam/ctrl.h
> +++ b/drivers/crypto/caam/ctrl.h
> @@ -8,6 +8,8 @@
> #ifndef CTRL_H
> #define CTRL_H
>
> +int caam_reinstantiate_rng(struct device *ctrldev);
> +
> /* Prototypes for backend-level services exposed to APIs */ extern bool
> caam_dpaa2;
>
> --
> 2.25.1
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG failure
2022-01-13 7:23 ` [EXT] " Gaurav Jain
@ 2022-01-17 14:08 ` Petr Benes
2022-01-18 9:04 ` Gaurav Jain
2022-01-20 9:59 ` Petr Benes
1 sibling, 1 reply; 10+ messages in thread
From: Petr Benes @ 2022-01-17 14:08 UTC (permalink / raw)
To: Gaurav Jain
Cc: Horia Geanta, Pankaj Gupta, Herbert Xu, Michal Vokáč,
David S. Miller, Petr Benes, l.stach, robert.hancock,
linux-crypto, linux-kernel, Varun Sethi, Vabhav Sharma
Hi Gaurav,
I've resumed work on this issue. I hope I'll get RTSTATUS and RTMCTL
soon. Meanwhile, what is the problem with armv8? I'll try some build
for armv8, but it will take time.
Regards,
Petr
On Thu, 13 Jan 2022 at 08:23, Gaurav Jain <gaurav.jain@nxp.com> wrote:
>
> Hello Michal
>
> >
> > -----Original Message-----
> > From: Michal Vokáč <michal.vokac@ysoft.com>
> > Sent: Friday, November 12, 2021 8:57 PM
> > To: Horia Geanta <horia.geanta@nxp.com>; Pankaj Gupta
> > <pankaj.gupta@nxp.com>; Herbert Xu <herbert@gondor.apana.org.au>
> > Cc: David S. Miller <davem@davemloft.net>; linux-crypto@vger.kernel.org;
> > linux-kernel@vger.kernel.org; l.stach@pengutronix.de;
> > robert.hancock@calian.com; Petr Benes <petr.benes@ysoft.com>;
> > petrben@gmail.com; Michal Vokáč <michal.vokac@ysoft.com>
> > Subject: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG
> > failure
> >
> > Caution: EXT Email
> >
> > From: Petr Benes <petr.benes@ysoft.com>
> >
> > Each time TRNG generates entropy, statistical tests are run.
> > If they fail, RETRY_COUNT value is decremented. Once it reaches 0, HW RNG
> > returns an error, and needs to be reset.
>
> The least-significant 16 bits of the RTSTATUS register reflect the result of each of statistical tests.
> Can you dump RTSTATUS to see which test has failed? Please dump RTMCTL as well.
>
> Have you tried to build this patch for armv8? For me it is failing.
>
> Regards
> Gaurav Jain
>
> > RETRY_COUNT could be programmed in RTSCMISC register and is set to 1 by
> > default. Hence, we are left without hwrng after the first error, which could
> > happen even under normal conditions.
> >
> > Cc: petrben@gmail.com
> > Signed-off-by: Petr Benes <petr.benes@ysoft.com>
> > Signed-off-by: Michal Vokáč <michal.vokac@ysoft.com>
> > ---
> > Hi,
> > we are also experiencing this issue:
> >
> > caam_jr 2101000.jr0: 20003c5b: CCB: desc idx 60: RNG: Hardware error.
> >
> > It is happening on both i.MX6S and i.MX6DL SoCs we use.
> > On Solo I can reproduce it really fast. Sometimes it happens right after the board
> > is NFS booted, sometimes I need to stress the HWRNG for a while (generate few
> > hundred KBs of random data). On some DualLite SoCs it is happening at least
> > once a day.
> >
> > We are using the v5.10 LTS branch but I can confirm that this is happening on all
> > kernels since v5.7 to the latest linux-next.
> >
> > We also tried to increase the RTSDCTL_ENT_DLY_MIN delay as suggested in this
> > thread [1]. It helped and the issue never occurred since then but we are looking
> > for more universal and permanent solution suitable for upstream, hence we
> > came up with this patch.
> >
> > Any comments will be appreciated.
> > Thanks, Michal
> >
> > [1]
> > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flkml.org
> > %2Flkml%2F2021%2F8%2F30%2F296&data=04%7C01%7Cpankaj.gupta%40
> > nxp.com%7C5d6bf460b260415aeda008d9a5f0ffab%7C686ea1d3bc2b4c6fa92cd
> > 99c5c301635%7C0%7C0%7C637723276775699794%7CUnknown%7CTWFpbGZs
> > b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
> > %3D%7C1000&sdata=VxN9MqDFbISptW3OX9XTtZ%2FwQTsEbF6dETxXB%2
> > BHrywg%3D&reserved=0
> >
> > drivers/crypto/caam/caamrng.c | 42 ++++++++++++++++++++++++++++++++---
> > drivers/crypto/caam/ctrl.c | 13 +++++++++++
> > drivers/crypto/caam/ctrl.h | 2 ++
> > 3 files changed, 54 insertions(+), 3 deletions(-)
> >
> > diff --git a/drivers/crypto/caam/caamrng.c b/drivers/crypto/caam/caamrng.c
> > index 77d048dfe5d0..2be5584ae591 100644
> > --- a/drivers/crypto/caam/caamrng.c
> > +++ b/drivers/crypto/caam/caamrng.c
> > @@ -21,6 +21,7 @@
> > #include "desc_constr.h"
> > #include "jr.h"
> > #include "error.h"
> > +#include "ctrl.h"
> >
> > #define CAAM_RNG_MAX_FIFO_STORE_SIZE 16
> >
> > @@ -113,6 +114,35 @@ static int caam_rng_read_one(struct device *jrdev,
> > return err ?: (ret ?: len);
> > }
> >
> > +static void caam_rng_retry_reset(struct caam_rng_ctx *context) {
> > + struct device *ctrldev = context->ctrldev;
> > + struct caam_drv_private *ctrlpriv = dev_get_drvdata(ctrldev);
> > + struct caam_ctrl __iomem *ctrl;
> > + struct rng4tst __iomem *r4tst;
> > + u32 __iomem *rtstatus;
> > + u32 retry_count;
> > +
> > + ctrl = (struct caam_ctrl __iomem *)ctrlpriv->ctrl;
> > + r4tst = &ctrl->r4tst[0];
> > +
> > + /*
> > + * There is unfortunately no member for RTSTATUS register in
> > + * struct rng4tst and the structure doesn't look stable
> > + */
> > + rtstatus = (u32 *)((char *)&ctrl->r4tst[0] + 0x3C);
> > + retry_count = (rd_reg32(rtstatus) >> 16) & 0xf;
> > + dev_dbg(ctrldev, "CAAM RNG retry count %d\n", retry_count);
> > + if (retry_count == 0) {
> > + dev_err(ctrldev, "CAAM RNG resetting retry count to 1\n");
> > + clrsetbits_32(&r4tst->rtmctl, 0, RTMCTL_PRGM | RTMCTL_ACC);
> > + wr_reg32(&r4tst->rtscmisc, (rd_reg32(&r4tst->rtscmisc) & 0x7f) | (1 <<
> > 16));
> > + clrsetbits_32(&r4tst->rtmctl, RTMCTL_PRGM | RTMCTL_ACC,
> > + RTMCTL_SAMP_MODE_RAW_ES_SC);
> > + caam_reinstantiate_rng(ctrldev);
> > + }
> > +}
> > +
> > static void caam_rng_fill_async(struct caam_rng_ctx *ctx) {
> > struct scatterlist sg[1];
> > @@ -129,8 +159,10 @@ static void caam_rng_fill_async(struct caam_rng_ctx
> > *ctx)
> > sg[0].length,
> > ctx->desc_async,
> > &done);
> > - if (len < 0)
> > + if (len < 0) {
> > + caam_rng_retry_reset(ctx);
> > return;
> > + }
> >
> > kfifo_dma_in_finish(&ctx->fifo, len); } @@ -145,13 +177,17 @@ static void
> > caam_rng_worker(struct work_struct *work) static int caam_read(struct hwrng
> > *rng, void *dst, size_t max, bool wait) {
> > struct caam_rng_ctx *ctx = to_caam_rng_ctx(rng);
> > - int out;
> > + int out, ret;
> >
> > if (wait) {
> > struct completion done;
> >
> > - return caam_rng_read_one(ctx->jrdev, dst, max,
> > + ret = caam_rng_read_one(ctx->jrdev, dst, max,
> > ctx->desc_sync, &done);
> > + if (ret < 0)
> > + caam_rng_retry_reset(ctx);
> > +
> > + return ret;
> > }
> >
> > out = kfifo_out(&ctx->fifo, dst, max); diff --git a/drivers/crypto/caam/ctrl.c
> > b/drivers/crypto/caam/ctrl.c index ca0361b2dbb0..e421f8d1982b 100644
> > --- a/drivers/crypto/caam/ctrl.c
> > +++ b/drivers/crypto/caam/ctrl.c
> > @@ -339,6 +339,19 @@ static int instantiate_rng(struct device *ctrldev, int
> > state_handle_mask,
> > return devm_add_action_or_reset(ctrldev, devm_deinstantiate_rng,
> > ctrldev); }
> >
> > +/*
> > + * caam_reinstantiate_rng - reinstantiates RNG. Intended for a case when RNG
> > falls into
> > + * HW error condition. That happens if TRNG fails statistical
> > + * check and RTY_CNT value set in RTSCMISC decrements to zero.
> > + * It is exported to caamrng.c
> > + * @ctrldev - pointer to device
> > + */
> > +
> > +int caam_reinstantiate_rng(struct device *ctrldev) {
> > + return instantiate_rng(ctrldev, 0, 0); }
> > +
> > /*
> > * kick_trng - sets the various parameters for enabling the initialization
> > * of the RNG4 block in CAAM
> > diff --git a/drivers/crypto/caam/ctrl.h b/drivers/crypto/caam/ctrl.h index
> > f3ecd67922a7..26ff5a49a865 100644
> > --- a/drivers/crypto/caam/ctrl.h
> > +++ b/drivers/crypto/caam/ctrl.h
> > @@ -8,6 +8,8 @@
> > #ifndef CTRL_H
> > #define CTRL_H
> >
> > +int caam_reinstantiate_rng(struct device *ctrldev);
> > +
> > /* Prototypes for backend-level services exposed to APIs */ extern bool
> > caam_dpaa2;
> >
> > --
> > 2.25.1
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG failure
2022-01-17 14:08 ` Petr Benes
@ 2022-01-18 9:04 ` Gaurav Jain
0 siblings, 0 replies; 10+ messages in thread
From: Gaurav Jain @ 2022-01-18 9:04 UTC (permalink / raw)
To: Petr Benes
Cc: Horia Geanta, Pankaj Gupta, Herbert Xu, Michal Vokáč,
David S. Miller, Petr Benes, l.stach, robert.hancock,
linux-crypto, linux-kernel, Varun Sethi, Vabhav Sharma
Hi Petr
Recently I have sent patches in upstream uboot for adding caam driver support.
http://patchwork.ozlabs.org/project/uboot/list/?series=280725
if possible pls add this series in uboot and try to reproduce the problem.
Problem with armv8:
ERROR: modpost: "caam_reinstantiate_rng" [drivers/crypto/caam/caam_jr.ko] undefined!
make[1]: *** [scripts/Makefile.modpost:134: modules-only.symvers] Error 1
make[1]: *** Deleting file 'modules-only.symvers'
make: *** [Makefile:1755: modules] Error 2
Regards
Gaurav Jain
> -----Original Message-----
> From: Petr Benes <petrben@gmail.com>
> Sent: Monday, January 17, 2022 7:39 PM
> To: Gaurav Jain <gaurav.jain@nxp.com>
> Cc: Horia Geanta <horia.geanta@nxp.com>; Pankaj Gupta
> <pankaj.gupta@nxp.com>; Herbert Xu <herbert@gondor.apana.org.au>; Michal
> Vokáč <michal.vokac@ysoft.com>; David S. Miller <davem@davemloft.net>;
> Petr Benes <petr.benes@ysoft.com>; l.stach@pengutronix.de;
> robert.hancock@calian.com; linux-crypto@vger.kernel.org; linux-
> kernel@vger.kernel.org; Varun Sethi <V.Sethi@nxp.com>; Vabhav Sharma
> <vabhav.sharma@nxp.com>
> Subject: Re: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG
> failure
>
> Caution: EXT Email
>
> Hi Gaurav,
>
> I've resumed work on this issue. I hope I'll get RTSTATUS and RTMCTL soon.
> Meanwhile, what is the problem with armv8? I'll try some build for armv8, but it
> will take time.
>
> Regards,
> Petr
>
> On Thu, 13 Jan 2022 at 08:23, Gaurav Jain <gaurav.jain@nxp.com> wrote:
> >
> > Hello Michal
> >
> > >
> > > -----Original Message-----
> > > From: Michal Vokáč <michal.vokac@ysoft.com>
> > > Sent: Friday, November 12, 2021 8:57 PM
> > > To: Horia Geanta <horia.geanta@nxp.com>; Pankaj Gupta
> > > <pankaj.gupta@nxp.com>; Herbert Xu <herbert@gondor.apana.org.au>
> > > Cc: David S. Miller <davem@davemloft.net>;
> > > linux-crypto@vger.kernel.org; linux-kernel@vger.kernel.org;
> > > l.stach@pengutronix.de; robert.hancock@calian.com; Petr Benes
> > > <petr.benes@ysoft.com>; petrben@gmail.com; Michal Vokáč
> > > <michal.vokac@ysoft.com>
> > > Subject: [EXT] [RFC PATCH] crypto: caam - restore retry count after
> > > HW RNG failure
> > >
> > > Caution: EXT Email
> > >
> > > From: Petr Benes <petr.benes@ysoft.com>
> > >
> > > Each time TRNG generates entropy, statistical tests are run.
> > > If they fail, RETRY_COUNT value is decremented. Once it reaches 0,
> > > HW RNG returns an error, and needs to be reset.
> >
> > The least-significant 16 bits of the RTSTATUS register reflect the result of each
> of statistical tests.
> > Can you dump RTSTATUS to see which test has failed? Please dump RTMCTL as
> well.
> >
> > Have you tried to build this patch for armv8? For me it is failing.
> >
> > Regards
> > Gaurav Jain
> >
> > > RETRY_COUNT could be programmed in RTSCMISC register and is set to 1
> > > by default. Hence, we are left without hwrng after the first error,
> > > which could happen even under normal conditions.
> > >
> > > Cc: petrben@gmail.com
> > > Signed-off-by: Petr Benes <petr.benes@ysoft.com>
> > > Signed-off-by: Michal Vokáč <michal.vokac@ysoft.com>
> > > ---
> > > Hi,
> > > we are also experiencing this issue:
> > >
> > > caam_jr 2101000.jr0: 20003c5b: CCB: desc idx 60: RNG: Hardware error.
> > >
> > > It is happening on both i.MX6S and i.MX6DL SoCs we use.
> > > On Solo I can reproduce it really fast. Sometimes it happens right
> > > after the board is NFS booted, sometimes I need to stress the HWRNG
> > > for a while (generate few hundred KBs of random data). On some
> > > DualLite SoCs it is happening at least once a day.
> > >
> > > We are using the v5.10 LTS branch but I can confirm that this is
> > > happening on all kernels since v5.7 to the latest linux-next.
> > >
> > > We also tried to increase the RTSDCTL_ENT_DLY_MIN delay as suggested
> > > in this thread [1]. It helped and the issue never occurred since
> > > then but we are looking for more universal and permanent solution
> > > suitable for upstream, hence we came up with this patch.
> > >
> > > Any comments will be appreciated.
> > > Thanks, Michal
> > >
> > > [1]
> > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flk
> > >
> ml.org%2F&data=04%7C01%7Cgaurav.jain%40nxp.com%7Cbda11e9702e1
> 428
> > >
> 1fc3608d9d9c2e529%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6
> 3778
> > >
> 0253361778015%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ
> IjoiV2
> > >
> luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Pe0skMXk%2F
> epS
> > > AnqjZRZ6tLKqO4SKbDvX3ysA0w7W6y4%3D&reserved=0
> > > %2Flkml%2F2021%2F8%2F30%2F296&data=04%7C01%7Cpankaj.gupta
> %40
> > >
> nxp.com%7C5d6bf460b260415aeda008d9a5f0ffab%7C686ea1d3bc2b4c6fa92cd
> > >
> 99c5c301635%7C0%7C0%7C637723276775699794%7CUnknown%7CTWFpbGZs
> > >
> b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
> > > %3D%7C1000&sdata=VxN9MqDFbISptW3OX9XTtZ%2FwQTsEbF6dETxX
> B%2
> > > BHrywg%3D&reserved=0
> > >
> > > drivers/crypto/caam/caamrng.c | 42
> ++++++++++++++++++++++++++++++++---
> > > drivers/crypto/caam/ctrl.c | 13 +++++++++++
> > > drivers/crypto/caam/ctrl.h | 2 ++
> > > 3 files changed, 54 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/drivers/crypto/caam/caamrng.c
> > > b/drivers/crypto/caam/caamrng.c index 77d048dfe5d0..2be5584ae591
> > > 100644
> > > --- a/drivers/crypto/caam/caamrng.c
> > > +++ b/drivers/crypto/caam/caamrng.c
> > > @@ -21,6 +21,7 @@
> > > #include "desc_constr.h"
> > > #include "jr.h"
> > > #include "error.h"
> > > +#include "ctrl.h"
> > >
> > > #define CAAM_RNG_MAX_FIFO_STORE_SIZE 16
> > >
> > > @@ -113,6 +114,35 @@ static int caam_rng_read_one(struct device *jrdev,
> > > return err ?: (ret ?: len);
> > > }
> > >
> > > +static void caam_rng_retry_reset(struct caam_rng_ctx *context) {
> > > + struct device *ctrldev = context->ctrldev;
> > > + struct caam_drv_private *ctrlpriv = dev_get_drvdata(ctrldev);
> > > + struct caam_ctrl __iomem *ctrl;
> > > + struct rng4tst __iomem *r4tst;
> > > + u32 __iomem *rtstatus;
> > > + u32 retry_count;
> > > +
> > > + ctrl = (struct caam_ctrl __iomem *)ctrlpriv->ctrl;
> > > + r4tst = &ctrl->r4tst[0];
> > > +
> > > + /*
> > > + * There is unfortunately no member for RTSTATUS register in
> > > + * struct rng4tst and the structure doesn't look stable
> > > + */
> > > + rtstatus = (u32 *)((char *)&ctrl->r4tst[0] + 0x3C);
> > > + retry_count = (rd_reg32(rtstatus) >> 16) & 0xf;
> > > + dev_dbg(ctrldev, "CAAM RNG retry count %d\n", retry_count);
> > > + if (retry_count == 0) {
> > > + dev_err(ctrldev, "CAAM RNG resetting retry count to 1\n");
> > > + clrsetbits_32(&r4tst->rtmctl, 0, RTMCTL_PRGM | RTMCTL_ACC);
> > > + wr_reg32(&r4tst->rtscmisc,
> > > + (rd_reg32(&r4tst->rtscmisc) & 0x7f) | (1 <<
> > > 16));
> > > + clrsetbits_32(&r4tst->rtmctl, RTMCTL_PRGM | RTMCTL_ACC,
> > > + RTMCTL_SAMP_MODE_RAW_ES_SC);
> > > + caam_reinstantiate_rng(ctrldev);
> > > + }
> > > +}
> > > +
> > > static void caam_rng_fill_async(struct caam_rng_ctx *ctx) {
> > > struct scatterlist sg[1];
> > > @@ -129,8 +159,10 @@ static void caam_rng_fill_async(struct
> > > caam_rng_ctx
> > > *ctx)
> > > sg[0].length,
> > > ctx->desc_async,
> > > &done);
> > > - if (len < 0)
> > > + if (len < 0) {
> > > + caam_rng_retry_reset(ctx);
> > > return;
> > > + }
> > >
> > > kfifo_dma_in_finish(&ctx->fifo, len); } @@ -145,13 +177,17
> > > @@ static void caam_rng_worker(struct work_struct *work) static int
> > > caam_read(struct hwrng *rng, void *dst, size_t max, bool wait) {
> > > struct caam_rng_ctx *ctx = to_caam_rng_ctx(rng);
> > > - int out;
> > > + int out, ret;
> > >
> > > if (wait) {
> > > struct completion done;
> > >
> > > - return caam_rng_read_one(ctx->jrdev, dst, max,
> > > + ret = caam_rng_read_one(ctx->jrdev, dst, max,
> > > ctx->desc_sync, &done);
> > > + if (ret < 0)
> > > + caam_rng_retry_reset(ctx);
> > > +
> > > + return ret;
> > > }
> > >
> > > out = kfifo_out(&ctx->fifo, dst, max); diff --git
> > > a/drivers/crypto/caam/ctrl.c b/drivers/crypto/caam/ctrl.c index
> > > ca0361b2dbb0..e421f8d1982b 100644
> > > --- a/drivers/crypto/caam/ctrl.c
> > > +++ b/drivers/crypto/caam/ctrl.c
> > > @@ -339,6 +339,19 @@ static int instantiate_rng(struct device
> > > *ctrldev, int state_handle_mask,
> > > return devm_add_action_or_reset(ctrldev,
> > > devm_deinstantiate_rng, ctrldev); }
> > >
> > > +/*
> > > + * caam_reinstantiate_rng - reinstantiates RNG. Intended for a case
> > > +when RNG
> > > falls into
> > > + * HW error condition. That happens if TRNG fails statistical
> > > + * check and RTY_CNT value set in RTSCMISC decrements to
> zero.
> > > + * It is exported to caamrng.c
> > > + * @ctrldev - pointer to device
> > > + */
> > > +
> > > +int caam_reinstantiate_rng(struct device *ctrldev) {
> > > + return instantiate_rng(ctrldev, 0, 0); }
> > > +
> > > /*
> > > * kick_trng - sets the various parameters for enabling the initialization
> > > * of the RNG4 block in CAAM
> > > diff --git a/drivers/crypto/caam/ctrl.h b/drivers/crypto/caam/ctrl.h
> > > index
> > > f3ecd67922a7..26ff5a49a865 100644
> > > --- a/drivers/crypto/caam/ctrl.h
> > > +++ b/drivers/crypto/caam/ctrl.h
> > > @@ -8,6 +8,8 @@
> > > #ifndef CTRL_H
> > > #define CTRL_H
> > >
> > > +int caam_reinstantiate_rng(struct device *ctrldev);
> > > +
> > > /* Prototypes for backend-level services exposed to APIs */ extern
> > > bool caam_dpaa2;
> > >
> > > --
> > > 2.25.1
> >
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG failure
2022-01-13 7:23 ` [EXT] " Gaurav Jain
2022-01-17 14:08 ` Petr Benes
@ 2022-01-20 9:59 ` Petr Benes
2022-01-24 7:04 ` Gaurav Jain
1 sibling, 1 reply; 10+ messages in thread
From: Petr Benes @ 2022-01-20 9:59 UTC (permalink / raw)
To: Gaurav Jain
Cc: Horia Geanta, Pankaj Gupta, Herbert Xu, Michal Vokáč,
David S. Miller, Petr Benes, l.stach, robert.hancock,
linux-crypto, linux-kernel, Varun Sethi, Vabhav Sharma
Hi Gaurav,
I've tested 3 i.MX6DL devices over ~day and got these results (decimal values).
RTMCTL: 12289 all cases
RTSTATUS:
device A
1 x 49152
10 x 32768
device B
13 x 32768
1 x 49152
1 x 1024
1 x 2048
device C
1 x 32768
Regards,
Petr
On Thu, 13 Jan 2022 at 08:23, Gaurav Jain <gaurav.jain@nxp.com> wrote:
>
> Hello Michal
>
> >
> > -----Original Message-----
> > From: Michal Vokáč <michal.vokac@ysoft.com>
> > Sent: Friday, November 12, 2021 8:57 PM
> > To: Horia Geanta <horia.geanta@nxp.com>; Pankaj Gupta
> > <pankaj.gupta@nxp.com>; Herbert Xu <herbert@gondor.apana.org.au>
> > Cc: David S. Miller <davem@davemloft.net>; linux-crypto@vger.kernel.org;
> > linux-kernel@vger.kernel.org; l.stach@pengutronix.de;
> > robert.hancock@calian.com; Petr Benes <petr.benes@ysoft.com>;
> > petrben@gmail.com; Michal Vokáč <michal.vokac@ysoft.com>
> > Subject: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG
> > failure
> >
> > Caution: EXT Email
> >
> > From: Petr Benes <petr.benes@ysoft.com>
> >
> > Each time TRNG generates entropy, statistical tests are run.
> > If they fail, RETRY_COUNT value is decremented. Once it reaches 0, HW RNG
> > returns an error, and needs to be reset.
>
> The least-significant 16 bits of the RTSTATUS register reflect the result of each of statistical tests.
> Can you dump RTSTATUS to see which test has failed? Please dump RTMCTL as well.
>
> Have you tried to build this patch for armv8? For me it is failing.
>
> Regards
> Gaurav Jain
>
> > RETRY_COUNT could be programmed in RTSCMISC register and is set to 1 by
> > default. Hence, we are left without hwrng after the first error, which could
> > happen even under normal conditions.
> >
> > Cc: petrben@gmail.com
> > Signed-off-by: Petr Benes <petr.benes@ysoft.com>
> > Signed-off-by: Michal Vokáč <michal.vokac@ysoft.com>
> > ---
> > Hi,
> > we are also experiencing this issue:
> >
> > caam_jr 2101000.jr0: 20003c5b: CCB: desc idx 60: RNG: Hardware error.
> >
> > It is happening on both i.MX6S and i.MX6DL SoCs we use.
> > On Solo I can reproduce it really fast. Sometimes it happens right after the board
> > is NFS booted, sometimes I need to stress the HWRNG for a while (generate few
> > hundred KBs of random data). On some DualLite SoCs it is happening at least
> > once a day.
> >
> > We are using the v5.10 LTS branch but I can confirm that this is happening on all
> > kernels since v5.7 to the latest linux-next.
> >
> > We also tried to increase the RTSDCTL_ENT_DLY_MIN delay as suggested in this
> > thread [1]. It helped and the issue never occurred since then but we are looking
> > for more universal and permanent solution suitable for upstream, hence we
> > came up with this patch.
> >
> > Any comments will be appreciated.
> > Thanks, Michal
> >
> > [1]
> > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flkml.org
> > %2Flkml%2F2021%2F8%2F30%2F296&data=04%7C01%7Cpankaj.gupta%40
> > nxp.com%7C5d6bf460b260415aeda008d9a5f0ffab%7C686ea1d3bc2b4c6fa92cd
> > 99c5c301635%7C0%7C0%7C637723276775699794%7CUnknown%7CTWFpbGZs
> > b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
> > %3D%7C1000&sdata=VxN9MqDFbISptW3OX9XTtZ%2FwQTsEbF6dETxXB%2
> > BHrywg%3D&reserved=0
> >
> > drivers/crypto/caam/caamrng.c | 42 ++++++++++++++++++++++++++++++++---
> > drivers/crypto/caam/ctrl.c | 13 +++++++++++
> > drivers/crypto/caam/ctrl.h | 2 ++
> > 3 files changed, 54 insertions(+), 3 deletions(-)
> >
> > diff --git a/drivers/crypto/caam/caamrng.c b/drivers/crypto/caam/caamrng.c
> > index 77d048dfe5d0..2be5584ae591 100644
> > --- a/drivers/crypto/caam/caamrng.c
> > +++ b/drivers/crypto/caam/caamrng.c
> > @@ -21,6 +21,7 @@
> > #include "desc_constr.h"
> > #include "jr.h"
> > #include "error.h"
> > +#include "ctrl.h"
> >
> > #define CAAM_RNG_MAX_FIFO_STORE_SIZE 16
> >
> > @@ -113,6 +114,35 @@ static int caam_rng_read_one(struct device *jrdev,
> > return err ?: (ret ?: len);
> > }
> >
> > +static void caam_rng_retry_reset(struct caam_rng_ctx *context) {
> > + struct device *ctrldev = context->ctrldev;
> > + struct caam_drv_private *ctrlpriv = dev_get_drvdata(ctrldev);
> > + struct caam_ctrl __iomem *ctrl;
> > + struct rng4tst __iomem *r4tst;
> > + u32 __iomem *rtstatus;
> > + u32 retry_count;
> > +
> > + ctrl = (struct caam_ctrl __iomem *)ctrlpriv->ctrl;
> > + r4tst = &ctrl->r4tst[0];
> > +
> > + /*
> > + * There is unfortunately no member for RTSTATUS register in
> > + * struct rng4tst and the structure doesn't look stable
> > + */
> > + rtstatus = (u32 *)((char *)&ctrl->r4tst[0] + 0x3C);
> > + retry_count = (rd_reg32(rtstatus) >> 16) & 0xf;
> > + dev_dbg(ctrldev, "CAAM RNG retry count %d\n", retry_count);
> > + if (retry_count == 0) {
> > + dev_err(ctrldev, "CAAM RNG resetting retry count to 1\n");
> > + clrsetbits_32(&r4tst->rtmctl, 0, RTMCTL_PRGM | RTMCTL_ACC);
> > + wr_reg32(&r4tst->rtscmisc, (rd_reg32(&r4tst->rtscmisc) & 0x7f) | (1 <<
> > 16));
> > + clrsetbits_32(&r4tst->rtmctl, RTMCTL_PRGM | RTMCTL_ACC,
> > + RTMCTL_SAMP_MODE_RAW_ES_SC);
> > + caam_reinstantiate_rng(ctrldev);
> > + }
> > +}
> > +
> > static void caam_rng_fill_async(struct caam_rng_ctx *ctx) {
> > struct scatterlist sg[1];
> > @@ -129,8 +159,10 @@ static void caam_rng_fill_async(struct caam_rng_ctx
> > *ctx)
> > sg[0].length,
> > ctx->desc_async,
> > &done);
> > - if (len < 0)
> > + if (len < 0) {
> > + caam_rng_retry_reset(ctx);
> > return;
> > + }
> >
> > kfifo_dma_in_finish(&ctx->fifo, len); } @@ -145,13 +177,17 @@ static void
> > caam_rng_worker(struct work_struct *work) static int caam_read(struct hwrng
> > *rng, void *dst, size_t max, bool wait) {
> > struct caam_rng_ctx *ctx = to_caam_rng_ctx(rng);
> > - int out;
> > + int out, ret;
> >
> > if (wait) {
> > struct completion done;
> >
> > - return caam_rng_read_one(ctx->jrdev, dst, max,
> > + ret = caam_rng_read_one(ctx->jrdev, dst, max,
> > ctx->desc_sync, &done);
> > + if (ret < 0)
> > + caam_rng_retry_reset(ctx);
> > +
> > + return ret;
> > }
> >
> > out = kfifo_out(&ctx->fifo, dst, max); diff --git a/drivers/crypto/caam/ctrl.c
> > b/drivers/crypto/caam/ctrl.c index ca0361b2dbb0..e421f8d1982b 100644
> > --- a/drivers/crypto/caam/ctrl.c
> > +++ b/drivers/crypto/caam/ctrl.c
> > @@ -339,6 +339,19 @@ static int instantiate_rng(struct device *ctrldev, int
> > state_handle_mask,
> > return devm_add_action_or_reset(ctrldev, devm_deinstantiate_rng,
> > ctrldev); }
> >
> > +/*
> > + * caam_reinstantiate_rng - reinstantiates RNG. Intended for a case when RNG
> > falls into
> > + * HW error condition. That happens if TRNG fails statistical
> > + * check and RTY_CNT value set in RTSCMISC decrements to zero.
> > + * It is exported to caamrng.c
> > + * @ctrldev - pointer to device
> > + */
> > +
> > +int caam_reinstantiate_rng(struct device *ctrldev) {
> > + return instantiate_rng(ctrldev, 0, 0); }
> > +
> > /*
> > * kick_trng - sets the various parameters for enabling the initialization
> > * of the RNG4 block in CAAM
> > diff --git a/drivers/crypto/caam/ctrl.h b/drivers/crypto/caam/ctrl.h index
> > f3ecd67922a7..26ff5a49a865 100644
> > --- a/drivers/crypto/caam/ctrl.h
> > +++ b/drivers/crypto/caam/ctrl.h
> > @@ -8,6 +8,8 @@
> > #ifndef CTRL_H
> > #define CTRL_H
> >
> > +int caam_reinstantiate_rng(struct device *ctrldev);
> > +
> > /* Prototypes for backend-level services exposed to APIs */ extern bool
> > caam_dpaa2;
> >
> > --
> > 2.25.1
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG failure
2022-01-20 9:59 ` Petr Benes
@ 2022-01-24 7:04 ` Gaurav Jain
2022-01-24 12:25 ` Petr Benes
0 siblings, 1 reply; 10+ messages in thread
From: Gaurav Jain @ 2022-01-24 7:04 UTC (permalink / raw)
To: Petr Benes
Cc: Horia Geanta, Pankaj Gupta, Herbert Xu, Michal Vokáč,
David S. Miller, Petr Benes, l.stach, robert.hancock,
linux-crypto, linux-kernel, Varun Sethi, Vabhav Sharma
Hello Petr
> -----Original Message-----
> From: Petr Benes <petrben@gmail.com>
> Sent: Thursday, January 20, 2022 3:29 PM
> To: Gaurav Jain <gaurav.jain@nxp.com>
> Cc: Horia Geanta <horia.geanta@nxp.com>; Pankaj Gupta
> <pankaj.gupta@nxp.com>; Herbert Xu <herbert@gondor.apana.org.au>; Michal
> Vokáč <michal.vokac@ysoft.com>; David S. Miller <davem@davemloft.net>;
> Petr Benes <petr.benes@ysoft.com>; l.stach@pengutronix.de;
> robert.hancock@calian.com; linux-crypto@vger.kernel.org; linux-
> kernel@vger.kernel.org; Varun Sethi <V.Sethi@nxp.com>; Vabhav Sharma
> <vabhav.sharma@nxp.com>
> Subject: Re: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG
> failure
>
> Caution: EXT Email
>
> Hi Gaurav,
>
> I've tested 3 i.MX6DL devices over ~day and got these results (decimal values).
>
> RTMCTL: 12289 all cases
>
> RTSTATUS:
>
> device A
> 1 x 49152
> 10 x 32768
>
> device B
> 13 x 32768
> 1 x 49152
> 1 x 1024
> 1 x 2048
>
> device C
> 1 x 32768
>
Thank you Petr for the register dump. Looks like statistical tests are failing.
We are internally looking into TRNG configuration which could cause statistical test failure.
Can you confirm if you have reproduced the problem with the patches suggested in Uboot?
Link for the same is http://patchwork.ozlabs.org/project/uboot/list/?series=280725
With this patch series TRNG is configured in uboot.
Regards
Gaurav Jain
> Regards,
> Petr
>
> On Thu, 13 Jan 2022 at 08:23, Gaurav Jain <gaurav.jain@nxp.com> wrote:
> >
> > Hello Michal
> >
> > >
> > > -----Original Message-----
> > > From: Michal Vokáč <michal.vokac@ysoft.com>
> > > Sent: Friday, November 12, 2021 8:57 PM
> > > To: Horia Geanta <horia.geanta@nxp.com>; Pankaj Gupta
> > > <pankaj.gupta@nxp.com>; Herbert Xu <herbert@gondor.apana.org.au>
> > > Cc: David S. Miller <davem@davemloft.net>;
> > > linux-crypto@vger.kernel.org; linux-kernel@vger.kernel.org;
> > > l.stach@pengutronix.de; robert.hancock@calian.com; Petr Benes
> > > <petr.benes@ysoft.com>; petrben@gmail.com; Michal Vokáč
> > > <michal.vokac@ysoft.com>
> > > Subject: [EXT] [RFC PATCH] crypto: caam - restore retry count after
> > > HW RNG failure
> > >
> > > Caution: EXT Email
> > >
> > > From: Petr Benes <petr.benes@ysoft.com>
> > >
> > > Each time TRNG generates entropy, statistical tests are run.
> > > If they fail, RETRY_COUNT value is decremented. Once it reaches 0,
> > > HW RNG returns an error, and needs to be reset.
> >
> > The least-significant 16 bits of the RTSTATUS register reflect the result of each
> of statistical tests.
> > Can you dump RTSTATUS to see which test has failed? Please dump RTMCTL as
> well.
> >
> > Have you tried to build this patch for armv8? For me it is failing.
> >
> > Regards
> > Gaurav Jain
> >
> > > RETRY_COUNT could be programmed in RTSCMISC register and is set to 1
> > > by default. Hence, we are left without hwrng after the first error,
> > > which could happen even under normal conditions.
> > >
> > > Cc: petrben@gmail.com
> > > Signed-off-by: Petr Benes <petr.benes@ysoft.com>
> > > Signed-off-by: Michal Vokáč <michal.vokac@ysoft.com>
> > > ---
> > > Hi,
> > > we are also experiencing this issue:
> > >
> > > caam_jr 2101000.jr0: 20003c5b: CCB: desc idx 60: RNG: Hardware error.
> > >
> > > It is happening on both i.MX6S and i.MX6DL SoCs we use.
> > > On Solo I can reproduce it really fast. Sometimes it happens right
> > > after the board is NFS booted, sometimes I need to stress the HWRNG
> > > for a while (generate few hundred KBs of random data). On some
> > > DualLite SoCs it is happening at least once a day.
> > >
> > > We are using the v5.10 LTS branch but I can confirm that this is
> > > happening on all kernels since v5.7 to the latest linux-next.
> > >
> > > We also tried to increase the RTSDCTL_ENT_DLY_MIN delay as suggested
> > > in this thread [1]. It helped and the issue never occurred since
> > > then but we are looking for more universal and permanent solution
> > > suitable for upstream, hence we came up with this patch.
> > >
> > > Any comments will be appreciated.
> > > Thanks, Michal
> > >
> > > [1]
> > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flk
> > >
> ml.org%2F&data=04%7C01%7Cgaurav.jain%40nxp.com%7Cad793bd423a6
> 4f2
> > >
> b03d408d9dbfb842f%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6
> 3778
> > >
> 2695582123236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ
> IjoiV2
> > >
> luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=PlPYDNHg9%2
> BCL
> > > iZrcdmOx%2FPdbAZmFpPZFvLe1jR4YhEI%3D&reserved=0
> > > %2Flkml%2F2021%2F8%2F30%2F296&data=04%7C01%7Cpankaj.gupta
> %40
> > >
> nxp.com%7C5d6bf460b260415aeda008d9a5f0ffab%7C686ea1d3bc2b4c6fa92cd
> > >
> 99c5c301635%7C0%7C0%7C637723276775699794%7CUnknown%7CTWFpbGZs
> > >
> b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
> > > %3D%7C1000&sdata=VxN9MqDFbISptW3OX9XTtZ%2FwQTsEbF6dETxX
> B%2
> > > BHrywg%3D&reserved=0
> > >
> > > drivers/crypto/caam/caamrng.c | 42
> ++++++++++++++++++++++++++++++++---
> > > drivers/crypto/caam/ctrl.c | 13 +++++++++++
> > > drivers/crypto/caam/ctrl.h | 2 ++
> > > 3 files changed, 54 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/drivers/crypto/caam/caamrng.c
> > > b/drivers/crypto/caam/caamrng.c index 77d048dfe5d0..2be5584ae591
> > > 100644
> > > --- a/drivers/crypto/caam/caamrng.c
> > > +++ b/drivers/crypto/caam/caamrng.c
> > > @@ -21,6 +21,7 @@
> > > #include "desc_constr.h"
> > > #include "jr.h"
> > > #include "error.h"
> > > +#include "ctrl.h"
> > >
> > > #define CAAM_RNG_MAX_FIFO_STORE_SIZE 16
> > >
> > > @@ -113,6 +114,35 @@ static int caam_rng_read_one(struct device *jrdev,
> > > return err ?: (ret ?: len);
> > > }
> > >
> > > +static void caam_rng_retry_reset(struct caam_rng_ctx *context) {
> > > + struct device *ctrldev = context->ctrldev;
> > > + struct caam_drv_private *ctrlpriv = dev_get_drvdata(ctrldev);
> > > + struct caam_ctrl __iomem *ctrl;
> > > + struct rng4tst __iomem *r4tst;
> > > + u32 __iomem *rtstatus;
> > > + u32 retry_count;
> > > +
> > > + ctrl = (struct caam_ctrl __iomem *)ctrlpriv->ctrl;
> > > + r4tst = &ctrl->r4tst[0];
> > > +
> > > + /*
> > > + * There is unfortunately no member for RTSTATUS register in
> > > + * struct rng4tst and the structure doesn't look stable
> > > + */
> > > + rtstatus = (u32 *)((char *)&ctrl->r4tst[0] + 0x3C);
> > > + retry_count = (rd_reg32(rtstatus) >> 16) & 0xf;
> > > + dev_dbg(ctrldev, "CAAM RNG retry count %d\n", retry_count);
> > > + if (retry_count == 0) {
> > > + dev_err(ctrldev, "CAAM RNG resetting retry count to 1\n");
> > > + clrsetbits_32(&r4tst->rtmctl, 0, RTMCTL_PRGM | RTMCTL_ACC);
> > > + wr_reg32(&r4tst->rtscmisc,
> > > + (rd_reg32(&r4tst->rtscmisc) & 0x7f) | (1 <<
> > > 16));
> > > + clrsetbits_32(&r4tst->rtmctl, RTMCTL_PRGM | RTMCTL_ACC,
> > > + RTMCTL_SAMP_MODE_RAW_ES_SC);
> > > + caam_reinstantiate_rng(ctrldev);
> > > + }
> > > +}
> > > +
> > > static void caam_rng_fill_async(struct caam_rng_ctx *ctx) {
> > > struct scatterlist sg[1];
> > > @@ -129,8 +159,10 @@ static void caam_rng_fill_async(struct
> > > caam_rng_ctx
> > > *ctx)
> > > sg[0].length,
> > > ctx->desc_async,
> > > &done);
> > > - if (len < 0)
> > > + if (len < 0) {
> > > + caam_rng_retry_reset(ctx);
> > > return;
> > > + }
> > >
> > > kfifo_dma_in_finish(&ctx->fifo, len); } @@ -145,13 +177,17
> > > @@ static void caam_rng_worker(struct work_struct *work) static int
> > > caam_read(struct hwrng *rng, void *dst, size_t max, bool wait) {
> > > struct caam_rng_ctx *ctx = to_caam_rng_ctx(rng);
> > > - int out;
> > > + int out, ret;
> > >
> > > if (wait) {
> > > struct completion done;
> > >
> > > - return caam_rng_read_one(ctx->jrdev, dst, max,
> > > + ret = caam_rng_read_one(ctx->jrdev, dst, max,
> > > ctx->desc_sync, &done);
> > > + if (ret < 0)
> > > + caam_rng_retry_reset(ctx);
> > > +
> > > + return ret;
> > > }
> > >
> > > out = kfifo_out(&ctx->fifo, dst, max); diff --git
> > > a/drivers/crypto/caam/ctrl.c b/drivers/crypto/caam/ctrl.c index
> > > ca0361b2dbb0..e421f8d1982b 100644
> > > --- a/drivers/crypto/caam/ctrl.c
> > > +++ b/drivers/crypto/caam/ctrl.c
> > > @@ -339,6 +339,19 @@ static int instantiate_rng(struct device
> > > *ctrldev, int state_handle_mask,
> > > return devm_add_action_or_reset(ctrldev,
> > > devm_deinstantiate_rng, ctrldev); }
> > >
> > > +/*
> > > + * caam_reinstantiate_rng - reinstantiates RNG. Intended for a case
> > > +when RNG
> > > falls into
> > > + * HW error condition. That happens if TRNG fails statistical
> > > + * check and RTY_CNT value set in RTSCMISC decrements to
> zero.
> > > + * It is exported to caamrng.c
> > > + * @ctrldev - pointer to device
> > > + */
> > > +
> > > +int caam_reinstantiate_rng(struct device *ctrldev) {
> > > + return instantiate_rng(ctrldev, 0, 0); }
> > > +
> > > /*
> > > * kick_trng - sets the various parameters for enabling the initialization
> > > * of the RNG4 block in CAAM
> > > diff --git a/drivers/crypto/caam/ctrl.h b/drivers/crypto/caam/ctrl.h
> > > index
> > > f3ecd67922a7..26ff5a49a865 100644
> > > --- a/drivers/crypto/caam/ctrl.h
> > > +++ b/drivers/crypto/caam/ctrl.h
> > > @@ -8,6 +8,8 @@
> > > #ifndef CTRL_H
> > > #define CTRL_H
> > >
> > > +int caam_reinstantiate_rng(struct device *ctrldev);
> > > +
> > > /* Prototypes for backend-level services exposed to APIs */ extern
> > > bool caam_dpaa2;
> > >
> > > --
> > > 2.25.1
> >
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG failure
2022-01-24 7:04 ` Gaurav Jain
@ 2022-01-24 12:25 ` Petr Benes
0 siblings, 0 replies; 10+ messages in thread
From: Petr Benes @ 2022-01-24 12:25 UTC (permalink / raw)
To: Gaurav Jain
Cc: Horia Geanta, Pankaj Gupta, Herbert Xu, Michal Vokáč,
David S. Miller, Petr Benes, l.stach, robert.hancock,
linux-crypto, linux-kernel, Varun Sethi, Vabhav Sharma
Hi Gaurav,
On Mon, 24 Jan 2022 at 08:04, Gaurav Jain <gaurav.jain@nxp.com> wrote:
>
> Hello Petr
>
> > -----Original Message-----
> > From: Petr Benes <petrben@gmail.com>
> > Sent: Thursday, January 20, 2022 3:29 PM
> > To: Gaurav Jain <gaurav.jain@nxp.com>
> > Cc: Horia Geanta <horia.geanta@nxp.com>; Pankaj Gupta
> > <pankaj.gupta@nxp.com>; Herbert Xu <herbert@gondor.apana.org.au>; Michal
> > Vokáč <michal.vokac@ysoft.com>; David S. Miller <davem@davemloft.net>;
> > Petr Benes <petr.benes@ysoft.com>; l.stach@pengutronix.de;
> > robert.hancock@calian.com; linux-crypto@vger.kernel.org; linux-
> > kernel@vger.kernel.org; Varun Sethi <V.Sethi@nxp.com>; Vabhav Sharma
> > <vabhav.sharma@nxp.com>
> > Subject: Re: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG
> > failure
> >
> > Caution: EXT Email
> >
> > Hi Gaurav,
> >
> > I've tested 3 i.MX6DL devices over ~day and got these results (decimal values).
> >
> > RTMCTL: 12289 all cases
> >
> > RTSTATUS:
> >
> > device A
> > 1 x 49152
> > 10 x 32768
> >
> > device B
> > 13 x 32768
> > 1 x 49152
> > 1 x 1024
> > 1 x 2048
> >
> > device C
> > 1 x 32768
> >
>
> Thank you Petr for the register dump. Looks like statistical tests are failing.
> We are internally looking into TRNG configuration which could cause statistical test failure.
>
> Can you confirm if you have reproduced the problem with the patches suggested in Uboot?
> Link for the same is http://patchwork.ozlabs.org/project/uboot/list/?series=280725
> With this patch series TRNG is configured in uboot.
Unfortunately, we are behind with U-Boot (and do not setup CAAM
components there anyway),
it would need some planning on my side. I'm not sure I can secure
enough time for it. So, cannot
provide the confirmation at this moment nor any time soon . If it
(U-Boot patch) solves the problem,
do you plan to provide a solely Linux solution? If I got it correctly,
there is a small chance to hit this
issue under normal conditions, some retry mechanism is desired.
Regards,
Petr
>
> Regards
> Gaurav Jain
>
> > Regards,
> > Petr
> >
> > On Thu, 13 Jan 2022 at 08:23, Gaurav Jain <gaurav.jain@nxp.com> wrote:
> > >
> > > Hello Michal
> > >
> > > >
> > > > -----Original Message-----
> > > > From: Michal Vokáč <michal.vokac@ysoft.com>
> > > > Sent: Friday, November 12, 2021 8:57 PM
> > > > To: Horia Geanta <horia.geanta@nxp.com>; Pankaj Gupta
> > > > <pankaj.gupta@nxp.com>; Herbert Xu <herbert@gondor.apana.org.au>
> > > > Cc: David S. Miller <davem@davemloft.net>;
> > > > linux-crypto@vger.kernel.org; linux-kernel@vger.kernel.org;
> > > > l.stach@pengutronix.de; robert.hancock@calian.com; Petr Benes
> > > > <petr.benes@ysoft.com>; petrben@gmail.com; Michal Vokáč
> > > > <michal.vokac@ysoft.com>
> > > > Subject: [EXT] [RFC PATCH] crypto: caam - restore retry count after
> > > > HW RNG failure
> > > >
> > > > Caution: EXT Email
> > > >
> > > > From: Petr Benes <petr.benes@ysoft.com>
> > > >
> > > > Each time TRNG generates entropy, statistical tests are run.
> > > > If they fail, RETRY_COUNT value is decremented. Once it reaches 0,
> > > > HW RNG returns an error, and needs to be reset.
> > >
> > > The least-significant 16 bits of the RTSTATUS register reflect the result of each
> > of statistical tests.
> > > Can you dump RTSTATUS to see which test has failed? Please dump RTMCTL as
> > well.
> > >
> > > Have you tried to build this patch for armv8? For me it is failing.
> > >
> > > Regards
> > > Gaurav Jain
> > >
> > > > RETRY_COUNT could be programmed in RTSCMISC register and is set to 1
> > > > by default. Hence, we are left without hwrng after the first error,
> > > > which could happen even under normal conditions.
> > > >
> > > > Cc: petrben@gmail.com
> > > > Signed-off-by: Petr Benes <petr.benes@ysoft.com>
> > > > Signed-off-by: Michal Vokáč <michal.vokac@ysoft.com>
> > > > ---
> > > > Hi,
> > > > we are also experiencing this issue:
> > > >
> > > > caam_jr 2101000.jr0: 20003c5b: CCB: desc idx 60: RNG: Hardware error.
> > > >
> > > > It is happening on both i.MX6S and i.MX6DL SoCs we use.
> > > > On Solo I can reproduce it really fast. Sometimes it happens right
> > > > after the board is NFS booted, sometimes I need to stress the HWRNG
> > > > for a while (generate few hundred KBs of random data). On some
> > > > DualLite SoCs it is happening at least once a day.
> > > >
> > > > We are using the v5.10 LTS branch but I can confirm that this is
> > > > happening on all kernels since v5.7 to the latest linux-next.
> > > >
> > > > We also tried to increase the RTSDCTL_ENT_DLY_MIN delay as suggested
> > > > in this thread [1]. It helped and the issue never occurred since
> > > > then but we are looking for more universal and permanent solution
> > > > suitable for upstream, hence we came up with this patch.
> > > >
> > > > Any comments will be appreciated.
> > > > Thanks, Michal
> > > >
> > > > [1]
> > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flk
> > > >
> > ml.org%2F&data=04%7C01%7Cgaurav.jain%40nxp.com%7Cad793bd423a6
> > 4f2
> > > >
> > b03d408d9dbfb842f%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6
> > 3778
> > > >
> > 2695582123236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ
> > IjoiV2
> > > >
> > luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=PlPYDNHg9%2
> > BCL
> > > > iZrcdmOx%2FPdbAZmFpPZFvLe1jR4YhEI%3D&reserved=0
> > > > %2Flkml%2F2021%2F8%2F30%2F296&data=04%7C01%7Cpankaj.gupta
> > %40
> > > >
> > nxp.com%7C5d6bf460b260415aeda008d9a5f0ffab%7C686ea1d3bc2b4c6fa92cd
> > > >
> > 99c5c301635%7C0%7C0%7C637723276775699794%7CUnknown%7CTWFpbGZs
> > > >
> > b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
> > > > %3D%7C1000&sdata=VxN9MqDFbISptW3OX9XTtZ%2FwQTsEbF6dETxX
> > B%2
> > > > BHrywg%3D&reserved=0
> > > >
> > > > drivers/crypto/caam/caamrng.c | 42
> > ++++++++++++++++++++++++++++++++---
> > > > drivers/crypto/caam/ctrl.c | 13 +++++++++++
> > > > drivers/crypto/caam/ctrl.h | 2 ++
> > > > 3 files changed, 54 insertions(+), 3 deletions(-)
> > > >
> > > > diff --git a/drivers/crypto/caam/caamrng.c
> > > > b/drivers/crypto/caam/caamrng.c index 77d048dfe5d0..2be5584ae591
> > > > 100644
> > > > --- a/drivers/crypto/caam/caamrng.c
> > > > +++ b/drivers/crypto/caam/caamrng.c
> > > > @@ -21,6 +21,7 @@
> > > > #include "desc_constr.h"
> > > > #include "jr.h"
> > > > #include "error.h"
> > > > +#include "ctrl.h"
> > > >
> > > > #define CAAM_RNG_MAX_FIFO_STORE_SIZE 16
> > > >
> > > > @@ -113,6 +114,35 @@ static int caam_rng_read_one(struct device *jrdev,
> > > > return err ?: (ret ?: len);
> > > > }
> > > >
> > > > +static void caam_rng_retry_reset(struct caam_rng_ctx *context) {
> > > > + struct device *ctrldev = context->ctrldev;
> > > > + struct caam_drv_private *ctrlpriv = dev_get_drvdata(ctrldev);
> > > > + struct caam_ctrl __iomem *ctrl;
> > > > + struct rng4tst __iomem *r4tst;
> > > > + u32 __iomem *rtstatus;
> > > > + u32 retry_count;
> > > > +
> > > > + ctrl = (struct caam_ctrl __iomem *)ctrlpriv->ctrl;
> > > > + r4tst = &ctrl->r4tst[0];
> > > > +
> > > > + /*
> > > > + * There is unfortunately no member for RTSTATUS register in
> > > > + * struct rng4tst and the structure doesn't look stable
> > > > + */
> > > > + rtstatus = (u32 *)((char *)&ctrl->r4tst[0] + 0x3C);
> > > > + retry_count = (rd_reg32(rtstatus) >> 16) & 0xf;
> > > > + dev_dbg(ctrldev, "CAAM RNG retry count %d\n", retry_count);
> > > > + if (retry_count == 0) {
> > > > + dev_err(ctrldev, "CAAM RNG resetting retry count to 1\n");
> > > > + clrsetbits_32(&r4tst->rtmctl, 0, RTMCTL_PRGM | RTMCTL_ACC);
> > > > + wr_reg32(&r4tst->rtscmisc,
> > > > + (rd_reg32(&r4tst->rtscmisc) & 0x7f) | (1 <<
> > > > 16));
> > > > + clrsetbits_32(&r4tst->rtmctl, RTMCTL_PRGM | RTMCTL_ACC,
> > > > + RTMCTL_SAMP_MODE_RAW_ES_SC);
> > > > + caam_reinstantiate_rng(ctrldev);
> > > > + }
> > > > +}
> > > > +
> > > > static void caam_rng_fill_async(struct caam_rng_ctx *ctx) {
> > > > struct scatterlist sg[1];
> > > > @@ -129,8 +159,10 @@ static void caam_rng_fill_async(struct
> > > > caam_rng_ctx
> > > > *ctx)
> > > > sg[0].length,
> > > > ctx->desc_async,
> > > > &done);
> > > > - if (len < 0)
> > > > + if (len < 0) {
> > > > + caam_rng_retry_reset(ctx);
> > > > return;
> > > > + }
> > > >
> > > > kfifo_dma_in_finish(&ctx->fifo, len); } @@ -145,13 +177,17
> > > > @@ static void caam_rng_worker(struct work_struct *work) static int
> > > > caam_read(struct hwrng *rng, void *dst, size_t max, bool wait) {
> > > > struct caam_rng_ctx *ctx = to_caam_rng_ctx(rng);
> > > > - int out;
> > > > + int out, ret;
> > > >
> > > > if (wait) {
> > > > struct completion done;
> > > >
> > > > - return caam_rng_read_one(ctx->jrdev, dst, max,
> > > > + ret = caam_rng_read_one(ctx->jrdev, dst, max,
> > > > ctx->desc_sync, &done);
> > > > + if (ret < 0)
> > > > + caam_rng_retry_reset(ctx);
> > > > +
> > > > + return ret;
> > > > }
> > > >
> > > > out = kfifo_out(&ctx->fifo, dst, max); diff --git
> > > > a/drivers/crypto/caam/ctrl.c b/drivers/crypto/caam/ctrl.c index
> > > > ca0361b2dbb0..e421f8d1982b 100644
> > > > --- a/drivers/crypto/caam/ctrl.c
> > > > +++ b/drivers/crypto/caam/ctrl.c
> > > > @@ -339,6 +339,19 @@ static int instantiate_rng(struct device
> > > > *ctrldev, int state_handle_mask,
> > > > return devm_add_action_or_reset(ctrldev,
> > > > devm_deinstantiate_rng, ctrldev); }
> > > >
> > > > +/*
> > > > + * caam_reinstantiate_rng - reinstantiates RNG. Intended for a case
> > > > +when RNG
> > > > falls into
> > > > + * HW error condition. That happens if TRNG fails statistical
> > > > + * check and RTY_CNT value set in RTSCMISC decrements to
> > zero.
> > > > + * It is exported to caamrng.c
> > > > + * @ctrldev - pointer to device
> > > > + */
> > > > +
> > > > +int caam_reinstantiate_rng(struct device *ctrldev) {
> > > > + return instantiate_rng(ctrldev, 0, 0); }
> > > > +
> > > > /*
> > > > * kick_trng - sets the various parameters for enabling the initialization
> > > > * of the RNG4 block in CAAM
> > > > diff --git a/drivers/crypto/caam/ctrl.h b/drivers/crypto/caam/ctrl.h
> > > > index
> > > > f3ecd67922a7..26ff5a49a865 100644
> > > > --- a/drivers/crypto/caam/ctrl.h
> > > > +++ b/drivers/crypto/caam/ctrl.h
> > > > @@ -8,6 +8,8 @@
> > > > #ifndef CTRL_H
> > > > #define CTRL_H
> > > >
> > > > +int caam_reinstantiate_rng(struct device *ctrldev);
> > > > +
> > > > /* Prototypes for backend-level services exposed to APIs */ extern
> > > > bool caam_dpaa2;
> > > >
> > > > --
> > > > 2.25.1
> > >
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2022-01-24 12:25 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-12-14 19:00 CAAM RNG trouble Lucas Stach
2021-03-02 17:33 ` Horia Geantă
2021-03-03 0:37 ` Robert Hancock
2021-11-12 15:27 ` [RFC PATCH] crypto: caam - restore retry count after HW RNG failure Michal Vokáč
[not found] ` <DU2PR04MB863088F218774551FA254EB795539@DU2PR04MB8630.eurprd04.prod.outlook.com>
2022-01-13 7:23 ` [EXT] " Gaurav Jain
2022-01-17 14:08 ` Petr Benes
2022-01-18 9:04 ` Gaurav Jain
2022-01-20 9:59 ` Petr Benes
2022-01-24 7:04 ` Gaurav Jain
2022-01-24 12:25 ` Petr Benes
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.