* CAAM RNG trouble @ 2020-12-14 19:00 Lucas Stach 2021-03-02 17:33 ` Horia Geantă 0 siblings, 1 reply; 10+ messages in thread From: Lucas Stach @ 2020-12-14 19:00 UTC (permalink / raw) To: Horia Geantă, Aymen Sghaier, Alexandru Porosanu; +Cc: linux-crypto, kernel Hi all, I've been looking into a CAAM RNG issue for a while, where I could need some input from people knowing the CAAM hardware better than I do. Basically the issue is that on some i.MX6 units the RNG functionality sometimes fails with this error: caam_jr 2101000.jr0: 20003c5b: CCB: desc idx 60: RNG: Hardware error. I can tell that it is related to the entropy delay. On all failing units the RNG4 gets instantiated with the default entropy delay of 3200. If I dial up the delay to 3600 or 4000 the RNG works reliably. As a negative test I changed the initial delay to 400. With this change all units are able to successfully instantiate the RNG handles at an entropy delay of 2000 or 2400, but then reliably fail at getting random data with the error shown above. I guess the issue is related to prediction resistance on the handles, which causes the PRNG to be re- seeded from the TRNG fairly often. Now I don't have a good idea on how to arrive at a reliably working entropy delay setting, as apparently the simple "are we able to instantiate the handle" check is not enough to actually guarantee a working RNG setup. Any suggestions? Regards, Lucas ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: CAAM RNG trouble 2020-12-14 19:00 CAAM RNG trouble Lucas Stach @ 2021-03-02 17:33 ` Horia Geantă 2021-03-03 0:37 ` Robert Hancock 0 siblings, 1 reply; 10+ messages in thread From: Horia Geantă @ 2021-03-02 17:33 UTC (permalink / raw) To: Lucas Stach, Aymen Sghaier, Alexandru Porosanu, Robert Hancock Cc: linux-crypto, kernel On 12/14/2020 9:00 PM, Lucas Stach wrote: > Hi all, > > I've been looking into a CAAM RNG issue for a while, where I could need > some input from people knowing the CAAM hardware better than I do. > Basically the issue is that on some i.MX6 units the RNG functionality > sometimes fails with this error: > caam_jr 2101000.jr0: 20003c5b: CCB: desc idx 60: RNG: Hardware error. > > I can tell that it is related to the entropy delay. On all failing > units the RNG4 gets instantiated with the default entropy delay of > 3200. If I dial up the delay to 3600 or 4000 the RNG works reliably. As > a negative test I changed the initial delay to 400. With this change > all units are able to successfully instantiate the RNG handles at an > entropy delay of 2000 or 2400, but then reliably fail at getting random > data with the error shown above. I guess the issue is related to > prediction resistance on the handles, which causes the PRNG to be re- > seeded from the TRNG fairly often. > > Now I don't have a good idea on how to arrive at a reliably working > entropy delay setting, as apparently the simple "are we able to > instantiate the handle" check is not enough to actually guarantee a > working RNG setup. Any suggestions? > The successful instantiation of the RNG state handle(s) means that the HW self-tests passed, but this doesn't mean RNG will work flawlessly. A properly configured RNG should have a certain (very low) failure rate. The logic in the caam rng driver is not checking this rate, since it's running only once with a given configuration. OTOH properly checking the RNG configuration would take some time, so it would be better to run it offline. The "characterization" should also account for temperature, voltage and process (fixed for a given SoC). From this perspective, the caam rng driver should be updated to statically configure the RNG with these offline-determined parameters. Ideally we'd be able to use a single set of parameters to cover all SoCs that have the same IP (RNG4 TRNG). Unfortunately we're not there yet. The situation became more visible after changing the caam rng driver to reseed the PRNG before every request (practically making the PRNG function like a TRNG, a hwrng framework requirement), since the HW self-tests are now running more often then before. Some questions that would give me more details about the exact issue you and Robert are facing: 1. What SoC exactly are you running on? 2. How fast and how often is the RNG hardware error occurring? Does this happen at boot time, only when stressing /dev/hwrng etc.? 3. Try dumping some of the RNG registers using below patch: -- >8 -- Subject: [PATCH] crypto: caam - rng debugging Dump RNG registers at hwrng.init time and in case descriptor returns RNG HW error. Signed-off-by: Horia Geantă <horia.geanta@nxp.com> --- drivers/crypto/caam/caamrng.c | 9 ++++++++- drivers/crypto/caam/ctrl.c | 29 +++++++++++++++++++++++++++++ drivers/crypto/caam/ctrl.h | 2 ++ drivers/crypto/caam/regs.h | 5 ++++- 4 files changed, 43 insertions(+), 2 deletions(-) diff --git a/drivers/crypto/caam/caamrng.c b/drivers/crypto/caam/caamrng.c index 77d048dfe5d0..fc2192183696 100644 --- a/drivers/crypto/caam/caamrng.c +++ b/drivers/crypto/caam/caamrng.c @@ -16,6 +16,7 @@ #include "compat.h" +#include "ctrl.h" #include "regs.h" #include "intern.h" #include "desc_constr.h" @@ -57,9 +58,12 @@ static void caam_rng_done(struct device *jrdev, u32 *desc, u32 err, { struct caam_rng_job_ctx *jctx = context; - if (err) + if (err) { *jctx->err = caam_jr_strstatus(jrdev, err); + caam_dump_rng_regs(jrdev); + } + complete(jctx->done); } @@ -199,6 +203,9 @@ static int caam_init(struct hwrng *rng) return err; } + dev_dbg(ctx->jrdev, "CAAM RNG - register status at hwrng.init time\n"); + caam_dump_rng_regs(ctx->jrdev); + /* * Fill async buffer to have early randomness data for * hw_random diff --git a/drivers/crypto/caam/ctrl.c b/drivers/crypto/caam/ctrl.c index ca0361b2dbb0..52db32b599aa 100644 --- a/drivers/crypto/caam/ctrl.c +++ b/drivers/crypto/caam/ctrl.c @@ -27,6 +27,35 @@ EXPORT_SYMBOL(caam_dpaa2); #include "qi.h" #endif +void caam_dump_rng_regs(struct device *jrdev) +{ + struct device *ctrldev = jrdev->parent; + struct caam_drv_private *ctrlpriv = dev_get_drvdata(ctrldev); + struct caam_ctrl __iomem *ctrl; + struct rng4tst __iomem *r4tst; + u32 rtmctl; + + dev_dbg(jrdev, "RNG register dump:\n"); + + ctrl = (struct caam_ctrl __iomem *)ctrlpriv->ctrl; + r4tst = &ctrl->r4tst[0]; + + dev_dbg(jrdev, "\trdsta = 0x%08x\n", rd_reg32(&r4tst->rdsta)); + + rtmctl = rd_reg32(&r4tst->rtmctl); + dev_dbg(jrdev, "\trtmctl = 0x%08x\n", rtmctl); + dev_dbg(jrdev, "\trtstatus = 0x%08x\n", rd_reg32(&r4tst->rtstatus)); + + /* Group of registers that can be read only when RTMCTL[PRGM]=1 */ + clrsetbits_32(&r4tst->rtmctl, 0, RTMCTL_PRGM | RTMCTL_ACC); + dev_dbg(jrdev, "\trtscmisc = 0x%08x\n", rd_reg32(&r4tst->rtscmisc)); + dev_dbg(jrdev, "\trtfrqmin = 0x%08x\n", rd_reg32(&r4tst->rtfrqmin)); + dev_dbg(jrdev, "\trtfrqmax = 0x%08x\n", rd_reg32(&r4tst->rtfrqmax)); + clrsetbits_32(&r4tst->rtmctl, RTMCTL_PRGM | RTMCTL_ACC, RTMCTL_ERR); + +} +EXPORT_SYMBOL(caam_dump_rng_regs); + /* * Descriptor to instantiate RNG State Handle 0 in normal mode and * load the JDKEK, TDKEK and TDSK registers diff --git a/drivers/crypto/caam/ctrl.h b/drivers/crypto/caam/ctrl.h index f3ecd67922a7..806f4563990c 100644 --- a/drivers/crypto/caam/ctrl.h +++ b/drivers/crypto/caam/ctrl.h @@ -11,4 +11,6 @@ /* Prototypes for backend-level services exposed to APIs */ extern bool caam_dpaa2; +void caam_dump_rng_regs(struct device *ctrldev); + #endif /* CTRL_H */ diff --git a/drivers/crypto/caam/regs.h b/drivers/crypto/caam/regs.h index af61f3a2c0d4..dfc25a458a55 100644 --- a/drivers/crypto/caam/regs.h +++ b/drivers/crypto/caam/regs.h @@ -493,6 +493,7 @@ struct rngtst { /* RNG4 TRNG test registers */ struct rng4tst { #define RTMCTL_ACC BIT(5) /* TRNG access mode */ +#define RTMCTL_ERR BIT(12) /* TRNG error */ #define RTMCTL_PRGM BIT(16) /* 1 -> program mode, 0 -> run mode */ #define RTMCTL_SAMP_MODE_VON_NEUMANN_ES_SC 0 /* use von Neumann data in both entropy shifter and @@ -526,7 +527,9 @@ struct rng4tst { u32 rtfrqmax; /* PRGM=1: freq. count max. limit register */ u32 rtfrqcnt; /* PRGM=0: freq. count register */ }; - u32 rsvd1[40]; + u32 rsvd[7]; + u32 rtstatus; /* TRNG status register */ + u32 rsvd1[32]; #define RDSTA_SKVT 0x80000000 #define RDSTA_SKVN 0x40000000 #define RDSTA_PR0 BIT(4) -- 2.17.1 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: CAAM RNG trouble 2021-03-02 17:33 ` Horia Geantă @ 2021-03-03 0:37 ` Robert Hancock 2021-11-12 15:27 ` [RFC PATCH] crypto: caam - restore retry count after HW RNG failure Michal Vokáč 0 siblings, 1 reply; 10+ messages in thread From: Robert Hancock @ 2021-03-03 0:37 UTC (permalink / raw) To: l.stach, alexandru.porosanu, aymen.sghaier, horia.geanta Cc: linux-crypto, kernel On Tue, 2021-03-02 at 19:33 +0200, Horia Geantă wrote: > On 12/14/2020 9:00 PM, Lucas Stach wrote: > > Hi all, > > > > I've been looking into a CAAM RNG issue for a while, where I could need > > some input from people knowing the CAAM hardware better than I do. > > Basically the issue is that on some i.MX6 units the RNG functionality > > sometimes fails with this error: > > caam_jr 2101000.jr0: 20003c5b: CCB: desc idx 60: RNG: Hardware error. > > > > I can tell that it is related to the entropy delay. On all failing > > units the RNG4 gets instantiated with the default entropy delay of > > 3200. If I dial up the delay to 3600 or 4000 the RNG works reliably. As > > a negative test I changed the initial delay to 400. With this change > > all units are able to successfully instantiate the RNG handles at an > > entropy delay of 2000 or 2400, but then reliably fail at getting random > > data with the error shown above. I guess the issue is related to > > prediction resistance on the handles, which causes the PRNG to be re- > > seeded from the TRNG fairly often. > > > > Now I don't have a good idea on how to arrive at a reliably working > > entropy delay setting, as apparently the simple "are we able to > > instantiate the handle" check is not enough to actually guarantee a > > working RNG setup. Any suggestions? > > > The successful instantiation of the RNG state handle(s) means that > the HW self-tests passed, but this doesn't mean RNG will work flawlessly. > > A properly configured RNG should have a certain (very low) failure rate. > The logic in the caam rng driver is not checking this rate, since it's > running > only once with a given configuration. > OTOH properly checking the RNG configuration would take some time, so it > would > be better to run it offline. The "characterization" should also account for > temperature, voltage and process (fixed for a given SoC). > > From this perspective, the caam rng driver should be updated to statically > configure the RNG with these offline-determined parameters. > Ideally we'd be able to use a single set of parameters to cover all SoCs > that have the same IP (RNG4 TRNG). > Unfortunately we're not there yet. > > The situation became more visible after changing the caam rng driver to > reseed > the PRNG before every request (practically making the PRNG function like a > TRNG, > a hwrng framework requirement), since the HW self-tests are now running more > often then before. > > Some questions that would give me more details about the exact issue you > and Robert are facing: > > 1. What SoC exactly are you running on? > > 2. How fast and how often is the RNG hardware error occurring? > Does this happen at boot time, only when stressing /dev/hwrng etc.? We are using an iMX6D. In our case, it seems this is occurring relatively rarely - I have only seen this occur on a few boots. When it has happened, it started reporting errors at boot and regularly thereafter - probably as a result of accesses being made by the rngd daemon. > > 3. Try dumping some of the RNG registers using below patch: > > -- >8 -- > > Subject: [PATCH] crypto: caam - rng debugging > > Dump RNG registers at hwrng.init time and in case descriptor returns > RNG HW error. > > Signed-off-by: Horia Geantă <horia.geanta@nxp.com> > --- > drivers/crypto/caam/caamrng.c | 9 ++++++++- > drivers/crypto/caam/ctrl.c | 29 +++++++++++++++++++++++++++++ > drivers/crypto/caam/ctrl.h | 2 ++ > drivers/crypto/caam/regs.h | 5 ++++- > 4 files changed, 43 insertions(+), 2 deletions(-) > > diff --git a/drivers/crypto/caam/caamrng.c b/drivers/crypto/caam/caamrng.c > index 77d048dfe5d0..fc2192183696 100644 > --- a/drivers/crypto/caam/caamrng.c > +++ b/drivers/crypto/caam/caamrng.c > @@ -16,6 +16,7 @@ > > #include "compat.h" > > +#include "ctrl.h" > #include "regs.h" > #include "intern.h" > #include "desc_constr.h" > @@ -57,9 +58,12 @@ static void caam_rng_done(struct device *jrdev, u32 *desc, > u32 err, > { > struct caam_rng_job_ctx *jctx = context; > > - if (err) > + if (err) { > *jctx->err = caam_jr_strstatus(jrdev, err); > > + caam_dump_rng_regs(jrdev); > + } > + > complete(jctx->done); > } > > @@ -199,6 +203,9 @@ static int caam_init(struct hwrng *rng) > return err; > } > > + dev_dbg(ctx->jrdev, "CAAM RNG - register status at hwrng.init time\n"); > + caam_dump_rng_regs(ctx->jrdev); > + > /* > * Fill async buffer to have early randomness data for > * hw_random > diff --git a/drivers/crypto/caam/ctrl.c b/drivers/crypto/caam/ctrl.c > index ca0361b2dbb0..52db32b599aa 100644 > --- a/drivers/crypto/caam/ctrl.c > +++ b/drivers/crypto/caam/ctrl.c > @@ -27,6 +27,35 @@ EXPORT_SYMBOL(caam_dpaa2); > #include "qi.h" > #endif > > +void caam_dump_rng_regs(struct device *jrdev) > +{ > + struct device *ctrldev = jrdev->parent; > + struct caam_drv_private *ctrlpriv = dev_get_drvdata(ctrldev); > + struct caam_ctrl __iomem *ctrl; > + struct rng4tst __iomem *r4tst; > + u32 rtmctl; > + > + dev_dbg(jrdev, "RNG register dump:\n"); > + > + ctrl = (struct caam_ctrl __iomem *)ctrlpriv->ctrl; > + r4tst = &ctrl->r4tst[0]; > + > + dev_dbg(jrdev, "\trdsta = 0x%08x\n", rd_reg32(&r4tst->rdsta)); > + > + rtmctl = rd_reg32(&r4tst->rtmctl); > + dev_dbg(jrdev, "\trtmctl = 0x%08x\n", rtmctl); > + dev_dbg(jrdev, "\trtstatus = 0x%08x\n", rd_reg32(&r4tst->rtstatus)); > + > + /* Group of registers that can be read only when RTMCTL[PRGM]=1 */ > + clrsetbits_32(&r4tst->rtmctl, 0, RTMCTL_PRGM | RTMCTL_ACC); > + dev_dbg(jrdev, "\trtscmisc = 0x%08x\n", rd_reg32(&r4tst->rtscmisc)); > + dev_dbg(jrdev, "\trtfrqmin = 0x%08x\n", rd_reg32(&r4tst->rtfrqmin)); > + dev_dbg(jrdev, "\trtfrqmax = 0x%08x\n", rd_reg32(&r4tst->rtfrqmax)); > + clrsetbits_32(&r4tst->rtmctl, RTMCTL_PRGM | RTMCTL_ACC, RTMCTL_ERR); > + > +} > +EXPORT_SYMBOL(caam_dump_rng_regs); > + > /* > * Descriptor to instantiate RNG State Handle 0 in normal mode and > * load the JDKEK, TDKEK and TDSK registers > diff --git a/drivers/crypto/caam/ctrl.h b/drivers/crypto/caam/ctrl.h > index f3ecd67922a7..806f4563990c 100644 > --- a/drivers/crypto/caam/ctrl.h > +++ b/drivers/crypto/caam/ctrl.h > @@ -11,4 +11,6 @@ > /* Prototypes for backend-level services exposed to APIs */ > extern bool caam_dpaa2; > > +void caam_dump_rng_regs(struct device *ctrldev); > + > #endif /* CTRL_H */ > diff --git a/drivers/crypto/caam/regs.h b/drivers/crypto/caam/regs.h > index af61f3a2c0d4..dfc25a458a55 100644 > --- a/drivers/crypto/caam/regs.h > +++ b/drivers/crypto/caam/regs.h > @@ -493,6 +493,7 @@ struct rngtst { > /* RNG4 TRNG test registers */ > struct rng4tst { > #define RTMCTL_ACC BIT(5) /* TRNG access mode */ > +#define RTMCTL_ERR BIT(12) /* TRNG error */ > #define RTMCTL_PRGM BIT(16) /* 1 -> program mode, 0 -> run mode */ > #define RTMCTL_SAMP_MODE_VON_NEUMANN_ES_SC 0 /* use von Neumann data in > both entropy shifter and > @@ -526,7 +527,9 @@ struct rng4tst { > u32 rtfrqmax; /* PRGM=1: freq. count max. limit register */ > u32 rtfrqcnt; /* PRGM=0: freq. count register */ > }; > - u32 rsvd1[40]; > + u32 rsvd[7]; > + u32 rtstatus; /* TRNG status register */ > + u32 rsvd1[32]; > #define RDSTA_SKVT 0x80000000 > #define RDSTA_SKVN 0x40000000 > #define RDSTA_PR0 BIT(4) -- Robert Hancock Senior Hardware Designer, Calian Advanced Technologies www.calian.com ^ permalink raw reply [flat|nested] 10+ messages in thread
* [RFC PATCH] crypto: caam - restore retry count after HW RNG failure 2021-03-03 0:37 ` Robert Hancock @ 2021-11-12 15:27 ` Michal Vokáč [not found] ` <DU2PR04MB863088F218774551FA254EB795539@DU2PR04MB8630.eurprd04.prod.outlook.com> 0 siblings, 1 reply; 10+ messages in thread From: Michal Vokáč @ 2021-11-12 15:27 UTC (permalink / raw) To: Horia Geantă, Pankaj Gupta, Herbert Xu Cc: David S. Miller, linux-crypto, linux-kernel, l.stach, robert.hancock, Petr Benes, petrben, Michal Vokáč From: Petr Benes <petr.benes@ysoft.com> Each time TRNG generates entropy, statistical tests are run. If they fail, RETRY_COUNT value is decremented. Once it reaches 0, HW RNG returns an error, and needs to be reset. RETRY_COUNT could be programmed in RTSCMISC register and is set to 1 by default. Hence, we are left without hwrng after the first error, which could happen even under normal conditions. Cc: petrben@gmail.com Signed-off-by: Petr Benes <petr.benes@ysoft.com> Signed-off-by: Michal Vokáč <michal.vokac@ysoft.com> --- Hi, we are also experiencing this issue: caam_jr 2101000.jr0: 20003c5b: CCB: desc idx 60: RNG: Hardware error. It is happening on both i.MX6S and i.MX6DL SoCs we use. On Solo I can reproduce it really fast. Sometimes it happens right after the board is NFS booted, sometimes I need to stress the HWRNG for a while (generate few hundred KBs of random data). On some DualLite SoCs it is happening at least once a day. We are using the v5.10 LTS branch but I can confirm that this is happening on all kernels since v5.7 to the latest linux-next. We also tried to increase the RTSDCTL_ENT_DLY_MIN delay as suggested in this thread [1]. It helped and the issue never occurred since then but we are looking for more universal and permanent solution suitable for upstream, hence we came up with this patch. Any comments will be appreciated. Thanks, Michal [1] https://lkml.org/lkml/2021/8/30/296 drivers/crypto/caam/caamrng.c | 42 ++++++++++++++++++++++++++++++++--- drivers/crypto/caam/ctrl.c | 13 +++++++++++ drivers/crypto/caam/ctrl.h | 2 ++ 3 files changed, 54 insertions(+), 3 deletions(-) diff --git a/drivers/crypto/caam/caamrng.c b/drivers/crypto/caam/caamrng.c index 77d048dfe5d0..2be5584ae591 100644 --- a/drivers/crypto/caam/caamrng.c +++ b/drivers/crypto/caam/caamrng.c @@ -21,6 +21,7 @@ #include "desc_constr.h" #include "jr.h" #include "error.h" +#include "ctrl.h" #define CAAM_RNG_MAX_FIFO_STORE_SIZE 16 @@ -113,6 +114,35 @@ static int caam_rng_read_one(struct device *jrdev, return err ?: (ret ?: len); } +static void caam_rng_retry_reset(struct caam_rng_ctx *context) +{ + struct device *ctrldev = context->ctrldev; + struct caam_drv_private *ctrlpriv = dev_get_drvdata(ctrldev); + struct caam_ctrl __iomem *ctrl; + struct rng4tst __iomem *r4tst; + u32 __iomem *rtstatus; + u32 retry_count; + + ctrl = (struct caam_ctrl __iomem *)ctrlpriv->ctrl; + r4tst = &ctrl->r4tst[0]; + + /* + * There is unfortunately no member for RTSTATUS register in + * struct rng4tst and the structure doesn't look stable + */ + rtstatus = (u32 *)((char *)&ctrl->r4tst[0] + 0x3C); + retry_count = (rd_reg32(rtstatus) >> 16) & 0xf; + dev_dbg(ctrldev, "CAAM RNG retry count %d\n", retry_count); + if (retry_count == 0) { + dev_err(ctrldev, "CAAM RNG resetting retry count to 1\n"); + clrsetbits_32(&r4tst->rtmctl, 0, RTMCTL_PRGM | RTMCTL_ACC); + wr_reg32(&r4tst->rtscmisc, (rd_reg32(&r4tst->rtscmisc) & 0x7f) | (1 << 16)); + clrsetbits_32(&r4tst->rtmctl, RTMCTL_PRGM | RTMCTL_ACC, + RTMCTL_SAMP_MODE_RAW_ES_SC); + caam_reinstantiate_rng(ctrldev); + } +} + static void caam_rng_fill_async(struct caam_rng_ctx *ctx) { struct scatterlist sg[1]; @@ -129,8 +159,10 @@ static void caam_rng_fill_async(struct caam_rng_ctx *ctx) sg[0].length, ctx->desc_async, &done); - if (len < 0) + if (len < 0) { + caam_rng_retry_reset(ctx); return; + } kfifo_dma_in_finish(&ctx->fifo, len); } @@ -145,13 +177,17 @@ static void caam_rng_worker(struct work_struct *work) static int caam_read(struct hwrng *rng, void *dst, size_t max, bool wait) { struct caam_rng_ctx *ctx = to_caam_rng_ctx(rng); - int out; + int out, ret; if (wait) { struct completion done; - return caam_rng_read_one(ctx->jrdev, dst, max, + ret = caam_rng_read_one(ctx->jrdev, dst, max, ctx->desc_sync, &done); + if (ret < 0) + caam_rng_retry_reset(ctx); + + return ret; } out = kfifo_out(&ctx->fifo, dst, max); diff --git a/drivers/crypto/caam/ctrl.c b/drivers/crypto/caam/ctrl.c index ca0361b2dbb0..e421f8d1982b 100644 --- a/drivers/crypto/caam/ctrl.c +++ b/drivers/crypto/caam/ctrl.c @@ -339,6 +339,19 @@ static int instantiate_rng(struct device *ctrldev, int state_handle_mask, return devm_add_action_or_reset(ctrldev, devm_deinstantiate_rng, ctrldev); } +/* + * caam_reinstantiate_rng - reinstantiates RNG. Intended for a case when RNG falls into + * HW error condition. That happens if TRNG fails statistical + * check and RTY_CNT value set in RTSCMISC decrements to zero. + * It is exported to caamrng.c + * @ctrldev - pointer to device + */ + +int caam_reinstantiate_rng(struct device *ctrldev) +{ + return instantiate_rng(ctrldev, 0, 0); +} + /* * kick_trng - sets the various parameters for enabling the initialization * of the RNG4 block in CAAM diff --git a/drivers/crypto/caam/ctrl.h b/drivers/crypto/caam/ctrl.h index f3ecd67922a7..26ff5a49a865 100644 --- a/drivers/crypto/caam/ctrl.h +++ b/drivers/crypto/caam/ctrl.h @@ -8,6 +8,8 @@ #ifndef CTRL_H #define CTRL_H +int caam_reinstantiate_rng(struct device *ctrldev); + /* Prototypes for backend-level services exposed to APIs */ extern bool caam_dpaa2; -- 2.25.1 ^ permalink raw reply related [flat|nested] 10+ messages in thread
[parent not found: <DU2PR04MB863088F218774551FA254EB795539@DU2PR04MB8630.eurprd04.prod.outlook.com>]
* RE: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG failure [not found] ` <DU2PR04MB863088F218774551FA254EB795539@DU2PR04MB8630.eurprd04.prod.outlook.com> @ 2022-01-13 7:23 ` Gaurav Jain 2022-01-17 14:08 ` Petr Benes 2022-01-20 9:59 ` Petr Benes 0 siblings, 2 replies; 10+ messages in thread From: Gaurav Jain @ 2022-01-13 7:23 UTC (permalink / raw) To: Horia Geanta, Pankaj Gupta, "Herbert Xu", "Michal Vokáč" Cc: "David S. Miller", "Petr Benes", petrben, l.stach, robert.hancock, linux-crypto, linux-kernel, Varun Sethi, Vabhav Sharma Hello Michal > > -----Original Message----- > From: Michal Vokáč <michal.vokac@ysoft.com> > Sent: Friday, November 12, 2021 8:57 PM > To: Horia Geanta <horia.geanta@nxp.com>; Pankaj Gupta > <pankaj.gupta@nxp.com>; Herbert Xu <herbert@gondor.apana.org.au> > Cc: David S. Miller <davem@davemloft.net>; linux-crypto@vger.kernel.org; > linux-kernel@vger.kernel.org; l.stach@pengutronix.de; > robert.hancock@calian.com; Petr Benes <petr.benes@ysoft.com>; > petrben@gmail.com; Michal Vokáč <michal.vokac@ysoft.com> > Subject: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG > failure > > Caution: EXT Email > > From: Petr Benes <petr.benes@ysoft.com> > > Each time TRNG generates entropy, statistical tests are run. > If they fail, RETRY_COUNT value is decremented. Once it reaches 0, HW RNG > returns an error, and needs to be reset. The least-significant 16 bits of the RTSTATUS register reflect the result of each of statistical tests. Can you dump RTSTATUS to see which test has failed? Please dump RTMCTL as well. Have you tried to build this patch for armv8? For me it is failing. Regards Gaurav Jain > RETRY_COUNT could be programmed in RTSCMISC register and is set to 1 by > default. Hence, we are left without hwrng after the first error, which could > happen even under normal conditions. > > Cc: petrben@gmail.com > Signed-off-by: Petr Benes <petr.benes@ysoft.com> > Signed-off-by: Michal Vokáč <michal.vokac@ysoft.com> > --- > Hi, > we are also experiencing this issue: > > caam_jr 2101000.jr0: 20003c5b: CCB: desc idx 60: RNG: Hardware error. > > It is happening on both i.MX6S and i.MX6DL SoCs we use. > On Solo I can reproduce it really fast. Sometimes it happens right after the board > is NFS booted, sometimes I need to stress the HWRNG for a while (generate few > hundred KBs of random data). On some DualLite SoCs it is happening at least > once a day. > > We are using the v5.10 LTS branch but I can confirm that this is happening on all > kernels since v5.7 to the latest linux-next. > > We also tried to increase the RTSDCTL_ENT_DLY_MIN delay as suggested in this > thread [1]. It helped and the issue never occurred since then but we are looking > for more universal and permanent solution suitable for upstream, hence we > came up with this patch. > > Any comments will be appreciated. > Thanks, Michal > > [1] > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flkml.org > %2Flkml%2F2021%2F8%2F30%2F296&data=04%7C01%7Cpankaj.gupta%40 > nxp.com%7C5d6bf460b260415aeda008d9a5f0ffab%7C686ea1d3bc2b4c6fa92cd > 99c5c301635%7C0%7C0%7C637723276775699794%7CUnknown%7CTWFpbGZs > b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0 > %3D%7C1000&sdata=VxN9MqDFbISptW3OX9XTtZ%2FwQTsEbF6dETxXB%2 > BHrywg%3D&reserved=0 > > drivers/crypto/caam/caamrng.c | 42 ++++++++++++++++++++++++++++++++--- > drivers/crypto/caam/ctrl.c | 13 +++++++++++ > drivers/crypto/caam/ctrl.h | 2 ++ > 3 files changed, 54 insertions(+), 3 deletions(-) > > diff --git a/drivers/crypto/caam/caamrng.c b/drivers/crypto/caam/caamrng.c > index 77d048dfe5d0..2be5584ae591 100644 > --- a/drivers/crypto/caam/caamrng.c > +++ b/drivers/crypto/caam/caamrng.c > @@ -21,6 +21,7 @@ > #include "desc_constr.h" > #include "jr.h" > #include "error.h" > +#include "ctrl.h" > > #define CAAM_RNG_MAX_FIFO_STORE_SIZE 16 > > @@ -113,6 +114,35 @@ static int caam_rng_read_one(struct device *jrdev, > return err ?: (ret ?: len); > } > > +static void caam_rng_retry_reset(struct caam_rng_ctx *context) { > + struct device *ctrldev = context->ctrldev; > + struct caam_drv_private *ctrlpriv = dev_get_drvdata(ctrldev); > + struct caam_ctrl __iomem *ctrl; > + struct rng4tst __iomem *r4tst; > + u32 __iomem *rtstatus; > + u32 retry_count; > + > + ctrl = (struct caam_ctrl __iomem *)ctrlpriv->ctrl; > + r4tst = &ctrl->r4tst[0]; > + > + /* > + * There is unfortunately no member for RTSTATUS register in > + * struct rng4tst and the structure doesn't look stable > + */ > + rtstatus = (u32 *)((char *)&ctrl->r4tst[0] + 0x3C); > + retry_count = (rd_reg32(rtstatus) >> 16) & 0xf; > + dev_dbg(ctrldev, "CAAM RNG retry count %d\n", retry_count); > + if (retry_count == 0) { > + dev_err(ctrldev, "CAAM RNG resetting retry count to 1\n"); > + clrsetbits_32(&r4tst->rtmctl, 0, RTMCTL_PRGM | RTMCTL_ACC); > + wr_reg32(&r4tst->rtscmisc, (rd_reg32(&r4tst->rtscmisc) & 0x7f) | (1 << > 16)); > + clrsetbits_32(&r4tst->rtmctl, RTMCTL_PRGM | RTMCTL_ACC, > + RTMCTL_SAMP_MODE_RAW_ES_SC); > + caam_reinstantiate_rng(ctrldev); > + } > +} > + > static void caam_rng_fill_async(struct caam_rng_ctx *ctx) { > struct scatterlist sg[1]; > @@ -129,8 +159,10 @@ static void caam_rng_fill_async(struct caam_rng_ctx > *ctx) > sg[0].length, > ctx->desc_async, > &done); > - if (len < 0) > + if (len < 0) { > + caam_rng_retry_reset(ctx); > return; > + } > > kfifo_dma_in_finish(&ctx->fifo, len); } @@ -145,13 +177,17 @@ static void > caam_rng_worker(struct work_struct *work) static int caam_read(struct hwrng > *rng, void *dst, size_t max, bool wait) { > struct caam_rng_ctx *ctx = to_caam_rng_ctx(rng); > - int out; > + int out, ret; > > if (wait) { > struct completion done; > > - return caam_rng_read_one(ctx->jrdev, dst, max, > + ret = caam_rng_read_one(ctx->jrdev, dst, max, > ctx->desc_sync, &done); > + if (ret < 0) > + caam_rng_retry_reset(ctx); > + > + return ret; > } > > out = kfifo_out(&ctx->fifo, dst, max); diff --git a/drivers/crypto/caam/ctrl.c > b/drivers/crypto/caam/ctrl.c index ca0361b2dbb0..e421f8d1982b 100644 > --- a/drivers/crypto/caam/ctrl.c > +++ b/drivers/crypto/caam/ctrl.c > @@ -339,6 +339,19 @@ static int instantiate_rng(struct device *ctrldev, int > state_handle_mask, > return devm_add_action_or_reset(ctrldev, devm_deinstantiate_rng, > ctrldev); } > > +/* > + * caam_reinstantiate_rng - reinstantiates RNG. Intended for a case when RNG > falls into > + * HW error condition. That happens if TRNG fails statistical > + * check and RTY_CNT value set in RTSCMISC decrements to zero. > + * It is exported to caamrng.c > + * @ctrldev - pointer to device > + */ > + > +int caam_reinstantiate_rng(struct device *ctrldev) { > + return instantiate_rng(ctrldev, 0, 0); } > + > /* > * kick_trng - sets the various parameters for enabling the initialization > * of the RNG4 block in CAAM > diff --git a/drivers/crypto/caam/ctrl.h b/drivers/crypto/caam/ctrl.h index > f3ecd67922a7..26ff5a49a865 100644 > --- a/drivers/crypto/caam/ctrl.h > +++ b/drivers/crypto/caam/ctrl.h > @@ -8,6 +8,8 @@ > #ifndef CTRL_H > #define CTRL_H > > +int caam_reinstantiate_rng(struct device *ctrldev); > + > /* Prototypes for backend-level services exposed to APIs */ extern bool > caam_dpaa2; > > -- > 2.25.1 ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG failure 2022-01-13 7:23 ` [EXT] " Gaurav Jain @ 2022-01-17 14:08 ` Petr Benes 2022-01-18 9:04 ` Gaurav Jain 2022-01-20 9:59 ` Petr Benes 1 sibling, 1 reply; 10+ messages in thread From: Petr Benes @ 2022-01-17 14:08 UTC (permalink / raw) To: Gaurav Jain Cc: Horia Geanta, Pankaj Gupta, Herbert Xu, Michal Vokáč, David S. Miller, Petr Benes, l.stach, robert.hancock, linux-crypto, linux-kernel, Varun Sethi, Vabhav Sharma Hi Gaurav, I've resumed work on this issue. I hope I'll get RTSTATUS and RTMCTL soon. Meanwhile, what is the problem with armv8? I'll try some build for armv8, but it will take time. Regards, Petr On Thu, 13 Jan 2022 at 08:23, Gaurav Jain <gaurav.jain@nxp.com> wrote: > > Hello Michal > > > > > -----Original Message----- > > From: Michal Vokáč <michal.vokac@ysoft.com> > > Sent: Friday, November 12, 2021 8:57 PM > > To: Horia Geanta <horia.geanta@nxp.com>; Pankaj Gupta > > <pankaj.gupta@nxp.com>; Herbert Xu <herbert@gondor.apana.org.au> > > Cc: David S. Miller <davem@davemloft.net>; linux-crypto@vger.kernel.org; > > linux-kernel@vger.kernel.org; l.stach@pengutronix.de; > > robert.hancock@calian.com; Petr Benes <petr.benes@ysoft.com>; > > petrben@gmail.com; Michal Vokáč <michal.vokac@ysoft.com> > > Subject: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG > > failure > > > > Caution: EXT Email > > > > From: Petr Benes <petr.benes@ysoft.com> > > > > Each time TRNG generates entropy, statistical tests are run. > > If they fail, RETRY_COUNT value is decremented. Once it reaches 0, HW RNG > > returns an error, and needs to be reset. > > The least-significant 16 bits of the RTSTATUS register reflect the result of each of statistical tests. > Can you dump RTSTATUS to see which test has failed? Please dump RTMCTL as well. > > Have you tried to build this patch for armv8? For me it is failing. > > Regards > Gaurav Jain > > > RETRY_COUNT could be programmed in RTSCMISC register and is set to 1 by > > default. Hence, we are left without hwrng after the first error, which could > > happen even under normal conditions. > > > > Cc: petrben@gmail.com > > Signed-off-by: Petr Benes <petr.benes@ysoft.com> > > Signed-off-by: Michal Vokáč <michal.vokac@ysoft.com> > > --- > > Hi, > > we are also experiencing this issue: > > > > caam_jr 2101000.jr0: 20003c5b: CCB: desc idx 60: RNG: Hardware error. > > > > It is happening on both i.MX6S and i.MX6DL SoCs we use. > > On Solo I can reproduce it really fast. Sometimes it happens right after the board > > is NFS booted, sometimes I need to stress the HWRNG for a while (generate few > > hundred KBs of random data). On some DualLite SoCs it is happening at least > > once a day. > > > > We are using the v5.10 LTS branch but I can confirm that this is happening on all > > kernels since v5.7 to the latest linux-next. > > > > We also tried to increase the RTSDCTL_ENT_DLY_MIN delay as suggested in this > > thread [1]. It helped and the issue never occurred since then but we are looking > > for more universal and permanent solution suitable for upstream, hence we > > came up with this patch. > > > > Any comments will be appreciated. > > Thanks, Michal > > > > [1] > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flkml.org > > %2Flkml%2F2021%2F8%2F30%2F296&data=04%7C01%7Cpankaj.gupta%40 > > nxp.com%7C5d6bf460b260415aeda008d9a5f0ffab%7C686ea1d3bc2b4c6fa92cd > > 99c5c301635%7C0%7C0%7C637723276775699794%7CUnknown%7CTWFpbGZs > > b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0 > > %3D%7C1000&sdata=VxN9MqDFbISptW3OX9XTtZ%2FwQTsEbF6dETxXB%2 > > BHrywg%3D&reserved=0 > > > > drivers/crypto/caam/caamrng.c | 42 ++++++++++++++++++++++++++++++++--- > > drivers/crypto/caam/ctrl.c | 13 +++++++++++ > > drivers/crypto/caam/ctrl.h | 2 ++ > > 3 files changed, 54 insertions(+), 3 deletions(-) > > > > diff --git a/drivers/crypto/caam/caamrng.c b/drivers/crypto/caam/caamrng.c > > index 77d048dfe5d0..2be5584ae591 100644 > > --- a/drivers/crypto/caam/caamrng.c > > +++ b/drivers/crypto/caam/caamrng.c > > @@ -21,6 +21,7 @@ > > #include "desc_constr.h" > > #include "jr.h" > > #include "error.h" > > +#include "ctrl.h" > > > > #define CAAM_RNG_MAX_FIFO_STORE_SIZE 16 > > > > @@ -113,6 +114,35 @@ static int caam_rng_read_one(struct device *jrdev, > > return err ?: (ret ?: len); > > } > > > > +static void caam_rng_retry_reset(struct caam_rng_ctx *context) { > > + struct device *ctrldev = context->ctrldev; > > + struct caam_drv_private *ctrlpriv = dev_get_drvdata(ctrldev); > > + struct caam_ctrl __iomem *ctrl; > > + struct rng4tst __iomem *r4tst; > > + u32 __iomem *rtstatus; > > + u32 retry_count; > > + > > + ctrl = (struct caam_ctrl __iomem *)ctrlpriv->ctrl; > > + r4tst = &ctrl->r4tst[0]; > > + > > + /* > > + * There is unfortunately no member for RTSTATUS register in > > + * struct rng4tst and the structure doesn't look stable > > + */ > > + rtstatus = (u32 *)((char *)&ctrl->r4tst[0] + 0x3C); > > + retry_count = (rd_reg32(rtstatus) >> 16) & 0xf; > > + dev_dbg(ctrldev, "CAAM RNG retry count %d\n", retry_count); > > + if (retry_count == 0) { > > + dev_err(ctrldev, "CAAM RNG resetting retry count to 1\n"); > > + clrsetbits_32(&r4tst->rtmctl, 0, RTMCTL_PRGM | RTMCTL_ACC); > > + wr_reg32(&r4tst->rtscmisc, (rd_reg32(&r4tst->rtscmisc) & 0x7f) | (1 << > > 16)); > > + clrsetbits_32(&r4tst->rtmctl, RTMCTL_PRGM | RTMCTL_ACC, > > + RTMCTL_SAMP_MODE_RAW_ES_SC); > > + caam_reinstantiate_rng(ctrldev); > > + } > > +} > > + > > static void caam_rng_fill_async(struct caam_rng_ctx *ctx) { > > struct scatterlist sg[1]; > > @@ -129,8 +159,10 @@ static void caam_rng_fill_async(struct caam_rng_ctx > > *ctx) > > sg[0].length, > > ctx->desc_async, > > &done); > > - if (len < 0) > > + if (len < 0) { > > + caam_rng_retry_reset(ctx); > > return; > > + } > > > > kfifo_dma_in_finish(&ctx->fifo, len); } @@ -145,13 +177,17 @@ static void > > caam_rng_worker(struct work_struct *work) static int caam_read(struct hwrng > > *rng, void *dst, size_t max, bool wait) { > > struct caam_rng_ctx *ctx = to_caam_rng_ctx(rng); > > - int out; > > + int out, ret; > > > > if (wait) { > > struct completion done; > > > > - return caam_rng_read_one(ctx->jrdev, dst, max, > > + ret = caam_rng_read_one(ctx->jrdev, dst, max, > > ctx->desc_sync, &done); > > + if (ret < 0) > > + caam_rng_retry_reset(ctx); > > + > > + return ret; > > } > > > > out = kfifo_out(&ctx->fifo, dst, max); diff --git a/drivers/crypto/caam/ctrl.c > > b/drivers/crypto/caam/ctrl.c index ca0361b2dbb0..e421f8d1982b 100644 > > --- a/drivers/crypto/caam/ctrl.c > > +++ b/drivers/crypto/caam/ctrl.c > > @@ -339,6 +339,19 @@ static int instantiate_rng(struct device *ctrldev, int > > state_handle_mask, > > return devm_add_action_or_reset(ctrldev, devm_deinstantiate_rng, > > ctrldev); } > > > > +/* > > + * caam_reinstantiate_rng - reinstantiates RNG. Intended for a case when RNG > > falls into > > + * HW error condition. That happens if TRNG fails statistical > > + * check and RTY_CNT value set in RTSCMISC decrements to zero. > > + * It is exported to caamrng.c > > + * @ctrldev - pointer to device > > + */ > > + > > +int caam_reinstantiate_rng(struct device *ctrldev) { > > + return instantiate_rng(ctrldev, 0, 0); } > > + > > /* > > * kick_trng - sets the various parameters for enabling the initialization > > * of the RNG4 block in CAAM > > diff --git a/drivers/crypto/caam/ctrl.h b/drivers/crypto/caam/ctrl.h index > > f3ecd67922a7..26ff5a49a865 100644 > > --- a/drivers/crypto/caam/ctrl.h > > +++ b/drivers/crypto/caam/ctrl.h > > @@ -8,6 +8,8 @@ > > #ifndef CTRL_H > > #define CTRL_H > > > > +int caam_reinstantiate_rng(struct device *ctrldev); > > + > > /* Prototypes for backend-level services exposed to APIs */ extern bool > > caam_dpaa2; > > > > -- > > 2.25.1 > ^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG failure 2022-01-17 14:08 ` Petr Benes @ 2022-01-18 9:04 ` Gaurav Jain 0 siblings, 0 replies; 10+ messages in thread From: Gaurav Jain @ 2022-01-18 9:04 UTC (permalink / raw) To: Petr Benes Cc: Horia Geanta, Pankaj Gupta, Herbert Xu, Michal Vokáč, David S. Miller, Petr Benes, l.stach, robert.hancock, linux-crypto, linux-kernel, Varun Sethi, Vabhav Sharma Hi Petr Recently I have sent patches in upstream uboot for adding caam driver support. http://patchwork.ozlabs.org/project/uboot/list/?series=280725 if possible pls add this series in uboot and try to reproduce the problem. Problem with armv8: ERROR: modpost: "caam_reinstantiate_rng" [drivers/crypto/caam/caam_jr.ko] undefined! make[1]: *** [scripts/Makefile.modpost:134: modules-only.symvers] Error 1 make[1]: *** Deleting file 'modules-only.symvers' make: *** [Makefile:1755: modules] Error 2 Regards Gaurav Jain > -----Original Message----- > From: Petr Benes <petrben@gmail.com> > Sent: Monday, January 17, 2022 7:39 PM > To: Gaurav Jain <gaurav.jain@nxp.com> > Cc: Horia Geanta <horia.geanta@nxp.com>; Pankaj Gupta > <pankaj.gupta@nxp.com>; Herbert Xu <herbert@gondor.apana.org.au>; Michal > Vokáč <michal.vokac@ysoft.com>; David S. Miller <davem@davemloft.net>; > Petr Benes <petr.benes@ysoft.com>; l.stach@pengutronix.de; > robert.hancock@calian.com; linux-crypto@vger.kernel.org; linux- > kernel@vger.kernel.org; Varun Sethi <V.Sethi@nxp.com>; Vabhav Sharma > <vabhav.sharma@nxp.com> > Subject: Re: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG > failure > > Caution: EXT Email > > Hi Gaurav, > > I've resumed work on this issue. I hope I'll get RTSTATUS and RTMCTL soon. > Meanwhile, what is the problem with armv8? I'll try some build for armv8, but it > will take time. > > Regards, > Petr > > On Thu, 13 Jan 2022 at 08:23, Gaurav Jain <gaurav.jain@nxp.com> wrote: > > > > Hello Michal > > > > > > > > -----Original Message----- > > > From: Michal Vokáč <michal.vokac@ysoft.com> > > > Sent: Friday, November 12, 2021 8:57 PM > > > To: Horia Geanta <horia.geanta@nxp.com>; Pankaj Gupta > > > <pankaj.gupta@nxp.com>; Herbert Xu <herbert@gondor.apana.org.au> > > > Cc: David S. Miller <davem@davemloft.net>; > > > linux-crypto@vger.kernel.org; linux-kernel@vger.kernel.org; > > > l.stach@pengutronix.de; robert.hancock@calian.com; Petr Benes > > > <petr.benes@ysoft.com>; petrben@gmail.com; Michal Vokáč > > > <michal.vokac@ysoft.com> > > > Subject: [EXT] [RFC PATCH] crypto: caam - restore retry count after > > > HW RNG failure > > > > > > Caution: EXT Email > > > > > > From: Petr Benes <petr.benes@ysoft.com> > > > > > > Each time TRNG generates entropy, statistical tests are run. > > > If they fail, RETRY_COUNT value is decremented. Once it reaches 0, > > > HW RNG returns an error, and needs to be reset. > > > > The least-significant 16 bits of the RTSTATUS register reflect the result of each > of statistical tests. > > Can you dump RTSTATUS to see which test has failed? Please dump RTMCTL as > well. > > > > Have you tried to build this patch for armv8? For me it is failing. > > > > Regards > > Gaurav Jain > > > > > RETRY_COUNT could be programmed in RTSCMISC register and is set to 1 > > > by default. Hence, we are left without hwrng after the first error, > > > which could happen even under normal conditions. > > > > > > Cc: petrben@gmail.com > > > Signed-off-by: Petr Benes <petr.benes@ysoft.com> > > > Signed-off-by: Michal Vokáč <michal.vokac@ysoft.com> > > > --- > > > Hi, > > > we are also experiencing this issue: > > > > > > caam_jr 2101000.jr0: 20003c5b: CCB: desc idx 60: RNG: Hardware error. > > > > > > It is happening on both i.MX6S and i.MX6DL SoCs we use. > > > On Solo I can reproduce it really fast. Sometimes it happens right > > > after the board is NFS booted, sometimes I need to stress the HWRNG > > > for a while (generate few hundred KBs of random data). On some > > > DualLite SoCs it is happening at least once a day. > > > > > > We are using the v5.10 LTS branch but I can confirm that this is > > > happening on all kernels since v5.7 to the latest linux-next. > > > > > > We also tried to increase the RTSDCTL_ENT_DLY_MIN delay as suggested > > > in this thread [1]. It helped and the issue never occurred since > > > then but we are looking for more universal and permanent solution > > > suitable for upstream, hence we came up with this patch. > > > > > > Any comments will be appreciated. > > > Thanks, Michal > > > > > > [1] > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flk > > > > ml.org%2F&data=04%7C01%7Cgaurav.jain%40nxp.com%7Cbda11e9702e1 > 428 > > > > 1fc3608d9d9c2e529%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6 > 3778 > > > > 0253361778015%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ > IjoiV2 > > > > luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Pe0skMXk%2F > epS > > > AnqjZRZ6tLKqO4SKbDvX3ysA0w7W6y4%3D&reserved=0 > > > %2Flkml%2F2021%2F8%2F30%2F296&data=04%7C01%7Cpankaj.gupta > %40 > > > > nxp.com%7C5d6bf460b260415aeda008d9a5f0ffab%7C686ea1d3bc2b4c6fa92cd > > > > 99c5c301635%7C0%7C0%7C637723276775699794%7CUnknown%7CTWFpbGZs > > > > b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0 > > > %3D%7C1000&sdata=VxN9MqDFbISptW3OX9XTtZ%2FwQTsEbF6dETxX > B%2 > > > BHrywg%3D&reserved=0 > > > > > > drivers/crypto/caam/caamrng.c | 42 > ++++++++++++++++++++++++++++++++--- > > > drivers/crypto/caam/ctrl.c | 13 +++++++++++ > > > drivers/crypto/caam/ctrl.h | 2 ++ > > > 3 files changed, 54 insertions(+), 3 deletions(-) > > > > > > diff --git a/drivers/crypto/caam/caamrng.c > > > b/drivers/crypto/caam/caamrng.c index 77d048dfe5d0..2be5584ae591 > > > 100644 > > > --- a/drivers/crypto/caam/caamrng.c > > > +++ b/drivers/crypto/caam/caamrng.c > > > @@ -21,6 +21,7 @@ > > > #include "desc_constr.h" > > > #include "jr.h" > > > #include "error.h" > > > +#include "ctrl.h" > > > > > > #define CAAM_RNG_MAX_FIFO_STORE_SIZE 16 > > > > > > @@ -113,6 +114,35 @@ static int caam_rng_read_one(struct device *jrdev, > > > return err ?: (ret ?: len); > > > } > > > > > > +static void caam_rng_retry_reset(struct caam_rng_ctx *context) { > > > + struct device *ctrldev = context->ctrldev; > > > + struct caam_drv_private *ctrlpriv = dev_get_drvdata(ctrldev); > > > + struct caam_ctrl __iomem *ctrl; > > > + struct rng4tst __iomem *r4tst; > > > + u32 __iomem *rtstatus; > > > + u32 retry_count; > > > + > > > + ctrl = (struct caam_ctrl __iomem *)ctrlpriv->ctrl; > > > + r4tst = &ctrl->r4tst[0]; > > > + > > > + /* > > > + * There is unfortunately no member for RTSTATUS register in > > > + * struct rng4tst and the structure doesn't look stable > > > + */ > > > + rtstatus = (u32 *)((char *)&ctrl->r4tst[0] + 0x3C); > > > + retry_count = (rd_reg32(rtstatus) >> 16) & 0xf; > > > + dev_dbg(ctrldev, "CAAM RNG retry count %d\n", retry_count); > > > + if (retry_count == 0) { > > > + dev_err(ctrldev, "CAAM RNG resetting retry count to 1\n"); > > > + clrsetbits_32(&r4tst->rtmctl, 0, RTMCTL_PRGM | RTMCTL_ACC); > > > + wr_reg32(&r4tst->rtscmisc, > > > + (rd_reg32(&r4tst->rtscmisc) & 0x7f) | (1 << > > > 16)); > > > + clrsetbits_32(&r4tst->rtmctl, RTMCTL_PRGM | RTMCTL_ACC, > > > + RTMCTL_SAMP_MODE_RAW_ES_SC); > > > + caam_reinstantiate_rng(ctrldev); > > > + } > > > +} > > > + > > > static void caam_rng_fill_async(struct caam_rng_ctx *ctx) { > > > struct scatterlist sg[1]; > > > @@ -129,8 +159,10 @@ static void caam_rng_fill_async(struct > > > caam_rng_ctx > > > *ctx) > > > sg[0].length, > > > ctx->desc_async, > > > &done); > > > - if (len < 0) > > > + if (len < 0) { > > > + caam_rng_retry_reset(ctx); > > > return; > > > + } > > > > > > kfifo_dma_in_finish(&ctx->fifo, len); } @@ -145,13 +177,17 > > > @@ static void caam_rng_worker(struct work_struct *work) static int > > > caam_read(struct hwrng *rng, void *dst, size_t max, bool wait) { > > > struct caam_rng_ctx *ctx = to_caam_rng_ctx(rng); > > > - int out; > > > + int out, ret; > > > > > > if (wait) { > > > struct completion done; > > > > > > - return caam_rng_read_one(ctx->jrdev, dst, max, > > > + ret = caam_rng_read_one(ctx->jrdev, dst, max, > > > ctx->desc_sync, &done); > > > + if (ret < 0) > > > + caam_rng_retry_reset(ctx); > > > + > > > + return ret; > > > } > > > > > > out = kfifo_out(&ctx->fifo, dst, max); diff --git > > > a/drivers/crypto/caam/ctrl.c b/drivers/crypto/caam/ctrl.c index > > > ca0361b2dbb0..e421f8d1982b 100644 > > > --- a/drivers/crypto/caam/ctrl.c > > > +++ b/drivers/crypto/caam/ctrl.c > > > @@ -339,6 +339,19 @@ static int instantiate_rng(struct device > > > *ctrldev, int state_handle_mask, > > > return devm_add_action_or_reset(ctrldev, > > > devm_deinstantiate_rng, ctrldev); } > > > > > > +/* > > > + * caam_reinstantiate_rng - reinstantiates RNG. Intended for a case > > > +when RNG > > > falls into > > > + * HW error condition. That happens if TRNG fails statistical > > > + * check and RTY_CNT value set in RTSCMISC decrements to > zero. > > > + * It is exported to caamrng.c > > > + * @ctrldev - pointer to device > > > + */ > > > + > > > +int caam_reinstantiate_rng(struct device *ctrldev) { > > > + return instantiate_rng(ctrldev, 0, 0); } > > > + > > > /* > > > * kick_trng - sets the various parameters for enabling the initialization > > > * of the RNG4 block in CAAM > > > diff --git a/drivers/crypto/caam/ctrl.h b/drivers/crypto/caam/ctrl.h > > > index > > > f3ecd67922a7..26ff5a49a865 100644 > > > --- a/drivers/crypto/caam/ctrl.h > > > +++ b/drivers/crypto/caam/ctrl.h > > > @@ -8,6 +8,8 @@ > > > #ifndef CTRL_H > > > #define CTRL_H > > > > > > +int caam_reinstantiate_rng(struct device *ctrldev); > > > + > > > /* Prototypes for backend-level services exposed to APIs */ extern > > > bool caam_dpaa2; > > > > > > -- > > > 2.25.1 > > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG failure 2022-01-13 7:23 ` [EXT] " Gaurav Jain 2022-01-17 14:08 ` Petr Benes @ 2022-01-20 9:59 ` Petr Benes 2022-01-24 7:04 ` Gaurav Jain 1 sibling, 1 reply; 10+ messages in thread From: Petr Benes @ 2022-01-20 9:59 UTC (permalink / raw) To: Gaurav Jain Cc: Horia Geanta, Pankaj Gupta, Herbert Xu, Michal Vokáč, David S. Miller, Petr Benes, l.stach, robert.hancock, linux-crypto, linux-kernel, Varun Sethi, Vabhav Sharma Hi Gaurav, I've tested 3 i.MX6DL devices over ~day and got these results (decimal values). RTMCTL: 12289 all cases RTSTATUS: device A 1 x 49152 10 x 32768 device B 13 x 32768 1 x 49152 1 x 1024 1 x 2048 device C 1 x 32768 Regards, Petr On Thu, 13 Jan 2022 at 08:23, Gaurav Jain <gaurav.jain@nxp.com> wrote: > > Hello Michal > > > > > -----Original Message----- > > From: Michal Vokáč <michal.vokac@ysoft.com> > > Sent: Friday, November 12, 2021 8:57 PM > > To: Horia Geanta <horia.geanta@nxp.com>; Pankaj Gupta > > <pankaj.gupta@nxp.com>; Herbert Xu <herbert@gondor.apana.org.au> > > Cc: David S. Miller <davem@davemloft.net>; linux-crypto@vger.kernel.org; > > linux-kernel@vger.kernel.org; l.stach@pengutronix.de; > > robert.hancock@calian.com; Petr Benes <petr.benes@ysoft.com>; > > petrben@gmail.com; Michal Vokáč <michal.vokac@ysoft.com> > > Subject: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG > > failure > > > > Caution: EXT Email > > > > From: Petr Benes <petr.benes@ysoft.com> > > > > Each time TRNG generates entropy, statistical tests are run. > > If they fail, RETRY_COUNT value is decremented. Once it reaches 0, HW RNG > > returns an error, and needs to be reset. > > The least-significant 16 bits of the RTSTATUS register reflect the result of each of statistical tests. > Can you dump RTSTATUS to see which test has failed? Please dump RTMCTL as well. > > Have you tried to build this patch for armv8? For me it is failing. > > Regards > Gaurav Jain > > > RETRY_COUNT could be programmed in RTSCMISC register and is set to 1 by > > default. Hence, we are left without hwrng after the first error, which could > > happen even under normal conditions. > > > > Cc: petrben@gmail.com > > Signed-off-by: Petr Benes <petr.benes@ysoft.com> > > Signed-off-by: Michal Vokáč <michal.vokac@ysoft.com> > > --- > > Hi, > > we are also experiencing this issue: > > > > caam_jr 2101000.jr0: 20003c5b: CCB: desc idx 60: RNG: Hardware error. > > > > It is happening on both i.MX6S and i.MX6DL SoCs we use. > > On Solo I can reproduce it really fast. Sometimes it happens right after the board > > is NFS booted, sometimes I need to stress the HWRNG for a while (generate few > > hundred KBs of random data). On some DualLite SoCs it is happening at least > > once a day. > > > > We are using the v5.10 LTS branch but I can confirm that this is happening on all > > kernels since v5.7 to the latest linux-next. > > > > We also tried to increase the RTSDCTL_ENT_DLY_MIN delay as suggested in this > > thread [1]. It helped and the issue never occurred since then but we are looking > > for more universal and permanent solution suitable for upstream, hence we > > came up with this patch. > > > > Any comments will be appreciated. > > Thanks, Michal > > > > [1] > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flkml.org > > %2Flkml%2F2021%2F8%2F30%2F296&data=04%7C01%7Cpankaj.gupta%40 > > nxp.com%7C5d6bf460b260415aeda008d9a5f0ffab%7C686ea1d3bc2b4c6fa92cd > > 99c5c301635%7C0%7C0%7C637723276775699794%7CUnknown%7CTWFpbGZs > > b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0 > > %3D%7C1000&sdata=VxN9MqDFbISptW3OX9XTtZ%2FwQTsEbF6dETxXB%2 > > BHrywg%3D&reserved=0 > > > > drivers/crypto/caam/caamrng.c | 42 ++++++++++++++++++++++++++++++++--- > > drivers/crypto/caam/ctrl.c | 13 +++++++++++ > > drivers/crypto/caam/ctrl.h | 2 ++ > > 3 files changed, 54 insertions(+), 3 deletions(-) > > > > diff --git a/drivers/crypto/caam/caamrng.c b/drivers/crypto/caam/caamrng.c > > index 77d048dfe5d0..2be5584ae591 100644 > > --- a/drivers/crypto/caam/caamrng.c > > +++ b/drivers/crypto/caam/caamrng.c > > @@ -21,6 +21,7 @@ > > #include "desc_constr.h" > > #include "jr.h" > > #include "error.h" > > +#include "ctrl.h" > > > > #define CAAM_RNG_MAX_FIFO_STORE_SIZE 16 > > > > @@ -113,6 +114,35 @@ static int caam_rng_read_one(struct device *jrdev, > > return err ?: (ret ?: len); > > } > > > > +static void caam_rng_retry_reset(struct caam_rng_ctx *context) { > > + struct device *ctrldev = context->ctrldev; > > + struct caam_drv_private *ctrlpriv = dev_get_drvdata(ctrldev); > > + struct caam_ctrl __iomem *ctrl; > > + struct rng4tst __iomem *r4tst; > > + u32 __iomem *rtstatus; > > + u32 retry_count; > > + > > + ctrl = (struct caam_ctrl __iomem *)ctrlpriv->ctrl; > > + r4tst = &ctrl->r4tst[0]; > > + > > + /* > > + * There is unfortunately no member for RTSTATUS register in > > + * struct rng4tst and the structure doesn't look stable > > + */ > > + rtstatus = (u32 *)((char *)&ctrl->r4tst[0] + 0x3C); > > + retry_count = (rd_reg32(rtstatus) >> 16) & 0xf; > > + dev_dbg(ctrldev, "CAAM RNG retry count %d\n", retry_count); > > + if (retry_count == 0) { > > + dev_err(ctrldev, "CAAM RNG resetting retry count to 1\n"); > > + clrsetbits_32(&r4tst->rtmctl, 0, RTMCTL_PRGM | RTMCTL_ACC); > > + wr_reg32(&r4tst->rtscmisc, (rd_reg32(&r4tst->rtscmisc) & 0x7f) | (1 << > > 16)); > > + clrsetbits_32(&r4tst->rtmctl, RTMCTL_PRGM | RTMCTL_ACC, > > + RTMCTL_SAMP_MODE_RAW_ES_SC); > > + caam_reinstantiate_rng(ctrldev); > > + } > > +} > > + > > static void caam_rng_fill_async(struct caam_rng_ctx *ctx) { > > struct scatterlist sg[1]; > > @@ -129,8 +159,10 @@ static void caam_rng_fill_async(struct caam_rng_ctx > > *ctx) > > sg[0].length, > > ctx->desc_async, > > &done); > > - if (len < 0) > > + if (len < 0) { > > + caam_rng_retry_reset(ctx); > > return; > > + } > > > > kfifo_dma_in_finish(&ctx->fifo, len); } @@ -145,13 +177,17 @@ static void > > caam_rng_worker(struct work_struct *work) static int caam_read(struct hwrng > > *rng, void *dst, size_t max, bool wait) { > > struct caam_rng_ctx *ctx = to_caam_rng_ctx(rng); > > - int out; > > + int out, ret; > > > > if (wait) { > > struct completion done; > > > > - return caam_rng_read_one(ctx->jrdev, dst, max, > > + ret = caam_rng_read_one(ctx->jrdev, dst, max, > > ctx->desc_sync, &done); > > + if (ret < 0) > > + caam_rng_retry_reset(ctx); > > + > > + return ret; > > } > > > > out = kfifo_out(&ctx->fifo, dst, max); diff --git a/drivers/crypto/caam/ctrl.c > > b/drivers/crypto/caam/ctrl.c index ca0361b2dbb0..e421f8d1982b 100644 > > --- a/drivers/crypto/caam/ctrl.c > > +++ b/drivers/crypto/caam/ctrl.c > > @@ -339,6 +339,19 @@ static int instantiate_rng(struct device *ctrldev, int > > state_handle_mask, > > return devm_add_action_or_reset(ctrldev, devm_deinstantiate_rng, > > ctrldev); } > > > > +/* > > + * caam_reinstantiate_rng - reinstantiates RNG. Intended for a case when RNG > > falls into > > + * HW error condition. That happens if TRNG fails statistical > > + * check and RTY_CNT value set in RTSCMISC decrements to zero. > > + * It is exported to caamrng.c > > + * @ctrldev - pointer to device > > + */ > > + > > +int caam_reinstantiate_rng(struct device *ctrldev) { > > + return instantiate_rng(ctrldev, 0, 0); } > > + > > /* > > * kick_trng - sets the various parameters for enabling the initialization > > * of the RNG4 block in CAAM > > diff --git a/drivers/crypto/caam/ctrl.h b/drivers/crypto/caam/ctrl.h index > > f3ecd67922a7..26ff5a49a865 100644 > > --- a/drivers/crypto/caam/ctrl.h > > +++ b/drivers/crypto/caam/ctrl.h > > @@ -8,6 +8,8 @@ > > #ifndef CTRL_H > > #define CTRL_H > > > > +int caam_reinstantiate_rng(struct device *ctrldev); > > + > > /* Prototypes for backend-level services exposed to APIs */ extern bool > > caam_dpaa2; > > > > -- > > 2.25.1 > ^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG failure 2022-01-20 9:59 ` Petr Benes @ 2022-01-24 7:04 ` Gaurav Jain 2022-01-24 12:25 ` Petr Benes 0 siblings, 1 reply; 10+ messages in thread From: Gaurav Jain @ 2022-01-24 7:04 UTC (permalink / raw) To: Petr Benes Cc: Horia Geanta, Pankaj Gupta, Herbert Xu, Michal Vokáč, David S. Miller, Petr Benes, l.stach, robert.hancock, linux-crypto, linux-kernel, Varun Sethi, Vabhav Sharma Hello Petr > -----Original Message----- > From: Petr Benes <petrben@gmail.com> > Sent: Thursday, January 20, 2022 3:29 PM > To: Gaurav Jain <gaurav.jain@nxp.com> > Cc: Horia Geanta <horia.geanta@nxp.com>; Pankaj Gupta > <pankaj.gupta@nxp.com>; Herbert Xu <herbert@gondor.apana.org.au>; Michal > Vokáč <michal.vokac@ysoft.com>; David S. Miller <davem@davemloft.net>; > Petr Benes <petr.benes@ysoft.com>; l.stach@pengutronix.de; > robert.hancock@calian.com; linux-crypto@vger.kernel.org; linux- > kernel@vger.kernel.org; Varun Sethi <V.Sethi@nxp.com>; Vabhav Sharma > <vabhav.sharma@nxp.com> > Subject: Re: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG > failure > > Caution: EXT Email > > Hi Gaurav, > > I've tested 3 i.MX6DL devices over ~day and got these results (decimal values). > > RTMCTL: 12289 all cases > > RTSTATUS: > > device A > 1 x 49152 > 10 x 32768 > > device B > 13 x 32768 > 1 x 49152 > 1 x 1024 > 1 x 2048 > > device C > 1 x 32768 > Thank you Petr for the register dump. Looks like statistical tests are failing. We are internally looking into TRNG configuration which could cause statistical test failure. Can you confirm if you have reproduced the problem with the patches suggested in Uboot? Link for the same is http://patchwork.ozlabs.org/project/uboot/list/?series=280725 With this patch series TRNG is configured in uboot. Regards Gaurav Jain > Regards, > Petr > > On Thu, 13 Jan 2022 at 08:23, Gaurav Jain <gaurav.jain@nxp.com> wrote: > > > > Hello Michal > > > > > > > > -----Original Message----- > > > From: Michal Vokáč <michal.vokac@ysoft.com> > > > Sent: Friday, November 12, 2021 8:57 PM > > > To: Horia Geanta <horia.geanta@nxp.com>; Pankaj Gupta > > > <pankaj.gupta@nxp.com>; Herbert Xu <herbert@gondor.apana.org.au> > > > Cc: David S. Miller <davem@davemloft.net>; > > > linux-crypto@vger.kernel.org; linux-kernel@vger.kernel.org; > > > l.stach@pengutronix.de; robert.hancock@calian.com; Petr Benes > > > <petr.benes@ysoft.com>; petrben@gmail.com; Michal Vokáč > > > <michal.vokac@ysoft.com> > > > Subject: [EXT] [RFC PATCH] crypto: caam - restore retry count after > > > HW RNG failure > > > > > > Caution: EXT Email > > > > > > From: Petr Benes <petr.benes@ysoft.com> > > > > > > Each time TRNG generates entropy, statistical tests are run. > > > If they fail, RETRY_COUNT value is decremented. Once it reaches 0, > > > HW RNG returns an error, and needs to be reset. > > > > The least-significant 16 bits of the RTSTATUS register reflect the result of each > of statistical tests. > > Can you dump RTSTATUS to see which test has failed? Please dump RTMCTL as > well. > > > > Have you tried to build this patch for armv8? For me it is failing. > > > > Regards > > Gaurav Jain > > > > > RETRY_COUNT could be programmed in RTSCMISC register and is set to 1 > > > by default. Hence, we are left without hwrng after the first error, > > > which could happen even under normal conditions. > > > > > > Cc: petrben@gmail.com > > > Signed-off-by: Petr Benes <petr.benes@ysoft.com> > > > Signed-off-by: Michal Vokáč <michal.vokac@ysoft.com> > > > --- > > > Hi, > > > we are also experiencing this issue: > > > > > > caam_jr 2101000.jr0: 20003c5b: CCB: desc idx 60: RNG: Hardware error. > > > > > > It is happening on both i.MX6S and i.MX6DL SoCs we use. > > > On Solo I can reproduce it really fast. Sometimes it happens right > > > after the board is NFS booted, sometimes I need to stress the HWRNG > > > for a while (generate few hundred KBs of random data). On some > > > DualLite SoCs it is happening at least once a day. > > > > > > We are using the v5.10 LTS branch but I can confirm that this is > > > happening on all kernels since v5.7 to the latest linux-next. > > > > > > We also tried to increase the RTSDCTL_ENT_DLY_MIN delay as suggested > > > in this thread [1]. It helped and the issue never occurred since > > > then but we are looking for more universal and permanent solution > > > suitable for upstream, hence we came up with this patch. > > > > > > Any comments will be appreciated. > > > Thanks, Michal > > > > > > [1] > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flk > > > > ml.org%2F&data=04%7C01%7Cgaurav.jain%40nxp.com%7Cad793bd423a6 > 4f2 > > > > b03d408d9dbfb842f%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6 > 3778 > > > > 2695582123236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ > IjoiV2 > > > > luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=PlPYDNHg9%2 > BCL > > > iZrcdmOx%2FPdbAZmFpPZFvLe1jR4YhEI%3D&reserved=0 > > > %2Flkml%2F2021%2F8%2F30%2F296&data=04%7C01%7Cpankaj.gupta > %40 > > > > nxp.com%7C5d6bf460b260415aeda008d9a5f0ffab%7C686ea1d3bc2b4c6fa92cd > > > > 99c5c301635%7C0%7C0%7C637723276775699794%7CUnknown%7CTWFpbGZs > > > > b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0 > > > %3D%7C1000&sdata=VxN9MqDFbISptW3OX9XTtZ%2FwQTsEbF6dETxX > B%2 > > > BHrywg%3D&reserved=0 > > > > > > drivers/crypto/caam/caamrng.c | 42 > ++++++++++++++++++++++++++++++++--- > > > drivers/crypto/caam/ctrl.c | 13 +++++++++++ > > > drivers/crypto/caam/ctrl.h | 2 ++ > > > 3 files changed, 54 insertions(+), 3 deletions(-) > > > > > > diff --git a/drivers/crypto/caam/caamrng.c > > > b/drivers/crypto/caam/caamrng.c index 77d048dfe5d0..2be5584ae591 > > > 100644 > > > --- a/drivers/crypto/caam/caamrng.c > > > +++ b/drivers/crypto/caam/caamrng.c > > > @@ -21,6 +21,7 @@ > > > #include "desc_constr.h" > > > #include "jr.h" > > > #include "error.h" > > > +#include "ctrl.h" > > > > > > #define CAAM_RNG_MAX_FIFO_STORE_SIZE 16 > > > > > > @@ -113,6 +114,35 @@ static int caam_rng_read_one(struct device *jrdev, > > > return err ?: (ret ?: len); > > > } > > > > > > +static void caam_rng_retry_reset(struct caam_rng_ctx *context) { > > > + struct device *ctrldev = context->ctrldev; > > > + struct caam_drv_private *ctrlpriv = dev_get_drvdata(ctrldev); > > > + struct caam_ctrl __iomem *ctrl; > > > + struct rng4tst __iomem *r4tst; > > > + u32 __iomem *rtstatus; > > > + u32 retry_count; > > > + > > > + ctrl = (struct caam_ctrl __iomem *)ctrlpriv->ctrl; > > > + r4tst = &ctrl->r4tst[0]; > > > + > > > + /* > > > + * There is unfortunately no member for RTSTATUS register in > > > + * struct rng4tst and the structure doesn't look stable > > > + */ > > > + rtstatus = (u32 *)((char *)&ctrl->r4tst[0] + 0x3C); > > > + retry_count = (rd_reg32(rtstatus) >> 16) & 0xf; > > > + dev_dbg(ctrldev, "CAAM RNG retry count %d\n", retry_count); > > > + if (retry_count == 0) { > > > + dev_err(ctrldev, "CAAM RNG resetting retry count to 1\n"); > > > + clrsetbits_32(&r4tst->rtmctl, 0, RTMCTL_PRGM | RTMCTL_ACC); > > > + wr_reg32(&r4tst->rtscmisc, > > > + (rd_reg32(&r4tst->rtscmisc) & 0x7f) | (1 << > > > 16)); > > > + clrsetbits_32(&r4tst->rtmctl, RTMCTL_PRGM | RTMCTL_ACC, > > > + RTMCTL_SAMP_MODE_RAW_ES_SC); > > > + caam_reinstantiate_rng(ctrldev); > > > + } > > > +} > > > + > > > static void caam_rng_fill_async(struct caam_rng_ctx *ctx) { > > > struct scatterlist sg[1]; > > > @@ -129,8 +159,10 @@ static void caam_rng_fill_async(struct > > > caam_rng_ctx > > > *ctx) > > > sg[0].length, > > > ctx->desc_async, > > > &done); > > > - if (len < 0) > > > + if (len < 0) { > > > + caam_rng_retry_reset(ctx); > > > return; > > > + } > > > > > > kfifo_dma_in_finish(&ctx->fifo, len); } @@ -145,13 +177,17 > > > @@ static void caam_rng_worker(struct work_struct *work) static int > > > caam_read(struct hwrng *rng, void *dst, size_t max, bool wait) { > > > struct caam_rng_ctx *ctx = to_caam_rng_ctx(rng); > > > - int out; > > > + int out, ret; > > > > > > if (wait) { > > > struct completion done; > > > > > > - return caam_rng_read_one(ctx->jrdev, dst, max, > > > + ret = caam_rng_read_one(ctx->jrdev, dst, max, > > > ctx->desc_sync, &done); > > > + if (ret < 0) > > > + caam_rng_retry_reset(ctx); > > > + > > > + return ret; > > > } > > > > > > out = kfifo_out(&ctx->fifo, dst, max); diff --git > > > a/drivers/crypto/caam/ctrl.c b/drivers/crypto/caam/ctrl.c index > > > ca0361b2dbb0..e421f8d1982b 100644 > > > --- a/drivers/crypto/caam/ctrl.c > > > +++ b/drivers/crypto/caam/ctrl.c > > > @@ -339,6 +339,19 @@ static int instantiate_rng(struct device > > > *ctrldev, int state_handle_mask, > > > return devm_add_action_or_reset(ctrldev, > > > devm_deinstantiate_rng, ctrldev); } > > > > > > +/* > > > + * caam_reinstantiate_rng - reinstantiates RNG. Intended for a case > > > +when RNG > > > falls into > > > + * HW error condition. That happens if TRNG fails statistical > > > + * check and RTY_CNT value set in RTSCMISC decrements to > zero. > > > + * It is exported to caamrng.c > > > + * @ctrldev - pointer to device > > > + */ > > > + > > > +int caam_reinstantiate_rng(struct device *ctrldev) { > > > + return instantiate_rng(ctrldev, 0, 0); } > > > + > > > /* > > > * kick_trng - sets the various parameters for enabling the initialization > > > * of the RNG4 block in CAAM > > > diff --git a/drivers/crypto/caam/ctrl.h b/drivers/crypto/caam/ctrl.h > > > index > > > f3ecd67922a7..26ff5a49a865 100644 > > > --- a/drivers/crypto/caam/ctrl.h > > > +++ b/drivers/crypto/caam/ctrl.h > > > @@ -8,6 +8,8 @@ > > > #ifndef CTRL_H > > > #define CTRL_H > > > > > > +int caam_reinstantiate_rng(struct device *ctrldev); > > > + > > > /* Prototypes for backend-level services exposed to APIs */ extern > > > bool caam_dpaa2; > > > > > > -- > > > 2.25.1 > > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG failure 2022-01-24 7:04 ` Gaurav Jain @ 2022-01-24 12:25 ` Petr Benes 0 siblings, 0 replies; 10+ messages in thread From: Petr Benes @ 2022-01-24 12:25 UTC (permalink / raw) To: Gaurav Jain Cc: Horia Geanta, Pankaj Gupta, Herbert Xu, Michal Vokáč, David S. Miller, Petr Benes, l.stach, robert.hancock, linux-crypto, linux-kernel, Varun Sethi, Vabhav Sharma Hi Gaurav, On Mon, 24 Jan 2022 at 08:04, Gaurav Jain <gaurav.jain@nxp.com> wrote: > > Hello Petr > > > -----Original Message----- > > From: Petr Benes <petrben@gmail.com> > > Sent: Thursday, January 20, 2022 3:29 PM > > To: Gaurav Jain <gaurav.jain@nxp.com> > > Cc: Horia Geanta <horia.geanta@nxp.com>; Pankaj Gupta > > <pankaj.gupta@nxp.com>; Herbert Xu <herbert@gondor.apana.org.au>; Michal > > Vokáč <michal.vokac@ysoft.com>; David S. Miller <davem@davemloft.net>; > > Petr Benes <petr.benes@ysoft.com>; l.stach@pengutronix.de; > > robert.hancock@calian.com; linux-crypto@vger.kernel.org; linux- > > kernel@vger.kernel.org; Varun Sethi <V.Sethi@nxp.com>; Vabhav Sharma > > <vabhav.sharma@nxp.com> > > Subject: Re: [EXT] [RFC PATCH] crypto: caam - restore retry count after HW RNG > > failure > > > > Caution: EXT Email > > > > Hi Gaurav, > > > > I've tested 3 i.MX6DL devices over ~day and got these results (decimal values). > > > > RTMCTL: 12289 all cases > > > > RTSTATUS: > > > > device A > > 1 x 49152 > > 10 x 32768 > > > > device B > > 13 x 32768 > > 1 x 49152 > > 1 x 1024 > > 1 x 2048 > > > > device C > > 1 x 32768 > > > > Thank you Petr for the register dump. Looks like statistical tests are failing. > We are internally looking into TRNG configuration which could cause statistical test failure. > > Can you confirm if you have reproduced the problem with the patches suggested in Uboot? > Link for the same is http://patchwork.ozlabs.org/project/uboot/list/?series=280725 > With this patch series TRNG is configured in uboot. Unfortunately, we are behind with U-Boot (and do not setup CAAM components there anyway), it would need some planning on my side. I'm not sure I can secure enough time for it. So, cannot provide the confirmation at this moment nor any time soon . If it (U-Boot patch) solves the problem, do you plan to provide a solely Linux solution? If I got it correctly, there is a small chance to hit this issue under normal conditions, some retry mechanism is desired. Regards, Petr > > Regards > Gaurav Jain > > > Regards, > > Petr > > > > On Thu, 13 Jan 2022 at 08:23, Gaurav Jain <gaurav.jain@nxp.com> wrote: > > > > > > Hello Michal > > > > > > > > > > > -----Original Message----- > > > > From: Michal Vokáč <michal.vokac@ysoft.com> > > > > Sent: Friday, November 12, 2021 8:57 PM > > > > To: Horia Geanta <horia.geanta@nxp.com>; Pankaj Gupta > > > > <pankaj.gupta@nxp.com>; Herbert Xu <herbert@gondor.apana.org.au> > > > > Cc: David S. Miller <davem@davemloft.net>; > > > > linux-crypto@vger.kernel.org; linux-kernel@vger.kernel.org; > > > > l.stach@pengutronix.de; robert.hancock@calian.com; Petr Benes > > > > <petr.benes@ysoft.com>; petrben@gmail.com; Michal Vokáč > > > > <michal.vokac@ysoft.com> > > > > Subject: [EXT] [RFC PATCH] crypto: caam - restore retry count after > > > > HW RNG failure > > > > > > > > Caution: EXT Email > > > > > > > > From: Petr Benes <petr.benes@ysoft.com> > > > > > > > > Each time TRNG generates entropy, statistical tests are run. > > > > If they fail, RETRY_COUNT value is decremented. Once it reaches 0, > > > > HW RNG returns an error, and needs to be reset. > > > > > > The least-significant 16 bits of the RTSTATUS register reflect the result of each > > of statistical tests. > > > Can you dump RTSTATUS to see which test has failed? Please dump RTMCTL as > > well. > > > > > > Have you tried to build this patch for armv8? For me it is failing. > > > > > > Regards > > > Gaurav Jain > > > > > > > RETRY_COUNT could be programmed in RTSCMISC register and is set to 1 > > > > by default. Hence, we are left without hwrng after the first error, > > > > which could happen even under normal conditions. > > > > > > > > Cc: petrben@gmail.com > > > > Signed-off-by: Petr Benes <petr.benes@ysoft.com> > > > > Signed-off-by: Michal Vokáč <michal.vokac@ysoft.com> > > > > --- > > > > Hi, > > > > we are also experiencing this issue: > > > > > > > > caam_jr 2101000.jr0: 20003c5b: CCB: desc idx 60: RNG: Hardware error. > > > > > > > > It is happening on both i.MX6S and i.MX6DL SoCs we use. > > > > On Solo I can reproduce it really fast. Sometimes it happens right > > > > after the board is NFS booted, sometimes I need to stress the HWRNG > > > > for a while (generate few hundred KBs of random data). On some > > > > DualLite SoCs it is happening at least once a day. > > > > > > > > We are using the v5.10 LTS branch but I can confirm that this is > > > > happening on all kernels since v5.7 to the latest linux-next. > > > > > > > > We also tried to increase the RTSDCTL_ENT_DLY_MIN delay as suggested > > > > in this thread [1]. It helped and the issue never occurred since > > > > then but we are looking for more universal and permanent solution > > > > suitable for upstream, hence we came up with this patch. > > > > > > > > Any comments will be appreciated. > > > > Thanks, Michal > > > > > > > > [1] > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flk > > > > > > ml.org%2F&data=04%7C01%7Cgaurav.jain%40nxp.com%7Cad793bd423a6 > > 4f2 > > > > > > b03d408d9dbfb842f%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6 > > 3778 > > > > > > 2695582123236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ > > IjoiV2 > > > > > > luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=PlPYDNHg9%2 > > BCL > > > > iZrcdmOx%2FPdbAZmFpPZFvLe1jR4YhEI%3D&reserved=0 > > > > %2Flkml%2F2021%2F8%2F30%2F296&data=04%7C01%7Cpankaj.gupta > > %40 > > > > > > nxp.com%7C5d6bf460b260415aeda008d9a5f0ffab%7C686ea1d3bc2b4c6fa92cd > > > > > > 99c5c301635%7C0%7C0%7C637723276775699794%7CUnknown%7CTWFpbGZs > > > > > > b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0 > > > > %3D%7C1000&sdata=VxN9MqDFbISptW3OX9XTtZ%2FwQTsEbF6dETxX > > B%2 > > > > BHrywg%3D&reserved=0 > > > > > > > > drivers/crypto/caam/caamrng.c | 42 > > ++++++++++++++++++++++++++++++++--- > > > > drivers/crypto/caam/ctrl.c | 13 +++++++++++ > > > > drivers/crypto/caam/ctrl.h | 2 ++ > > > > 3 files changed, 54 insertions(+), 3 deletions(-) > > > > > > > > diff --git a/drivers/crypto/caam/caamrng.c > > > > b/drivers/crypto/caam/caamrng.c index 77d048dfe5d0..2be5584ae591 > > > > 100644 > > > > --- a/drivers/crypto/caam/caamrng.c > > > > +++ b/drivers/crypto/caam/caamrng.c > > > > @@ -21,6 +21,7 @@ > > > > #include "desc_constr.h" > > > > #include "jr.h" > > > > #include "error.h" > > > > +#include "ctrl.h" > > > > > > > > #define CAAM_RNG_MAX_FIFO_STORE_SIZE 16 > > > > > > > > @@ -113,6 +114,35 @@ static int caam_rng_read_one(struct device *jrdev, > > > > return err ?: (ret ?: len); > > > > } > > > > > > > > +static void caam_rng_retry_reset(struct caam_rng_ctx *context) { > > > > + struct device *ctrldev = context->ctrldev; > > > > + struct caam_drv_private *ctrlpriv = dev_get_drvdata(ctrldev); > > > > + struct caam_ctrl __iomem *ctrl; > > > > + struct rng4tst __iomem *r4tst; > > > > + u32 __iomem *rtstatus; > > > > + u32 retry_count; > > > > + > > > > + ctrl = (struct caam_ctrl __iomem *)ctrlpriv->ctrl; > > > > + r4tst = &ctrl->r4tst[0]; > > > > + > > > > + /* > > > > + * There is unfortunately no member for RTSTATUS register in > > > > + * struct rng4tst and the structure doesn't look stable > > > > + */ > > > > + rtstatus = (u32 *)((char *)&ctrl->r4tst[0] + 0x3C); > > > > + retry_count = (rd_reg32(rtstatus) >> 16) & 0xf; > > > > + dev_dbg(ctrldev, "CAAM RNG retry count %d\n", retry_count); > > > > + if (retry_count == 0) { > > > > + dev_err(ctrldev, "CAAM RNG resetting retry count to 1\n"); > > > > + clrsetbits_32(&r4tst->rtmctl, 0, RTMCTL_PRGM | RTMCTL_ACC); > > > > + wr_reg32(&r4tst->rtscmisc, > > > > + (rd_reg32(&r4tst->rtscmisc) & 0x7f) | (1 << > > > > 16)); > > > > + clrsetbits_32(&r4tst->rtmctl, RTMCTL_PRGM | RTMCTL_ACC, > > > > + RTMCTL_SAMP_MODE_RAW_ES_SC); > > > > + caam_reinstantiate_rng(ctrldev); > > > > + } > > > > +} > > > > + > > > > static void caam_rng_fill_async(struct caam_rng_ctx *ctx) { > > > > struct scatterlist sg[1]; > > > > @@ -129,8 +159,10 @@ static void caam_rng_fill_async(struct > > > > caam_rng_ctx > > > > *ctx) > > > > sg[0].length, > > > > ctx->desc_async, > > > > &done); > > > > - if (len < 0) > > > > + if (len < 0) { > > > > + caam_rng_retry_reset(ctx); > > > > return; > > > > + } > > > > > > > > kfifo_dma_in_finish(&ctx->fifo, len); } @@ -145,13 +177,17 > > > > @@ static void caam_rng_worker(struct work_struct *work) static int > > > > caam_read(struct hwrng *rng, void *dst, size_t max, bool wait) { > > > > struct caam_rng_ctx *ctx = to_caam_rng_ctx(rng); > > > > - int out; > > > > + int out, ret; > > > > > > > > if (wait) { > > > > struct completion done; > > > > > > > > - return caam_rng_read_one(ctx->jrdev, dst, max, > > > > + ret = caam_rng_read_one(ctx->jrdev, dst, max, > > > > ctx->desc_sync, &done); > > > > + if (ret < 0) > > > > + caam_rng_retry_reset(ctx); > > > > + > > > > + return ret; > > > > } > > > > > > > > out = kfifo_out(&ctx->fifo, dst, max); diff --git > > > > a/drivers/crypto/caam/ctrl.c b/drivers/crypto/caam/ctrl.c index > > > > ca0361b2dbb0..e421f8d1982b 100644 > > > > --- a/drivers/crypto/caam/ctrl.c > > > > +++ b/drivers/crypto/caam/ctrl.c > > > > @@ -339,6 +339,19 @@ static int instantiate_rng(struct device > > > > *ctrldev, int state_handle_mask, > > > > return devm_add_action_or_reset(ctrldev, > > > > devm_deinstantiate_rng, ctrldev); } > > > > > > > > +/* > > > > + * caam_reinstantiate_rng - reinstantiates RNG. Intended for a case > > > > +when RNG > > > > falls into > > > > + * HW error condition. That happens if TRNG fails statistical > > > > + * check and RTY_CNT value set in RTSCMISC decrements to > > zero. > > > > + * It is exported to caamrng.c > > > > + * @ctrldev - pointer to device > > > > + */ > > > > + > > > > +int caam_reinstantiate_rng(struct device *ctrldev) { > > > > + return instantiate_rng(ctrldev, 0, 0); } > > > > + > > > > /* > > > > * kick_trng - sets the various parameters for enabling the initialization > > > > * of the RNG4 block in CAAM > > > > diff --git a/drivers/crypto/caam/ctrl.h b/drivers/crypto/caam/ctrl.h > > > > index > > > > f3ecd67922a7..26ff5a49a865 100644 > > > > --- a/drivers/crypto/caam/ctrl.h > > > > +++ b/drivers/crypto/caam/ctrl.h > > > > @@ -8,6 +8,8 @@ > > > > #ifndef CTRL_H > > > > #define CTRL_H > > > > > > > > +int caam_reinstantiate_rng(struct device *ctrldev); > > > > + > > > > /* Prototypes for backend-level services exposed to APIs */ extern > > > > bool caam_dpaa2; > > > > > > > > -- > > > > 2.25.1 > > > ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2022-01-24 12:25 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-12-14 19:00 CAAM RNG trouble Lucas Stach
2021-03-02 17:33 ` Horia Geantă
2021-03-03 0:37 ` Robert Hancock
2021-11-12 15:27 ` [RFC PATCH] crypto: caam - restore retry count after HW RNG failure Michal Vokáč
[not found] ` <DU2PR04MB863088F218774551FA254EB795539@DU2PR04MB8630.eurprd04.prod.outlook.com>
2022-01-13 7:23 ` [EXT] " Gaurav Jain
2022-01-17 14:08 ` Petr Benes
2022-01-18 9:04 ` Gaurav Jain
2022-01-20 9:59 ` Petr Benes
2022-01-24 7:04 ` Gaurav Jain
2022-01-24 12:25 ` Petr Benes
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.