* [PATCH v7 1/1] lib: rsa: fix allocated size for rr and rrtmp in rsa_gen_key_prop()
@ 2020-07-07 20:57 Heinrich Schuchardt
2020-07-09 0:24 ` Tom Rini
0 siblings, 1 reply; 2+ messages in thread
From: Heinrich Schuchardt @ 2020-07-07 20:57 UTC (permalink / raw)
To: u-boot
From: Heiko Stuebner <heiko.stuebner@theobroma-systems.com>
When calculating rrtmp/rr rsa_gen_key_prop() tries to make
(((rlen + 31) >> 5) + 1) steps in the rr uint32_t array and
(((rlen + 7) >> 3) + 1) / 4 steps in uint32_t rrtmp[]
with rlen being num_bits * 2
On a 4096bit key this comes down to to 257 uint32_t elements
in rr and 256 elements in rrtmp but with the current allocation
rr and rrtmp only have 129 uint32_t elements.
On 2048bit keys this works by chance as the defined max_rsa_size=4096
allocates a suitable number of elements, but with an actual 4096bit key
this results in other memory parts getting overwritten.
So as suggested by Heinrich Schuchardt just use the actual bit-size
of the key as base for the size calculation, in turn making the code
compatible to any future keysizes.
Suggested-by: Heinrich Schuchardt <xypron.debian@gmx.de>
Signed-off-by: Heiko Stuebner <heiko.stuebner@theobroma-systems.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
rrtmp needs 2 + (((*prop)->num_bits * 2) >> 5) array elements.
Reviewed-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
---
v7:
rrtmp needs one more u32.
There is no label out. It is called err.
---
lib/rsa/rsa-keyprop.c | 14 +++++++++-----
1 file changed, 9 insertions(+), 5 deletions(-)
diff --git a/lib/rsa/rsa-keyprop.c b/lib/rsa/rsa-keyprop.c
index 9464df0093..e991031ec0 100644
--- a/lib/rsa/rsa-keyprop.c
+++ b/lib/rsa/rsa-keyprop.c
@@ -654,14 +654,10 @@ int rsa_gen_key_prop(const void *key, uint32_t keylen, struct key_prop **prop)
{
struct rsa_key rsa_key;
uint32_t *n = NULL, *rr = NULL, *rrtmp = NULL;
- const int max_rsa_size = 4096;
int rlen, i, ret;
*prop = calloc(sizeof(**prop), 1);
- n = calloc(sizeof(uint32_t), 1 + (max_rsa_size >> 5));
- rr = calloc(sizeof(uint32_t), 1 + (max_rsa_size >> 5));
- rrtmp = calloc(sizeof(uint32_t), 1 + (max_rsa_size >> 5));
- if (!(*prop) || !n || !rr || !rrtmp) {
+ if (!(*prop)) {
ret = -ENOMEM;
goto err;
}
@@ -682,6 +678,14 @@ int rsa_gen_key_prop(const void *key, uint32_t keylen, struct key_prop **prop)
}
memcpy((void *)(*prop)->modulus, &rsa_key.n[i], rsa_key.n_sz - i);
+ n = calloc(sizeof(uint32_t), 1 + ((*prop)->num_bits >> 5));
+ rr = calloc(sizeof(uint32_t), 1 + (((*prop)->num_bits * 2) >> 5));
+ rrtmp = calloc(sizeof(uint32_t), 2 + (((*prop)->num_bits * 2) >> 5));
+ if (!n || !rr || !rrtmp) {
+ ret = -ENOMEM;
+ goto err;
+ }
+
/* exponent */
(*prop)->public_exponent = calloc(1, sizeof(uint64_t));
if (!(*prop)->public_exponent) {
--
2.27.0
^ permalink raw reply related [flat|nested] 2+ messages in thread
* [PATCH v7 1/1] lib: rsa: fix allocated size for rr and rrtmp in rsa_gen_key_prop()
2020-07-07 20:57 [PATCH v7 1/1] lib: rsa: fix allocated size for rr and rrtmp in rsa_gen_key_prop() Heinrich Schuchardt
@ 2020-07-09 0:24 ` Tom Rini
0 siblings, 0 replies; 2+ messages in thread
From: Tom Rini @ 2020-07-09 0:24 UTC (permalink / raw)
To: u-boot
On Tue, Jul 07, 2020 at 10:57:26PM +0200, Heinrich Schuchardt wrote:
> From: Heiko Stuebner <heiko.stuebner@theobroma-systems.com>
>
> When calculating rrtmp/rr rsa_gen_key_prop() tries to make
> (((rlen + 31) >> 5) + 1) steps in the rr uint32_t array and
> (((rlen + 7) >> 3) + 1) / 4 steps in uint32_t rrtmp[]
> with rlen being num_bits * 2
>
> On a 4096bit key this comes down to to 257 uint32_t elements
> in rr and 256 elements in rrtmp but with the current allocation
> rr and rrtmp only have 129 uint32_t elements.
>
> On 2048bit keys this works by chance as the defined max_rsa_size=4096
> allocates a suitable number of elements, but with an actual 4096bit key
> this results in other memory parts getting overwritten.
>
> So as suggested by Heinrich Schuchardt just use the actual bit-size
> of the key as base for the size calculation, in turn making the code
> compatible to any future keysizes.
>
> Suggested-by: Heinrich Schuchardt <xypron.debian@gmx.de>
> Signed-off-by: Heiko Stuebner <heiko.stuebner@theobroma-systems.com>
> Reviewed-by: Simon Glass <sjg@chromium.org>
>
> rrtmp needs 2 + (((*prop)->num_bits * 2) >> 5) array elements.
>
> Reviewed-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
Applied to u-boot/master, thanks!
--
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200708/441a74bd/attachment.sig>
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2020-07-09 0:24 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-07 20:57 [PATCH v7 1/1] lib: rsa: fix allocated size for rr and rrtmp in rsa_gen_key_prop() Heinrich Schuchardt
2020-07-09 0:24 ` Tom Rini
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.