From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-lf1-f43.google.com (mail-lf1-f43.google.com [209.85.167.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3BCAF168 for ; Mon, 24 Jan 2022 20:00:03 +0000 (UTC) Received: by mail-lf1-f43.google.com with SMTP id x11so52280310lfa.2 for ; Mon, 24 Jan 2022 12:00:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=X1fspDfOZTRESHgvCsTpy0M+wXpoDo9iEK0jnJXDU5Y=; b=liPSw+oHKbLKt49NG7WwlravT+ecqQGWP3Y1Q8azHjQmV39vlahZekjyxHrUorHG/s lgHi+ZN/pKPrXARZMuuyXAiagpmQ754zvGdFo3GwVY1mEUx8JkonDBNdCBKtOm/CsO0H l57Gfv77T9qgQK8hck4Gu9uzAdq07iWfS/Nn0d18+937IolDJ4rDoR6RYdCIOAMkGGiT o86l+PSF7Bih3oEb7/WysNBmRTnzCpUJ46UIpdYMj/oPP3A3W2RSItFoPKkWjpDdyRT2 pVIUwn2blEfUxS8k8l1bWErFN2xPD3V9KfxdKz0wOpdP13mwNeNWhey02A8C1wkng6w0 P6Ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=X1fspDfOZTRESHgvCsTpy0M+wXpoDo9iEK0jnJXDU5Y=; b=UcaHXt5GiE9oLwifnZGAK+taLPaC/a+y14aUBqiyc6bhYg/D7zcnc8klBp6QwmA9tc VdUnm+vR16CLexDcCQPVqKBof11/HQZ3Qn60R+gS9V4+kVzhIJ4AYY+HFMaq0L2C45FM EmiqrGkiX6hco3N1AX0OA/YBMSHTqbZoBZiBu3znVJdCtgkVkaBDv2/zYIo9O/yFYDV+ TUhQZmGZsJaY5kfbE7Xm9MZBJy8JJYrIk0MVSJNi9MIrLrZGuZEjBH/gtxwMqyHzd2wW U6kbs2+1kkISOG+7ytLiW3Ze79m+mWYSSJud8LoFPIgPzTmGGAPs7HL5gg7SqrfOOA4o gFqw== X-Gm-Message-State: AOAM533TZphGotepk71C7s3lh10NZTeAAgT5XCW+d6cWf+vt51cGgwmj q++Tp8tVAts2xX7uOG+DOvJ5K2ZPqUfJ0fPI67MXaQ== X-Google-Smtp-Source: ABdhPJyR9U54DVM3ct2LPaeHjq5X1d2P4BRZ/v7s+ft4w68F9k24e6xs01fvMqO6kK5J5ovEHtn/qv7W3jzDWk3bGTA= X-Received: by 2002:a05:6512:308d:: with SMTP id z13mr13877780lfd.523.1643054401020; Mon, 24 Jan 2022 12:00:01 -0800 (PST) Precedence: bulk X-Mailing-List: llvm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 References: <20220119135450.564115-1-Jason@zx2c4.com> <20220124192849.14755-1-Jason@zx2c4.com> In-Reply-To: <20220124192849.14755-1-Jason@zx2c4.com> From: Nick Desaulniers Date: Mon, 24 Jan 2022 11:59:48 -0800 Message-ID: Subject: Re: [PATCH v2] lib/crypto: blake2s: avoid indirect calls to compression function for Clang CFI To: "Jason A. Donenfeld" Cc: Linux ARM , Linux Crypto Mailing List , Linux Kernel Mailing List , llvm@lists.linux.dev, Sami Tolvanen , Nathan Chancellor , Eric Biggers , Herbert Xu , Miles Chen , Ard Biesheuvel Content-Type: text/plain; charset="UTF-8" On Mon, Jan 24, 2022 at 11:29 AM Jason A. Donenfeld wrote: > > blake2s_compress_generic is weakly aliased by blake2s_generic. The > current harness for function selection uses a function pointer, which is > ordinarily inlined and resolved at compile time. But when Clang's CFI is > enabled, CFI still triggers when making an indirect call via a weak > symbol. This seems like a bug in Clang's CFI, as though it's bucketing > weak symbols and strong symbols differently. It also only seems to > trigger when "full LTO" mode is used, rather than "thin LTO". > > [ 0.000000][ T0] Kernel panic - not syncing: CFI failure (target: blake2s_compress_generic+0x0/0x1444) > [ 0.000000][ T0] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.16.0-mainline-06981-g076c855b846e #1 > [ 0.000000][ T0] Hardware name: MT6873 (DT) > [ 0.000000][ T0] Call trace: > [ 0.000000][ T0] dump_backtrace+0xfc/0x1dc > [ 0.000000][ T0] dump_stack_lvl+0xa8/0x11c > [ 0.000000][ T0] panic+0x194/0x464 > [ 0.000000][ T0] __cfi_check_fail+0x54/0x58 > [ 0.000000][ T0] __cfi_slowpath_diag+0x354/0x4b0 > [ 0.000000][ T0] blake2s_update+0x14c/0x178 > [ 0.000000][ T0] _extract_entropy+0xf4/0x29c > [ 0.000000][ T0] crng_initialize_primary+0x24/0x94 > [ 0.000000][ T0] rand_initialize+0x2c/0x6c > [ 0.000000][ T0] start_kernel+0x2f8/0x65c > [ 0.000000][ T0] __primary_switched+0xc4/0x7be4 > [ 0.000000][ T0] Rebooting in 5 seconds.. > > Nonetheless, the function pointer method isn't so terrific anyway, so > this patch replaces it with a simple boolean, which also gets inlined > away. This successfully works around the Clang bug. Acked-by: Nick Desaulniers Thanks for the report. Once we have a fix in hand for LLVM, we can revisit removing this and raising the required LLVM version for CFI. > > In general, I'm not too keen on all of the indirection involved here; it > clearly does more harm than good. Hopefully the whole thing can get > cleaned up down the road when lib/crypto is overhauled more > comprehensively. But for now, we go with a simple bandaid. > > Fixes: 6048fdcc5f26 ("lib/crypto: blake2s: include as built-in") > Reported-by: Miles Chen > Tested-by: Miles Chen > Tested-by: Nathan Chancellor > Link: https://github.com/ClangBuiltLinux/linux/issues/1567 > Cc: Nick Desaulniers > Cc: Sami Tolvanen > Cc: Ard Biesheuvel > Signed-off-by: Jason A. Donenfeld > --- > Changes v1->v2: > - Wrapped columns at 80 for Eric. > > arch/arm/crypto/blake2s-shash.c | 4 ++-- > arch/x86/crypto/blake2s-shash.c | 4 ++-- > crypto/blake2s_generic.c | 4 ++-- > include/crypto/internal/blake2s.h | 40 +++++++++++++++++++------------ > lib/crypto/blake2s.c | 4 ++-- > 5 files changed, 33 insertions(+), 23 deletions(-) > > diff --git a/arch/arm/crypto/blake2s-shash.c b/arch/arm/crypto/blake2s-shash.c > index 17c1c3bfe2f5..763c73beea2d 100644 > --- a/arch/arm/crypto/blake2s-shash.c > +++ b/arch/arm/crypto/blake2s-shash.c > @@ -13,12 +13,12 @@ > static int crypto_blake2s_update_arm(struct shash_desc *desc, > const u8 *in, unsigned int inlen) > { > - return crypto_blake2s_update(desc, in, inlen, blake2s_compress); > + return crypto_blake2s_update(desc, in, inlen, false); > } > > static int crypto_blake2s_final_arm(struct shash_desc *desc, u8 *out) > { > - return crypto_blake2s_final(desc, out, blake2s_compress); > + return crypto_blake2s_final(desc, out, false); > } > > #define BLAKE2S_ALG(name, driver_name, digest_size) \ > diff --git a/arch/x86/crypto/blake2s-shash.c b/arch/x86/crypto/blake2s-shash.c > index f9e2fecdb761..59ae28abe35c 100644 > --- a/arch/x86/crypto/blake2s-shash.c > +++ b/arch/x86/crypto/blake2s-shash.c > @@ -18,12 +18,12 @@ > static int crypto_blake2s_update_x86(struct shash_desc *desc, > const u8 *in, unsigned int inlen) > { > - return crypto_blake2s_update(desc, in, inlen, blake2s_compress); > + return crypto_blake2s_update(desc, in, inlen, false); > } > > static int crypto_blake2s_final_x86(struct shash_desc *desc, u8 *out) > { > - return crypto_blake2s_final(desc, out, blake2s_compress); > + return crypto_blake2s_final(desc, out, false); > } > > #define BLAKE2S_ALG(name, driver_name, digest_size) \ > diff --git a/crypto/blake2s_generic.c b/crypto/blake2s_generic.c > index 72fe480f9bd6..5f96a21f8788 100644 > --- a/crypto/blake2s_generic.c > +++ b/crypto/blake2s_generic.c > @@ -15,12 +15,12 @@ > static int crypto_blake2s_update_generic(struct shash_desc *desc, > const u8 *in, unsigned int inlen) > { > - return crypto_blake2s_update(desc, in, inlen, blake2s_compress_generic); > + return crypto_blake2s_update(desc, in, inlen, true); > } > > static int crypto_blake2s_final_generic(struct shash_desc *desc, u8 *out) > { > - return crypto_blake2s_final(desc, out, blake2s_compress_generic); > + return crypto_blake2s_final(desc, out, true); > } > > #define BLAKE2S_ALG(name, driver_name, digest_size) \ > diff --git a/include/crypto/internal/blake2s.h b/include/crypto/internal/blake2s.h > index d39cfa0d333e..52363eee2b20 100644 > --- a/include/crypto/internal/blake2s.h > +++ b/include/crypto/internal/blake2s.h > @@ -24,14 +24,11 @@ static inline void blake2s_set_lastblock(struct blake2s_state *state) > state->f[0] = -1; > } > > -typedef void (*blake2s_compress_t)(struct blake2s_state *state, > - const u8 *block, size_t nblocks, u32 inc); > - > /* Helper functions for BLAKE2s shared by the library and shash APIs */ > > -static inline void __blake2s_update(struct blake2s_state *state, > - const u8 *in, size_t inlen, > - blake2s_compress_t compress) > +static __always_inline void > +__blake2s_update(struct blake2s_state *state, const u8 *in, size_t inlen, > + bool force_generic) > { > const size_t fill = BLAKE2S_BLOCK_SIZE - state->buflen; > > @@ -39,7 +36,12 @@ static inline void __blake2s_update(struct blake2s_state *state, > return; > if (inlen > fill) { > memcpy(state->buf + state->buflen, in, fill); > - (*compress)(state, state->buf, 1, BLAKE2S_BLOCK_SIZE); > + if (force_generic) > + blake2s_compress_generic(state, state->buf, 1, > + BLAKE2S_BLOCK_SIZE); > + else > + blake2s_compress(state, state->buf, 1, > + BLAKE2S_BLOCK_SIZE); > state->buflen = 0; > in += fill; > inlen -= fill; > @@ -47,7 +49,12 @@ static inline void __blake2s_update(struct blake2s_state *state, > if (inlen > BLAKE2S_BLOCK_SIZE) { > const size_t nblocks = DIV_ROUND_UP(inlen, BLAKE2S_BLOCK_SIZE); > /* Hash one less (full) block than strictly possible */ > - (*compress)(state, in, nblocks - 1, BLAKE2S_BLOCK_SIZE); > + if (force_generic) > + blake2s_compress_generic(state, in, nblocks - 1, > + BLAKE2S_BLOCK_SIZE); > + else > + blake2s_compress(state, in, nblocks - 1, > + BLAKE2S_BLOCK_SIZE); > in += BLAKE2S_BLOCK_SIZE * (nblocks - 1); > inlen -= BLAKE2S_BLOCK_SIZE * (nblocks - 1); > } > @@ -55,13 +62,16 @@ static inline void __blake2s_update(struct blake2s_state *state, > state->buflen += inlen; > } > > -static inline void __blake2s_final(struct blake2s_state *state, u8 *out, > - blake2s_compress_t compress) > +static __always_inline void > +__blake2s_final(struct blake2s_state *state, u8 *out, bool force_generic) > { > blake2s_set_lastblock(state); > memset(state->buf + state->buflen, 0, > BLAKE2S_BLOCK_SIZE - state->buflen); /* Padding */ > - (*compress)(state, state->buf, 1, state->buflen); > + if (force_generic) > + blake2s_compress_generic(state, state->buf, 1, state->buflen); > + else > + blake2s_compress(state, state->buf, 1, state->buflen); > cpu_to_le32_array(state->h, ARRAY_SIZE(state->h)); > memcpy(out, state->h, state->outlen); > } > @@ -99,20 +109,20 @@ static inline int crypto_blake2s_init(struct shash_desc *desc) > > static inline int crypto_blake2s_update(struct shash_desc *desc, > const u8 *in, unsigned int inlen, > - blake2s_compress_t compress) > + bool force_generic) > { > struct blake2s_state *state = shash_desc_ctx(desc); > > - __blake2s_update(state, in, inlen, compress); > + __blake2s_update(state, in, inlen, force_generic); > return 0; > } > > static inline int crypto_blake2s_final(struct shash_desc *desc, u8 *out, > - blake2s_compress_t compress) > + bool force_generic) > { > struct blake2s_state *state = shash_desc_ctx(desc); > > - __blake2s_final(state, out, compress); > + __blake2s_final(state, out, force_generic); > return 0; > } > > diff --git a/lib/crypto/blake2s.c b/lib/crypto/blake2s.c > index 9364f79937b8..c71c09621c09 100644 > --- a/lib/crypto/blake2s.c > +++ b/lib/crypto/blake2s.c > @@ -18,14 +18,14 @@ > > void blake2s_update(struct blake2s_state *state, const u8 *in, size_t inlen) > { > - __blake2s_update(state, in, inlen, blake2s_compress); > + __blake2s_update(state, in, inlen, false); > } > EXPORT_SYMBOL(blake2s_update); > > void blake2s_final(struct blake2s_state *state, u8 *out) > { > WARN_ON(IS_ENABLED(DEBUG) && !out); > - __blake2s_final(state, out, blake2s_compress); > + __blake2s_final(state, out, false); > memzero_explicit(state, sizeof(*state)); > } > EXPORT_SYMBOL(blake2s_final); > -- > 2.34.1 > -- Thanks, ~Nick Desaulniers From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B4B5EC433EF for ; Mon, 24 Jan 2022 20:01:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=jhLc8JLReTa5U7QwHid+xzouQ1k5ax1q7Y3naIVu6G0=; b=ZyAZlYvvyq09B5 GEMtxt8Pc8kglVFf21qnJwuG8ZZHNr7Gc15vyWMwcbsgzgXyNvhoqhKk1UrrHND35+kGwEeVYUhkE d2hhLipT+vjNz6mM5Va7JSJIEbiwJE+DvK1MvM4HvjFNWEycp1y2PhMMioIWa2mTiraThzTiLOFRq JYTD9TtDar1ft710qyPpF570gDB6ufvgiZe8Mha4NgSnX+7GGCaMw+yYtjKvpanfTk0JHfAHDnAGo RHccaGtzLa7sVKCBPc0oUk7IKunV/cf1e1DR9S6l5+bLUSbf3Ch24mPaBJzR7y6B27C0H9aHWdojx 5BSueC0zxAynXXY+Dy9w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nC5Vd-005004-FJ; Mon, 24 Jan 2022 20:00:13 +0000 Received: from mail-lf1-x12f.google.com ([2a00:1450:4864:20::12f]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nC5VT-004ztu-Lg for linux-arm-kernel@lists.infradead.org; Mon, 24 Jan 2022 20:00:05 +0000 Received: by mail-lf1-x12f.google.com with SMTP id b9so3165550lfq.6 for ; Mon, 24 Jan 2022 12:00:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=X1fspDfOZTRESHgvCsTpy0M+wXpoDo9iEK0jnJXDU5Y=; b=liPSw+oHKbLKt49NG7WwlravT+ecqQGWP3Y1Q8azHjQmV39vlahZekjyxHrUorHG/s lgHi+ZN/pKPrXARZMuuyXAiagpmQ754zvGdFo3GwVY1mEUx8JkonDBNdCBKtOm/CsO0H l57Gfv77T9qgQK8hck4Gu9uzAdq07iWfS/Nn0d18+937IolDJ4rDoR6RYdCIOAMkGGiT o86l+PSF7Bih3oEb7/WysNBmRTnzCpUJ46UIpdYMj/oPP3A3W2RSItFoPKkWjpDdyRT2 pVIUwn2blEfUxS8k8l1bWErFN2xPD3V9KfxdKz0wOpdP13mwNeNWhey02A8C1wkng6w0 P6Ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=X1fspDfOZTRESHgvCsTpy0M+wXpoDo9iEK0jnJXDU5Y=; b=4Gbrg8oaRuQddGStvipCmay1pzEeKWnempzM/LRZqPEmquVgXk4b0fQZi7siBRaAyg OivzhJcTYbLmJzOAEI1zQb6kuzZrF39anjxc9gskIaeb4rTskM1lXbCV/Q3ki8sSvSNe sybm62nIiYAWT67osdiXP6Dk4eAJefva4zIn5V3WQ21gYgD2d3sdKrw6A4vcercFzNsm XC75NesYMp8IACNCRCpCAGT06dAAEuo3Zveu1deeNpferbHOBYRSjzcuAsZHIMUMOJlO pW/jutsTzCaqINxxSJoYE/VYP2Ll/h4GfCOTaoXrwZ7hcmLLIK/X/b859LHr+kNezBFO HDMA== X-Gm-Message-State: AOAM532FOK/buOn9A+WNKEiQtZRpeJYKyJxqsJazEx4HNC753VCWtIsQ BnzsHO0vcKEMhNLZeHhXpLJQzvZdql0SoFsXuLdUXg== X-Google-Smtp-Source: ABdhPJyR9U54DVM3ct2LPaeHjq5X1d2P4BRZ/v7s+ft4w68F9k24e6xs01fvMqO6kK5J5ovEHtn/qv7W3jzDWk3bGTA= X-Received: by 2002:a05:6512:308d:: with SMTP id z13mr13877780lfd.523.1643054401020; Mon, 24 Jan 2022 12:00:01 -0800 (PST) MIME-Version: 1.0 References: <20220119135450.564115-1-Jason@zx2c4.com> <20220124192849.14755-1-Jason@zx2c4.com> In-Reply-To: <20220124192849.14755-1-Jason@zx2c4.com> From: Nick Desaulniers Date: Mon, 24 Jan 2022 11:59:48 -0800 Message-ID: Subject: Re: [PATCH v2] lib/crypto: blake2s: avoid indirect calls to compression function for Clang CFI To: "Jason A. Donenfeld" Cc: Linux ARM , Linux Crypto Mailing List , Linux Kernel Mailing List , llvm@lists.linux.dev, Sami Tolvanen , Nathan Chancellor , Eric Biggers , Herbert Xu , Miles Chen , Ard Biesheuvel X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220124_120003_783554_5A585153 X-CRM114-Status: GOOD ( 31.00 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, Jan 24, 2022 at 11:29 AM Jason A. Donenfeld wrote: > > blake2s_compress_generic is weakly aliased by blake2s_generic. The > current harness for function selection uses a function pointer, which is > ordinarily inlined and resolved at compile time. But when Clang's CFI is > enabled, CFI still triggers when making an indirect call via a weak > symbol. This seems like a bug in Clang's CFI, as though it's bucketing > weak symbols and strong symbols differently. It also only seems to > trigger when "full LTO" mode is used, rather than "thin LTO". > > [ 0.000000][ T0] Kernel panic - not syncing: CFI failure (target: blake2s_compress_generic+0x0/0x1444) > [ 0.000000][ T0] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.16.0-mainline-06981-g076c855b846e #1 > [ 0.000000][ T0] Hardware name: MT6873 (DT) > [ 0.000000][ T0] Call trace: > [ 0.000000][ T0] dump_backtrace+0xfc/0x1dc > [ 0.000000][ T0] dump_stack_lvl+0xa8/0x11c > [ 0.000000][ T0] panic+0x194/0x464 > [ 0.000000][ T0] __cfi_check_fail+0x54/0x58 > [ 0.000000][ T0] __cfi_slowpath_diag+0x354/0x4b0 > [ 0.000000][ T0] blake2s_update+0x14c/0x178 > [ 0.000000][ T0] _extract_entropy+0xf4/0x29c > [ 0.000000][ T0] crng_initialize_primary+0x24/0x94 > [ 0.000000][ T0] rand_initialize+0x2c/0x6c > [ 0.000000][ T0] start_kernel+0x2f8/0x65c > [ 0.000000][ T0] __primary_switched+0xc4/0x7be4 > [ 0.000000][ T0] Rebooting in 5 seconds.. > > Nonetheless, the function pointer method isn't so terrific anyway, so > this patch replaces it with a simple boolean, which also gets inlined > away. This successfully works around the Clang bug. Acked-by: Nick Desaulniers Thanks for the report. Once we have a fix in hand for LLVM, we can revisit removing this and raising the required LLVM version for CFI. > > In general, I'm not too keen on all of the indirection involved here; it > clearly does more harm than good. Hopefully the whole thing can get > cleaned up down the road when lib/crypto is overhauled more > comprehensively. But for now, we go with a simple bandaid. > > Fixes: 6048fdcc5f26 ("lib/crypto: blake2s: include as built-in") > Reported-by: Miles Chen > Tested-by: Miles Chen > Tested-by: Nathan Chancellor > Link: https://github.com/ClangBuiltLinux/linux/issues/1567 > Cc: Nick Desaulniers > Cc: Sami Tolvanen > Cc: Ard Biesheuvel > Signed-off-by: Jason A. Donenfeld > --- > Changes v1->v2: > - Wrapped columns at 80 for Eric. > > arch/arm/crypto/blake2s-shash.c | 4 ++-- > arch/x86/crypto/blake2s-shash.c | 4 ++-- > crypto/blake2s_generic.c | 4 ++-- > include/crypto/internal/blake2s.h | 40 +++++++++++++++++++------------ > lib/crypto/blake2s.c | 4 ++-- > 5 files changed, 33 insertions(+), 23 deletions(-) > > diff --git a/arch/arm/crypto/blake2s-shash.c b/arch/arm/crypto/blake2s-shash.c > index 17c1c3bfe2f5..763c73beea2d 100644 > --- a/arch/arm/crypto/blake2s-shash.c > +++ b/arch/arm/crypto/blake2s-shash.c > @@ -13,12 +13,12 @@ > static int crypto_blake2s_update_arm(struct shash_desc *desc, > const u8 *in, unsigned int inlen) > { > - return crypto_blake2s_update(desc, in, inlen, blake2s_compress); > + return crypto_blake2s_update(desc, in, inlen, false); > } > > static int crypto_blake2s_final_arm(struct shash_desc *desc, u8 *out) > { > - return crypto_blake2s_final(desc, out, blake2s_compress); > + return crypto_blake2s_final(desc, out, false); > } > > #define BLAKE2S_ALG(name, driver_name, digest_size) \ > diff --git a/arch/x86/crypto/blake2s-shash.c b/arch/x86/crypto/blake2s-shash.c > index f9e2fecdb761..59ae28abe35c 100644 > --- a/arch/x86/crypto/blake2s-shash.c > +++ b/arch/x86/crypto/blake2s-shash.c > @@ -18,12 +18,12 @@ > static int crypto_blake2s_update_x86(struct shash_desc *desc, > const u8 *in, unsigned int inlen) > { > - return crypto_blake2s_update(desc, in, inlen, blake2s_compress); > + return crypto_blake2s_update(desc, in, inlen, false); > } > > static int crypto_blake2s_final_x86(struct shash_desc *desc, u8 *out) > { > - return crypto_blake2s_final(desc, out, blake2s_compress); > + return crypto_blake2s_final(desc, out, false); > } > > #define BLAKE2S_ALG(name, driver_name, digest_size) \ > diff --git a/crypto/blake2s_generic.c b/crypto/blake2s_generic.c > index 72fe480f9bd6..5f96a21f8788 100644 > --- a/crypto/blake2s_generic.c > +++ b/crypto/blake2s_generic.c > @@ -15,12 +15,12 @@ > static int crypto_blake2s_update_generic(struct shash_desc *desc, > const u8 *in, unsigned int inlen) > { > - return crypto_blake2s_update(desc, in, inlen, blake2s_compress_generic); > + return crypto_blake2s_update(desc, in, inlen, true); > } > > static int crypto_blake2s_final_generic(struct shash_desc *desc, u8 *out) > { > - return crypto_blake2s_final(desc, out, blake2s_compress_generic); > + return crypto_blake2s_final(desc, out, true); > } > > #define BLAKE2S_ALG(name, driver_name, digest_size) \ > diff --git a/include/crypto/internal/blake2s.h b/include/crypto/internal/blake2s.h > index d39cfa0d333e..52363eee2b20 100644 > --- a/include/crypto/internal/blake2s.h > +++ b/include/crypto/internal/blake2s.h > @@ -24,14 +24,11 @@ static inline void blake2s_set_lastblock(struct blake2s_state *state) > state->f[0] = -1; > } > > -typedef void (*blake2s_compress_t)(struct blake2s_state *state, > - const u8 *block, size_t nblocks, u32 inc); > - > /* Helper functions for BLAKE2s shared by the library and shash APIs */ > > -static inline void __blake2s_update(struct blake2s_state *state, > - const u8 *in, size_t inlen, > - blake2s_compress_t compress) > +static __always_inline void > +__blake2s_update(struct blake2s_state *state, const u8 *in, size_t inlen, > + bool force_generic) > { > const size_t fill = BLAKE2S_BLOCK_SIZE - state->buflen; > > @@ -39,7 +36,12 @@ static inline void __blake2s_update(struct blake2s_state *state, > return; > if (inlen > fill) { > memcpy(state->buf + state->buflen, in, fill); > - (*compress)(state, state->buf, 1, BLAKE2S_BLOCK_SIZE); > + if (force_generic) > + blake2s_compress_generic(state, state->buf, 1, > + BLAKE2S_BLOCK_SIZE); > + else > + blake2s_compress(state, state->buf, 1, > + BLAKE2S_BLOCK_SIZE); > state->buflen = 0; > in += fill; > inlen -= fill; > @@ -47,7 +49,12 @@ static inline void __blake2s_update(struct blake2s_state *state, > if (inlen > BLAKE2S_BLOCK_SIZE) { > const size_t nblocks = DIV_ROUND_UP(inlen, BLAKE2S_BLOCK_SIZE); > /* Hash one less (full) block than strictly possible */ > - (*compress)(state, in, nblocks - 1, BLAKE2S_BLOCK_SIZE); > + if (force_generic) > + blake2s_compress_generic(state, in, nblocks - 1, > + BLAKE2S_BLOCK_SIZE); > + else > + blake2s_compress(state, in, nblocks - 1, > + BLAKE2S_BLOCK_SIZE); > in += BLAKE2S_BLOCK_SIZE * (nblocks - 1); > inlen -= BLAKE2S_BLOCK_SIZE * (nblocks - 1); > } > @@ -55,13 +62,16 @@ static inline void __blake2s_update(struct blake2s_state *state, > state->buflen += inlen; > } > > -static inline void __blake2s_final(struct blake2s_state *state, u8 *out, > - blake2s_compress_t compress) > +static __always_inline void > +__blake2s_final(struct blake2s_state *state, u8 *out, bool force_generic) > { > blake2s_set_lastblock(state); > memset(state->buf + state->buflen, 0, > BLAKE2S_BLOCK_SIZE - state->buflen); /* Padding */ > - (*compress)(state, state->buf, 1, state->buflen); > + if (force_generic) > + blake2s_compress_generic(state, state->buf, 1, state->buflen); > + else > + blake2s_compress(state, state->buf, 1, state->buflen); > cpu_to_le32_array(state->h, ARRAY_SIZE(state->h)); > memcpy(out, state->h, state->outlen); > } > @@ -99,20 +109,20 @@ static inline int crypto_blake2s_init(struct shash_desc *desc) > > static inline int crypto_blake2s_update(struct shash_desc *desc, > const u8 *in, unsigned int inlen, > - blake2s_compress_t compress) > + bool force_generic) > { > struct blake2s_state *state = shash_desc_ctx(desc); > > - __blake2s_update(state, in, inlen, compress); > + __blake2s_update(state, in, inlen, force_generic); > return 0; > } > > static inline int crypto_blake2s_final(struct shash_desc *desc, u8 *out, > - blake2s_compress_t compress) > + bool force_generic) > { > struct blake2s_state *state = shash_desc_ctx(desc); > > - __blake2s_final(state, out, compress); > + __blake2s_final(state, out, force_generic); > return 0; > } > > diff --git a/lib/crypto/blake2s.c b/lib/crypto/blake2s.c > index 9364f79937b8..c71c09621c09 100644 > --- a/lib/crypto/blake2s.c > +++ b/lib/crypto/blake2s.c > @@ -18,14 +18,14 @@ > > void blake2s_update(struct blake2s_state *state, const u8 *in, size_t inlen) > { > - __blake2s_update(state, in, inlen, blake2s_compress); > + __blake2s_update(state, in, inlen, false); > } > EXPORT_SYMBOL(blake2s_update); > > void blake2s_final(struct blake2s_state *state, u8 *out) > { > WARN_ON(IS_ENABLED(DEBUG) && !out); > - __blake2s_final(state, out, blake2s_compress); > + __blake2s_final(state, out, false); > memzero_explicit(state, sizeof(*state)); > } > EXPORT_SYMBOL(blake2s_final); > -- > 2.34.1 > -- Thanks, ~Nick Desaulniers _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel