From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.6 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A148CC17445 for ; Mon, 11 Nov 2019 19:14:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 70268206BA for ; Mon, 11 Nov 2019 19:14:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=chronox.de header.i=@chronox.de header.b="qQkTGOJc" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727860AbfKKTOZ (ORCPT ); Mon, 11 Nov 2019 14:14:25 -0500 Received: from mo4-p04-ob.smtp.rzone.de ([85.215.255.124]:30821 "EHLO mo4-p04-ob.smtp.rzone.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727112AbfKKTOY (ORCPT ); Mon, 11 Nov 2019 14:14:24 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1573499660; s=strato-dkim-0002; d=chronox.de; h=References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: X-RZG-CLASS-ID:X-RZG-AUTH:From:Subject:Sender; bh=hzT2sQmGMWFbo8zas0c/uXFDbupufRKgC7qKAjtgBIA=; b=qQkTGOJcrr4JVLrAdvxVG/sSObdyGFzr6+u1NhF1A/WMBGY9sxW9VhbiOfqQNo+rGv Cv17/BSS2xCWyXH1i+RZ4rfYj2wA/ldLzdxp7ZTgHbDNMR6My+x/9anyQaxHg3c7v1ek y5fBUsbwCh0Ajr/zbKF30vM46VOz1KqkUeSRq/OT6cP1+QgA0xDc+D6W2kOTmw4JAFp9 t0lZ8p3D7ZXwwMXDiAsHyJq2+YuHFfVYgs73mASx20OowdEvrue+2XOx3BTcLFJUnYts AELqKOeQCCv7C5M9aOTjDSOkkS0bliEs1kX+kZu9hZt7aaBiCY5UY+WH9WAwRYk539Vc fW6Q== X-RZG-AUTH: ":P2ERcEykfu11Y98lp/T7+hdri+uKZK8TKWEqNyiHySGSa9k9zmgLKehaO2hZDSTWbgzIOA==" X-RZG-CLASS-ID: mo00 Received: from positron.chronox.de by smtp.strato.de (RZmta 44.29.0 AUTH) with ESMTPSA id N09a57vABJDD3Xb (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (curve secp521r1 with 521 ECDH bits, eq. 15360 bits RSA)) (Client did not present a certificate); Mon, 11 Nov 2019 20:13:13 +0100 (CET) From: Stephan =?ISO-8859-1?Q?M=FCller?= To: Arnd Bergmann Cc: Greg Kroah-Hartman , linux-crypto@vger.kernel.org, LKML , linux-api@vger.kernel.org, "Eric W. Biederman" , "Alexander E. Patrakov" , "Ahmed S. Darwish" , "Theodore Y. Ts'o" , Willy Tarreau , Matthew Garrett , Vito Caputo , Andreas Dilger , Jan Kara , Ray Strode , William Jon McCann , zhangjs , Andy Lutomirski , Florian Weimer , Lennart Poettering , Nicolai Stange , "Peter, Matthias" , Marcelo Henrique Cerri , Roman Drahtmueller , Neil Horman Subject: [PATCH v24 02/12] LRNG - allocate one SDRNG instance per NUMA node Date: Mon, 11 Nov 2019 19:19:20 +0100 Message-ID: <5150773.fIWb0X7rEq@positron.chronox.de> In-Reply-To: <6157374.ptSnyUpaCn@positron.chronox.de> References: <6157374.ptSnyUpaCn@positron.chronox.de> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="iso-8859-1" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In order to improve NUMA-locality when serving getrandom(2) requests, allocate one DRNG instance per node. The SDRNG instance that is present right from the start of the kernel is reused as the first per-NUMA-node SDRNG. For all remaining online NUMA nodes a new SDRNG instance is allocated. During boot time, the multiple SDRNG instances are seeded sequentially. With this, the first SDRNG instance (referenced as the initial SDRNG in the code) is completely seeded with 256 bits of entropy before the next SDRNG instance is completely seeded. When random numbers are requested, the NUMA-node-local SDRNG is checked whether it has been already fully seeded. If this is not the case, the initial SDRNG is used to serve the request. CC: "Eric W. Biederman" CC: "Alexander E. Patrakov" CC: "Ahmed S. Darwish" CC: "Theodore Y. Ts'o" CC: Willy Tarreau CC: Matthew Garrett CC: Vito Caputo CC: Andreas Dilger CC: Jan Kara CC: Ray Strode CC: William Jon McCann CC: zhangjs CC: Andy Lutomirski CC: Florian Weimer CC: Lennart Poettering CC: Nicolai Stange Reviewed-by: Marcelo Henrique Cerri Reviewed-by: Roman Drahtmueller Tested-by: Roman Drahtm=FCller Tested-by: Marcelo Henrique Cerri Tested-by: Neil Horman Signed-off-by: Stephan Mueller =2D-- drivers/char/lrng/Makefile | 2 + drivers/char/lrng/lrng_internal.h | 5 ++ drivers/char/lrng/lrng_numa.c | 114 ++++++++++++++++++++++++++++++ 3 files changed, 121 insertions(+) create mode 100644 drivers/char/lrng/lrng_numa.c diff --git a/drivers/char/lrng/Makefile b/drivers/char/lrng/Makefile index 2761623715d2..a00cddb45773 100644 =2D-- a/drivers/char/lrng/Makefile +++ b/drivers/char/lrng/Makefile @@ -7,3 +7,5 @@ obj-y +=3D lrng_pool.o lrng_aux.o \ lrng_sw_noise.o lrng_archrandom.o \ lrng_sdrng.o lrng_chacha20.o \ lrng_interfaces.o \ + +obj-$(CONFIG_NUMA) +=3D lrng_numa.o diff --git a/drivers/char/lrng/lrng_internal.h b/drivers/char/lrng/lrng_int= ernal.h index 242f9b5b4f3d..e6ac2c527378 100644 =2D-- a/drivers/char/lrng/lrng_internal.h +++ b/drivers/char/lrng/lrng_internal.h @@ -263,8 +263,13 @@ int lrng_sdrng_get_sleep(u8 *outbuf, u32 outbuflen); void lrng_sdrng_force_reseed(void); void lrng_sdrng_seed_work(struct work_struct *dummy); =20 +#ifdef CONFIG_NUMA +struct lrng_sdrng **lrng_sdrng_instances(void); +void lrng_drngs_numa_alloc(void); +#else /* CONFIG_NUMA */ static inline struct lrng_sdrng **lrng_sdrng_instances(void) { return NULL= ; } static inline void lrng_drngs_numa_alloc(void) { return; } +#endif /* CONFIG_NUMA */ =20 /************************** Health Test linking code *********************= *****/ =20 diff --git a/drivers/char/lrng/lrng_numa.c b/drivers/char/lrng/lrng_numa.c new file mode 100644 index 000000000000..e88de91cb0aa =2D-- /dev/null +++ b/drivers/char/lrng/lrng_numa.c @@ -0,0 +1,114 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause +/* + * LRNG NUMA support + * + * Copyright (C) 2016 - 2019, Stephan Mueller + * + * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED + * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF + * WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT + * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE + * USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH + * DAMAGE. + */ + +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +#include +#include + +#include "lrng_internal.h" + +static struct lrng_sdrng **lrng_sdrng __read_mostly =3D NULL; + +struct lrng_sdrng **lrng_sdrng_instances(void) +{ + return lrng_sdrng; +} + +/* Allocate the data structures for the per-NUMA node DRNGs */ +static void _lrng_drngs_numa_alloc(struct work_struct *work) +{ + struct lrng_sdrng **sdrngs; + struct lrng_sdrng *lrng_sdrng_init =3D lrng_sdrng_init_instance(); + u32 node; + bool init_sdrng_used =3D false; + + mutex_lock(&lrng_crypto_cb_update); + + /* per-NUMA-node DRNGs are already present */ + if (lrng_sdrng) + goto unlock; + + sdrngs =3D kcalloc(nr_node_ids, sizeof(void *), GFP_KERNEL|__GFP_NOFAIL); + for_each_online_node(node) { + struct lrng_sdrng *sdrng; + + if (!init_sdrng_used) { + sdrngs[node] =3D lrng_sdrng_init; + init_sdrng_used =3D true; + continue; + } + + sdrng =3D kmalloc_node(sizeof(struct lrng_sdrng), + GFP_KERNEL|__GFP_NOFAIL, node); + memset(sdrng, 0, sizeof(lrng_sdrng)); + + sdrng->crypto_cb =3D lrng_sdrng_init->crypto_cb; + sdrng->sdrng =3D sdrng->crypto_cb->lrng_drng_alloc( + LRNG_DRNG_SECURITY_STRENGTH_BYTES); + if (IS_ERR(sdrng->sdrng)) { + kfree(sdrng); + goto err; + } + + mutex_init(&sdrng->lock); + spin_lock_init(&sdrng->spin_lock); + + /* + * No reseeding of NUMA DRNGs from previous DRNGs as this + * would complicate the code. Let it simply reseed. + */ + lrng_sdrng_reset(sdrng); + sdrngs[node] =3D sdrng; + + lrng_pool_inc_numa_node(); + pr_info("secondary DRNG for NUMA node %d allocated\n", node); + } + + /* Ensure that all NUMA nodes receive changed memory here. */ + mb(); + + if (!cmpxchg(&lrng_sdrng, NULL, sdrngs)) + goto unlock; + +err: + for_each_online_node(node) { + struct lrng_sdrng *sdrng =3D sdrngs[node]; + + if (sdrng =3D=3D lrng_sdrng_init) + continue; + + if (sdrng) { + sdrng->crypto_cb->lrng_drng_dealloc(sdrng->sdrng); + kfree(sdrng); + } + } + kfree(sdrngs); + +unlock: + mutex_unlock(&lrng_crypto_cb_update); +} + +static DECLARE_WORK(lrng_drngs_numa_alloc_work, _lrng_drngs_numa_alloc); + +void lrng_drngs_numa_alloc(void) +{ + schedule_work(&lrng_drngs_numa_alloc_work); +} =2D-=20 2.23.0