From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7BFAC433E2 for ; Wed, 13 May 2020 23:34:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9052420659 for ; Wed, 13 May 2020 23:34:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9052420659 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=hisilicon.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 06F8E90013A; Wed, 13 May 2020 19:34:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 020FC900022; Wed, 13 May 2020 19:34:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E518390013A; Wed, 13 May 2020 19:34:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0023.hostedemail.com [216.40.44.23]) by kanga.kvack.org (Postfix) with ESMTP id CD38F900022 for ; Wed, 13 May 2020 19:34:23 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 900CD440B for ; Wed, 13 May 2020 23:34:23 +0000 (UTC) X-FDA: 76813302006.24.balls90_7d880595cbb0a X-HE-Tag: balls90_7d880595cbb0a X-Filterd-Recvd-Size: 12128 Received: from huawei.com (szxga07-in.huawei.com [45.249.212.35]) by imf11.hostedemail.com (Postfix) with ESMTP for ; Wed, 13 May 2020 23:34:22 +0000 (UTC) Received: from DGGEMS404-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 1DE792BE79F28EFD01FA; Thu, 14 May 2020 07:34:19 +0800 (CST) Received: from SWX921481.china.huawei.com (10.126.202.158) by DGGEMS404-HUB.china.huawei.com (10.3.19.204) with Microsoft SMTP Server id 14.3.487.0; Thu, 14 May 2020 07:34:08 +0800 From: Barry Song To: , , , , CC: , , , , Barry Song Subject: [PATCH RESEND 1/1] mm/zswap: move to use crypto_acomp API for hardware acceleration Date: Thu, 14 May 2020 11:33:18 +1200 Message-ID: <20200513233318.10496-2-song.bao.hua@hisilicon.com> X-Mailer: git-send-email 2.21.0.windows.1 In-Reply-To: <20200513233318.10496-1-song.bao.hua@hisilicon.com> References: <20200513233318.10496-1-song.bao.hua@hisilicon.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [10.126.202.158] X-CFilter-Loop: Reflected Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: right now, new drivers are adapted to crypto_acomp APIs rather than crypt= o_comp APIs. Tradiontal compressors like lz4,lzo etc have been wrapped into acom= p via scomp backend as well. To use the new hardware acceleration for compression, zswap should move t= o use acomp APIs. Platforms without async accelerators can still work through s= comp. Signed-off-by: Barry Song --- mm/zswap.c | 150 ++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 108 insertions(+), 42 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index fbb782924ccc..44a982632deb 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -24,8 +24,10 @@ #include #include #include +#include #include #include +#include =20 #include #include @@ -127,9 +129,16 @@ module_param_named(same_filled_pages_enabled, zswap_= same_filled_pages_enabled, * data structures **********************************/ =20 +struct crypto_acomp_ctx { + struct crypto_acomp *acomp; + struct acomp_req *req; + struct crypto_wait wait; + struct mutex mutex; +}; + struct zswap_pool { struct zpool *zpool; - struct crypto_comp * __percpu *tfm; + struct crypto_acomp_ctx * __percpu *acomp_ctx; struct kref kref; struct list_head list; struct work_struct release_work; @@ -415,30 +424,59 @@ static int zswap_dstmem_dead(unsigned int cpu) static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *n= ode) { struct zswap_pool *pool =3D hlist_entry(node, struct zswap_pool, node); - struct crypto_comp *tfm; + struct crypto_acomp *acomp; + struct acomp_req *req; + struct crypto_acomp_ctx *acomp_ctx; =20 - if (WARN_ON(*per_cpu_ptr(pool->tfm, cpu))) + if (WARN_ON(*per_cpu_ptr(pool->acomp_ctx, cpu))) return 0; =20 - tfm =3D crypto_alloc_comp(pool->tfm_name, 0, 0); - if (IS_ERR_OR_NULL(tfm)) { - pr_err("could not alloc crypto comp %s : %ld\n", - pool->tfm_name, PTR_ERR(tfm)); + acomp_ctx =3D kzalloc(sizeof(*acomp_ctx), GFP_KERNEL); + if (IS_ERR_OR_NULL(acomp_ctx)) { + pr_err("Could not initialize acomp_ctx\n"); + return -ENOMEM; + } + acomp =3D crypto_alloc_acomp(pool->tfm_name, 0, 0); + if (IS_ERR_OR_NULL(acomp)) { + pr_err("could not alloc crypto acomp %s : %ld\n", + pool->tfm_name, PTR_ERR(acomp)); + return -ENOMEM; + } + acomp_ctx->acomp =3D acomp; + + req =3D acomp_request_alloc(acomp_ctx->acomp); + if (IS_ERR_OR_NULL(req)) { + pr_err("could not alloc crypto acomp %s : %ld\n", + pool->tfm_name, PTR_ERR(acomp)); return -ENOMEM; } - *per_cpu_ptr(pool->tfm, cpu) =3D tfm; + acomp_ctx->req =3D req; + + mutex_init(&acomp_ctx->mutex); + crypto_init_wait(&acomp_ctx->wait); + acomp_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG, + crypto_req_done, &acomp_ctx->wait); + + *per_cpu_ptr(pool->acomp_ctx, cpu) =3D acomp_ctx; + return 0; } =20 static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node= ) { struct zswap_pool *pool =3D hlist_entry(node, struct zswap_pool, node); - struct crypto_comp *tfm; + struct crypto_acomp_ctx *acomp_ctx; + + acomp_ctx =3D *per_cpu_ptr(pool->acomp_ctx, cpu); + if (!IS_ERR_OR_NULL(acomp_ctx)) { + if (!IS_ERR_OR_NULL(acomp_ctx->req)) + acomp_request_free(acomp_ctx->req); + if (!IS_ERR_OR_NULL(acomp_ctx->acomp)) + crypto_free_acomp(acomp_ctx->acomp); + kfree(acomp_ctx); + } + *per_cpu_ptr(pool->acomp_ctx, cpu) =3D NULL; =20 - tfm =3D *per_cpu_ptr(pool->tfm, cpu); - if (!IS_ERR_OR_NULL(tfm)) - crypto_free_comp(tfm); - *per_cpu_ptr(pool->tfm, cpu) =3D NULL; return 0; } =20 @@ -561,8 +599,9 @@ static struct zswap_pool *zswap_pool_create(char *typ= e, char *compressor) pr_debug("using %s zpool\n", zpool_get_type(pool->zpool)); =20 strlcpy(pool->tfm_name, compressor, sizeof(pool->tfm_name)); - pool->tfm =3D alloc_percpu(struct crypto_comp *); - if (!pool->tfm) { + + pool->acomp_ctx =3D alloc_percpu(struct crypto_acomp_ctx *); + if (!pool->acomp_ctx) { pr_err("percpu alloc failed\n"); goto error; } @@ -585,7 +624,7 @@ static struct zswap_pool *zswap_pool_create(char *typ= e, char *compressor) return pool; =20 error: - free_percpu(pool->tfm); + free_percpu(pool->acomp_ctx); if (pool->zpool) zpool_destroy_pool(pool->zpool); kfree(pool); @@ -596,14 +635,14 @@ static __init struct zswap_pool *__zswap_pool_creat= e_fallback(void) { bool has_comp, has_zpool; =20 - has_comp =3D crypto_has_comp(zswap_compressor, 0, 0); + has_comp =3D crypto_has_acomp(zswap_compressor, 0, 0); if (!has_comp && strcmp(zswap_compressor, CONFIG_ZSWAP_COMPRESSOR_DEFAULT)) { pr_err("compressor %s not available, using default %s\n", zswap_compressor, CONFIG_ZSWAP_COMPRESSOR_DEFAULT); param_free_charp(&zswap_compressor); zswap_compressor =3D CONFIG_ZSWAP_COMPRESSOR_DEFAULT; - has_comp =3D crypto_has_comp(zswap_compressor, 0, 0); + has_comp =3D crypto_has_acomp(zswap_compressor, 0, 0); } if (!has_comp) { pr_err("default compressor %s not available\n", @@ -639,7 +678,7 @@ static void zswap_pool_destroy(struct zswap_pool *poo= l) zswap_pool_debug("destroying", pool); =20 cpuhp_state_remove_instance(CPUHP_MM_ZSWP_POOL_PREPARE, &pool->node); - free_percpu(pool->tfm); + free_percpu(pool->acomp_ctx); zpool_destroy_pool(pool->zpool); kfree(pool); } @@ -723,7 +762,7 @@ static int __zswap_param_set(const char *val, const s= truct kernel_param *kp, } type =3D s; } else if (!compressor) { - if (!crypto_has_comp(s, 0, 0)) { + if (!crypto_has_acomp(s, 0, 0)) { pr_err("compressor %s not available\n", s); return -ENOENT; } @@ -774,7 +813,7 @@ static int __zswap_param_set(const char *val, const s= truct kernel_param *kp, * failed, maybe both compressor and zpool params were bad. * Allow changing this param, so pool creation will succeed * when the other param is changed. We already verified this - * param is ok in the zpool_has_pool() or crypto_has_comp() + * param is ok in the zpool_has_pool() or crypto_has_acomp() * checks above. */ ret =3D param_set_charp(s, kp); @@ -876,7 +915,9 @@ static int zswap_writeback_entry(struct zpool *pool, = unsigned long handle) pgoff_t offset; struct zswap_entry *entry; struct page *page; - struct crypto_comp *tfm; + struct scatterlist input, output; + struct crypto_acomp_ctx *acomp_ctx; + u8 *src, *dst; unsigned int dlen; int ret; @@ -916,14 +957,21 @@ static int zswap_writeback_entry(struct zpool *pool= , unsigned long handle) =20 case ZSWAP_SWAPCACHE_NEW: /* page is locked */ /* decompress */ + acomp_ctx =3D *this_cpu_ptr(entry->pool->acomp_ctx); + dlen =3D PAGE_SIZE; src =3D (u8 *)zhdr + sizeof(struct zswap_header); - dst =3D kmap_atomic(page); - tfm =3D *get_cpu_ptr(entry->pool->tfm); - ret =3D crypto_comp_decompress(tfm, src, entry->length, - dst, &dlen); - put_cpu_ptr(entry->pool->tfm); - kunmap_atomic(dst); + dst =3D kmap(page); + + mutex_lock(&acomp_ctx->mutex); + sg_init_one(&input, src, entry->length); + sg_init_one(&output, dst, dlen); + acomp_request_set_params(acomp_ctx->req, &input, &output, entry->lengt= h, dlen); + ret =3D crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acom= p_ctx->wait); + dlen =3D acomp_ctx->req->dlen; + mutex_unlock(&acomp_ctx->mutex); + + kunmap(page); BUG_ON(ret); BUG_ON(dlen !=3D PAGE_SIZE); =20 @@ -1004,7 +1052,8 @@ static int zswap_frontswap_store(unsigned type, pgo= ff_t offset, { struct zswap_tree *tree =3D zswap_trees[type]; struct zswap_entry *entry, *dupentry; - struct crypto_comp *tfm; + struct scatterlist input, output; + struct crypto_acomp_ctx *acomp_ctx; int ret; unsigned int hlen, dlen =3D PAGE_SIZE; unsigned long handle, value; @@ -1075,11 +1124,20 @@ static int zswap_frontswap_store(unsigned type, p= goff_t offset, =20 /* compress */ dst =3D get_cpu_var(zswap_dstmem); - tfm =3D *get_cpu_ptr(entry->pool->tfm); - src =3D kmap_atomic(page); - ret =3D crypto_comp_compress(tfm, src, PAGE_SIZE, dst, &dlen); - kunmap_atomic(src); - put_cpu_ptr(entry->pool->tfm); + acomp_ctx =3D *this_cpu_ptr(entry->pool->acomp_ctx); + put_cpu_var(zswap_dstmem); + + mutex_lock(&acomp_ctx->mutex); + + src =3D kmap(page); + sg_init_one(&input, src, PAGE_SIZE); + /* zswap_dstmem is of size (PAGE_SIZE * 2). Reflect same in sg_list */ + sg_init_one(&output, dst, PAGE_SIZE * 2); + acomp_request_set_params(acomp_ctx->req, &input, &output, PAGE_SIZE, dl= en); + ret =3D crypto_wait_req(crypto_acomp_compress(acomp_ctx->req), &acomp_c= tx->wait); + dlen =3D acomp_ctx->req->dlen; + kunmap(page); + if (ret) { ret =3D -EINVAL; goto put_dstmem; @@ -1103,7 +1161,7 @@ static int zswap_frontswap_store(unsigned type, pgo= ff_t offset, memcpy(buf, &zhdr, hlen); memcpy(buf + hlen, dst, dlen); zpool_unmap_handle(entry->pool->zpool, handle); - put_cpu_var(zswap_dstmem); + mutex_unlock(&acomp_ctx->mutex); =20 /* populate entry */ entry->offset =3D offset; @@ -1131,7 +1189,7 @@ static int zswap_frontswap_store(unsigned type, pgo= ff_t offset, return 0; =20 put_dstmem: - put_cpu_var(zswap_dstmem); + mutex_unlock(&acomp_ctx->mutex); zswap_pool_put(entry->pool); freepage: zswap_entry_cache_free(entry); @@ -1148,7 +1206,8 @@ static int zswap_frontswap_load(unsigned type, pgof= f_t offset, { struct zswap_tree *tree =3D zswap_trees[type]; struct zswap_entry *entry; - struct crypto_comp *tfm; + struct scatterlist input, output; + struct crypto_acomp_ctx *acomp_ctx; u8 *src, *dst; unsigned int dlen; int ret; @@ -1175,11 +1234,18 @@ static int zswap_frontswap_load(unsigned type, pg= off_t offset, src =3D zpool_map_handle(entry->pool->zpool, entry->handle, ZPOOL_MM_RO= ); if (zpool_evictable(entry->pool->zpool)) src +=3D sizeof(struct zswap_header); - dst =3D kmap_atomic(page); - tfm =3D *get_cpu_ptr(entry->pool->tfm); - ret =3D crypto_comp_decompress(tfm, src, entry->length, dst, &dlen); - put_cpu_ptr(entry->pool->tfm); - kunmap_atomic(dst); + dst =3D kmap(page); + + acomp_ctx =3D *this_cpu_ptr(entry->pool->acomp_ctx); + mutex_lock(&acomp_ctx->mutex); + sg_init_one(&input, src, entry->length); + sg_init_one(&output, dst, dlen); + acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length= , dlen); + ret =3D crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp= _ctx->wait); + dlen =3D acomp_ctx->req->dlen; + mutex_unlock(&acomp_ctx->mutex); + + kunmap(page); zpool_unmap_handle(entry->pool->zpool, entry->handle); BUG_ON(ret); =20 --=20 2.23.0