From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9670DC433F5 for ; Wed, 18 May 2022 06:33:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=fCK/oWwraCU84FEkUj+evvEKRsEhtYbBxrF/xfD5KcE=; b=R0GeQilpgcPxuW L+eJypSNGbqL8224BV+GODlBzYl1fYeZAp/TUvL6tOFl7gtBVY2n7kQfhAQuounuwvTF5sehNXXqS GHiATbuyGwphr/7o3e4PjZWrSL9xWCgJW6sj49HHNrkPAl+NNTDdg723Y1xuAOzOtwOSEJ4GPEA91 Z9BHSEs2JB/G1fT1nXp0tUr7PWevNBvdYBi74LQFGBZ3pcBw7XsNWxQA2hl3ZO6SlkWlcbW6lQRfd La2Wo8jxFACOSdJmyi2NA27cAUGGL/f3I4vo1V2ysFjNJMkrq2rQWars3rIid04M1tYauG87dxlMe Xs95wqfN+GuZTMTmHhqw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nrDE7-00HNM8-RF; Wed, 18 May 2022 06:32:08 +0000 Received: from esa8.hc1455-7.c3s2.iphmx.com ([139.138.61.253]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nrDDZ-00HN7r-NO for linux-arm-kernel@lists.infradead.org; Wed, 18 May 2022 06:31:36 +0000 X-IronPort-AV: E=McAfee;i="6400,9594,10350"; a="61156115" X-IronPort-AV: E=Sophos;i="5.91,234,1647270000"; d="scan'208";a="61156115" Received: from unknown (HELO yto-r1.gw.nic.fujitsu.com) ([218.44.52.217]) by esa8.hc1455-7.c3s2.iphmx.com with ESMTP; 18 May 2022 15:31:27 +0900 Received: from yto-m2.gw.nic.fujitsu.com (yto-nat-yto-m2.gw.nic.fujitsu.com [192.168.83.65]) by yto-r1.gw.nic.fujitsu.com (Postfix) with ESMTP id E6CE91066E for ; Wed, 18 May 2022 15:31:26 +0900 (JST) Received: from yto-om4.fujitsu.com (yto-om4.o.css.fujitsu.com [10.128.89.165]) by yto-m2.gw.nic.fujitsu.com (Postfix) with ESMTP id 0D0F7E6634 for ; Wed, 18 May 2022 15:31:26 +0900 (JST) Received: from cn-r05-10.example.com (n3235113.np.ts.nmh.cs.fujitsu.co.jp [10.123.235.113]) by yto-om4.fujitsu.com (Postfix) with ESMTP id 924EB4007E8A3; Wed, 18 May 2022 15:31:25 +0900 (JST) From: Kohei Tarumizu To: catalin.marinas@arm.com, will@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, gregkh@linuxfoundation.org, rafael@kernel.org, mchehab+huawei@kernel.org, eugenis@google.com, tony.luck@intel.com, pcc@google.com, peterz@infradead.org, marcos@orca.pet, conor.dooley@microchip.com, nicolas.ferre@microchip.com, marcan@marcan.st, linus.walleij@linaro.org, arnd@arndb.de, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: tarumizu.kohei@fujitsu.com Subject: [PATCH v4 3/8] soc: fujitsu: Add hardware prefetch control support for A64FX Date: Wed, 18 May 2022 15:30:27 +0900 Message-Id: <20220518063032.2377351-4-tarumizu.kohei@fujitsu.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220518063032.2377351-1-tarumizu.kohei@fujitsu.com> References: <20220518063032.2377351-1-tarumizu.kohei@fujitsu.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220517_233134_107893_5A00721F X-CRM114-Status: GOOD ( 25.53 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Adds module init/exit code to create sysfs attributes for A64FX with "stream_detect_prefetcher_enable", "stream_detect_prefetcher_strong" and "stream_detect_prefetcher_dist". This driver works only if part number is FUJITSU_CPU_PART_A64FX. The details of the registers to be read and written in this patch are described below. "https://github.com/fujitsu/A64FX/tree/master/doc/" A64FX_Specification_HPC_Extension_v1_EN.pdf Signed-off-by: Kohei Tarumizu --- drivers/soc/fujitsu/a64fx-pfctl.c | 373 ++++++++++++++++++++++++++++++ 1 file changed, 373 insertions(+) create mode 100644 drivers/soc/fujitsu/a64fx-pfctl.c diff --git a/drivers/soc/fujitsu/a64fx-pfctl.c b/drivers/soc/fujitsu/a64fx-pfctl.c new file mode 100644 index 000000000000..f765f53987d4 --- /dev/null +++ b/drivers/soc/fujitsu/a64fx-pfctl.c @@ -0,0 +1,373 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright 2022 FUJITSU LIMITED + * + * A64FX Hardware Prefetch Control support + */ + +#include +#include +#include +#include +#include +#include +#include + +/* + * Constants for these add the "A64FX_SDPF" prefix to the name described in + * section "1.3.4.2. IMP_PF_STREAM_DETECT_CTRL_EL0" of "A64FX specification". + * (https://github.com/fujitsu/A64FX/tree/master/doc/A64FX_Specification_HPC_Extension_v1_EN.pdf") + * See this document for register specification details. + */ +#define A64FX_SDPF_IMP_PF_STREAM_DETECT_CTRL_EL0 sys_reg(3, 3, 11, 4, 0) +#define A64FX_SDPF_V BIT_ULL(63) +#define A64FX_SDPF_L1PF_DIS BIT_ULL(59) +#define A64FX_SDPF_L2PF_DIS BIT_ULL(58) +#define A64FX_SDPF_L1W BIT_ULL(55) +#define A64FX_SDPF_L2W BIT_ULL(54) +#define A64FX_SDPF_L1_DIST GENMASK_ULL(27, 24) +#define A64FX_SDPF_L2_DIST GENMASK_ULL(19, 16) + +#define PFCTL_MIN_L1_DIST 256 +#define PFCTL_MIN_L2_DIST 1024 +#define PFCTL_DIST_AUTO_VAL 0 +#define PFCTL_STRONG_VAL 0 +#define PFCTL_WEAK_VAL 1 + +/* + * Define bitfield access macros for non-constant mask, because macros such as + * FIELD_FIT defined in include/linux/bitfield.h require constant mask. + */ +#define NC_FIELD_FIT(_mask, _val) \ + !((((typeof(_mask))_val) << __bf_shf(_mask)) & ~(_mask)) + +#define NC_FIELD_PREP(_mask, _val) \ + (((typeof(_mask))(_val) << __bf_shf(_mask)) & (_mask)) + +#define NC_FIELD_GET(_mask, _reg) \ + ((typeof(_mask))(((_reg) & (_mask)) >> __bf_shf(_mask))) + +struct a64fx_pfctl_attr { + struct device_attribute attr; + u64 mask; + void *data; +}; + +static const char strength_strong_string[] = "strong"; +static const char strength_weak_string[] = "weak"; +static const char dist_auto_string[] = "auto"; + +/* + * Returns the cpu number of the cpu_device(/sys/devices/system/cpu/cpuX) + * in the ancestor directory of prefetch_control. + * + * When initializing this driver, it is verified that the cache directory exists + * under cpuX device. Therefore, the third level up from prefetch_control is + * cpuX device as shown below. + * + * /sys/devices/system/cpu/cpuX/cache/indexX/prefetch_control + */ +static inline unsigned int pfctl_dev_get_cpu(struct device *pfctl_dev) +{ + return pfctl_dev->parent->parent->parent->id; +} + +/* Enable the value written to the A64FX_SDPF_IMP_PF_STREAM_DETECT_CTRL_EL0 */ +static inline void enable_verify(u64 *reg) +{ + *reg |= NC_FIELD_PREP(A64FX_SDPF_V, 1); +} + +static void _pfctl_read_mask(void *_reg) +{ + u64 *reg = (u64 *)_reg; + + *reg = read_sysreg_s(A64FX_SDPF_IMP_PF_STREAM_DETECT_CTRL_EL0); +} + +static u64 pfctl_read_mask(unsigned int cpu, u64 mask) +{ + u64 reg; + + smp_call_function_single(cpu, _pfctl_read_mask, ®, true); + + return NC_FIELD_GET(mask, reg); +} + +struct write_info { + u64 mask; + u64 val; +}; + +static void _pfctl_write_mask(void *info) +{ + struct write_info *winfo = info; + u64 reg; + + reg = read_sysreg_s(A64FX_SDPF_IMP_PF_STREAM_DETECT_CTRL_EL0); + + reg &= ~winfo->mask; + reg |= NC_FIELD_PREP(winfo->mask, winfo->val); + + enable_verify(®); + + write_sysreg_s(reg, A64FX_SDPF_IMP_PF_STREAM_DETECT_CTRL_EL0); +} + +static int pfctl_write_mask(unsigned int cpu, u64 mask, u64 val) +{ + struct write_info info = { + .mask = mask, + .val = val, + }; + + if (!NC_FIELD_FIT(mask, val)) + return -EINVAL; + + smp_call_function_single(cpu, _pfctl_write_mask, &info, true); + + return 0; +} + +static ssize_t +pfctl_enable_show(struct device *pfctl_dev, struct device_attribute *attr, + char *buf) +{ + unsigned int cpu = pfctl_dev_get_cpu(pfctl_dev); + struct a64fx_pfctl_attr *aa; + bool val; + + aa = container_of(attr, struct a64fx_pfctl_attr, attr); + + val = (bool)pfctl_read_mask(cpu, aa->mask); + + return sysfs_emit(buf, "%d\n", val ? 0 : 1); +} + +static ssize_t +pfctl_enable_store(struct device *pfctl_dev, struct device_attribute *attr, + const char *buf, size_t size) +{ + unsigned int cpu = pfctl_dev_get_cpu(pfctl_dev); + struct a64fx_pfctl_attr *aa; + bool val; + int ret; + + aa = container_of(attr, struct a64fx_pfctl_attr, attr); + + if (strtobool(buf, &val) < 0) + return -EINVAL; + + ret = pfctl_write_mask(cpu, aa->mask, val ? 0 : 1); + if (ret < 0) + return ret; + + return size; +} + +#define A64FX_PFCTL_ENABLE_ATTR(_level, _mask) \ + struct a64fx_pfctl_attr attr_l##_level##_enable = { \ + .attr = __ATTR(stream_detect_prefetcher_enable, 0600, \ + pfctl_enable_show, pfctl_enable_store), \ + .mask = _mask, } + + +static A64FX_PFCTL_ENABLE_ATTR(1, A64FX_SDPF_L1PF_DIS); +static A64FX_PFCTL_ENABLE_ATTR(2, A64FX_SDPF_L2PF_DIS); + +static ssize_t +pfctl_dist_show(struct device *pfctl_dev, struct device_attribute *attr, + char *buf) +{ + unsigned int cpu = pfctl_dev_get_cpu(pfctl_dev); + struct a64fx_pfctl_attr *aa; + u64 val, min_dist; + + aa = container_of(attr, struct a64fx_pfctl_attr, attr); + min_dist = (u64)aa->data; + + val = pfctl_read_mask(cpu, aa->mask); + + if (val == PFCTL_DIST_AUTO_VAL) + return sysfs_emit(buf, "%s\n", dist_auto_string); + else + return sysfs_emit(buf, "%llu\n", val * min_dist); +} + +static ssize_t +pfctl_dist_store(struct device *pfctl_dev, struct device_attribute *attr, + const char *buf, size_t size) +{ + unsigned int cpu = pfctl_dev_get_cpu(pfctl_dev); + struct a64fx_pfctl_attr *aa; + u64 min_dist, val; + int ret; + + aa = container_of(attr, struct a64fx_pfctl_attr, attr); + min_dist = (u64)aa->data; + + if (sysfs_streq(buf, dist_auto_string)) { + val = PFCTL_DIST_AUTO_VAL; + } else { + ret = kstrtoull(buf, 10, &val); + if (ret < 0 || val < 1) + return -EINVAL; + } + + val = roundup(val, min_dist) / min_dist; + + if (!NC_FIELD_FIT(aa->mask, val)) + return -EINVAL; + + ret = pfctl_write_mask(cpu, aa->mask, val); + if (ret < 0) + return ret; + + return size; +} + +#define PFCTL_DIST_ATTR(_level, _mask, _min_dist) \ + struct a64fx_pfctl_attr attr_l##_level##_dist = { \ + .attr = __ATTR(stream_detect_prefetcher_dist, 0600, \ + pfctl_dist_show, pfctl_dist_store), \ + .mask = _mask, \ + .data = (void *)(u64)_min_dist, } + +static PFCTL_DIST_ATTR(1, A64FX_SDPF_L1_DIST, PFCTL_MIN_L1_DIST); +static PFCTL_DIST_ATTR(2, A64FX_SDPF_L2_DIST, PFCTL_MIN_L2_DIST); + +static ssize_t +pfctl_strength_show(struct device *pfctl_dev, struct device_attribute *attr, + char *buf) +{ + unsigned int cpu = pfctl_dev_get_cpu(pfctl_dev); + struct a64fx_pfctl_attr *aa; + u64 val; + + aa = container_of(attr, struct a64fx_pfctl_attr, attr); + + val = pfctl_read_mask(cpu, aa->mask); + + switch (val) { + case PFCTL_STRONG_VAL: + return sysfs_emit(buf, "%s\n", strength_strong_string); + case PFCTL_WEAK_VAL: + return sysfs_emit(buf, "%s\n", strength_weak_string); + default: + return -EINVAL; + } +} + +static ssize_t +pfctl_strength_store(struct device *pfctl_dev, struct device_attribute *attr, + const char *buf, size_t size) +{ + unsigned int cpu = pfctl_dev_get_cpu(pfctl_dev); + struct a64fx_pfctl_attr *aa; + u64 val; + int ret; + + aa = container_of(attr, struct a64fx_pfctl_attr, attr); + + if (sysfs_streq(buf, strength_strong_string)) + val = PFCTL_STRONG_VAL; + else if (sysfs_streq(buf, strength_weak_string)) + val = PFCTL_WEAK_VAL; + else + return -EINVAL; + + ret = pfctl_write_mask(cpu, aa->mask, val); + if (ret < 0) + return ret; + + return size; +} + +#define PFCTL_STRENGTH_ATTR(_level, _mask) \ + struct a64fx_pfctl_attr attr_l##_level##_strength = { \ + .attr = __ATTR(stream_detect_prefetcher_strength, 0600, \ + pfctl_strength_show, \ + pfctl_strength_store), \ + .mask = _mask, } + +static PFCTL_STRENGTH_ATTR(1, A64FX_SDPF_L1W); +static PFCTL_STRENGTH_ATTR(2, A64FX_SDPF_L2W); + +static ssize_t +pfctl_strength_available_show(struct device *dev, struct device_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%s %s\n", strength_strong_string, + strength_weak_string); +} + +/* + * A64FX has same kind of available strength for any caches, so define only one + * attribute. + */ +struct a64fx_pfctl_attr attr_strength_available = { + .attr = __ATTR(stream_detect_prefetcher_strength_available, 0400, + pfctl_strength_available_show, NULL), }; + +static struct device_attribute *l1_attrs[] __ro_after_init = { + &attr_l1_enable.attr, + &attr_l1_dist.attr, + &attr_l1_strength.attr, + &attr_strength_available.attr, + NULL, +}; + +static struct device_attribute *l2_attrs[] __ro_after_init = { + &attr_l2_enable.attr, + &attr_l2_dist.attr, + &attr_l2_strength.attr, + &attr_strength_available.attr, + NULL, +}; + +static const struct pfctl_group a64fx_pfctl_groups[] = { + { + .level = 1, + .type = CACHE_TYPE_DATA, + .attrs = l1_attrs, + }, + { + .level = 2, + .type = CACHE_TYPE_UNIFIED, + .attrs = l2_attrs, + }, + { + .attrs = NULL, + }, +}; + +/* + * This driver returns a negative value if it does not support the Hardware + * Prefetch Control or if it is running on a VM guest. + */ +static int __init a64fx_pfctl_init(void) +{ + unsigned long implementor = read_cpuid_implementor(); + unsigned long part_number = read_cpuid_part_number(); + + if (!is_kernel_in_hyp_mode()) + return -EINVAL; + + if ((implementor != ARM_CPU_IMP_FUJITSU) || + (part_number != FUJITSU_CPU_PART_A64FX)) + return -ENODEV; + + return pfctl_register_attrs(a64fx_pfctl_groups); +} + +static void __exit a64fx_pfctl_exit(void) +{ + return pfctl_unregister_attrs(a64fx_pfctl_groups); +} + +late_initcall(a64fx_pfctl_init); +module_exit(a64fx_pfctl_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("FUJITSU LIMITED"); +MODULE_DESCRIPTION("A64FX Hardware Prefetch Control Driver"); -- 2.27.0 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0AFDEC433F5 for ; Wed, 18 May 2022 06:34:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231352AbiERGeP (ORCPT ); Wed, 18 May 2022 02:34:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60136 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231436AbiERGbb (ORCPT ); Wed, 18 May 2022 02:31:31 -0400 Received: from esa9.hc1455-7.c3s2.iphmx.com (esa9.hc1455-7.c3s2.iphmx.com [139.138.36.223]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 16754A473 for ; Tue, 17 May 2022 23:31:29 -0700 (PDT) X-IronPort-AV: E=McAfee;i="6400,9594,10350"; a="61256555" X-IronPort-AV: E=Sophos;i="5.91,234,1647270000"; d="scan'208";a="61256555" Received: from unknown (HELO yto-r1.gw.nic.fujitsu.com) ([218.44.52.217]) by esa9.hc1455-7.c3s2.iphmx.com with ESMTP; 18 May 2022 15:31:27 +0900 Received: from yto-m4.gw.nic.fujitsu.com (yto-nat-yto-m4.gw.nic.fujitsu.com [192.168.83.67]) by yto-r1.gw.nic.fujitsu.com (Postfix) with ESMTP id D0D32D687B for ; Wed, 18 May 2022 15:31:26 +0900 (JST) Received: from yto-om4.fujitsu.com (yto-om4.o.css.fujitsu.com [10.128.89.165]) by yto-m4.gw.nic.fujitsu.com (Postfix) with ESMTP id 0D550F0FE5 for ; Wed, 18 May 2022 15:31:26 +0900 (JST) Received: from cn-r05-10.example.com (n3235113.np.ts.nmh.cs.fujitsu.co.jp [10.123.235.113]) by yto-om4.fujitsu.com (Postfix) with ESMTP id 924EB4007E8A3; Wed, 18 May 2022 15:31:25 +0900 (JST) From: Kohei Tarumizu To: catalin.marinas@arm.com, will@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, gregkh@linuxfoundation.org, rafael@kernel.org, mchehab+huawei@kernel.org, eugenis@google.com, tony.luck@intel.com, pcc@google.com, peterz@infradead.org, marcos@orca.pet, conor.dooley@microchip.com, nicolas.ferre@microchip.com, marcan@marcan.st, linus.walleij@linaro.org, arnd@arndb.de, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: tarumizu.kohei@fujitsu.com Subject: [PATCH v4 3/8] soc: fujitsu: Add hardware prefetch control support for A64FX Date: Wed, 18 May 2022 15:30:27 +0900 Message-Id: <20220518063032.2377351-4-tarumizu.kohei@fujitsu.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220518063032.2377351-1-tarumizu.kohei@fujitsu.com> References: <20220518063032.2377351-1-tarumizu.kohei@fujitsu.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Adds module init/exit code to create sysfs attributes for A64FX with "stream_detect_prefetcher_enable", "stream_detect_prefetcher_strong" and "stream_detect_prefetcher_dist". This driver works only if part number is FUJITSU_CPU_PART_A64FX. The details of the registers to be read and written in this patch are described below. "https://github.com/fujitsu/A64FX/tree/master/doc/" A64FX_Specification_HPC_Extension_v1_EN.pdf Signed-off-by: Kohei Tarumizu --- drivers/soc/fujitsu/a64fx-pfctl.c | 373 ++++++++++++++++++++++++++++++ 1 file changed, 373 insertions(+) create mode 100644 drivers/soc/fujitsu/a64fx-pfctl.c diff --git a/drivers/soc/fujitsu/a64fx-pfctl.c b/drivers/soc/fujitsu/a64fx-pfctl.c new file mode 100644 index 000000000000..f765f53987d4 --- /dev/null +++ b/drivers/soc/fujitsu/a64fx-pfctl.c @@ -0,0 +1,373 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright 2022 FUJITSU LIMITED + * + * A64FX Hardware Prefetch Control support + */ + +#include +#include +#include +#include +#include +#include +#include + +/* + * Constants for these add the "A64FX_SDPF" prefix to the name described in + * section "1.3.4.2. IMP_PF_STREAM_DETECT_CTRL_EL0" of "A64FX specification". + * (https://github.com/fujitsu/A64FX/tree/master/doc/A64FX_Specification_HPC_Extension_v1_EN.pdf") + * See this document for register specification details. + */ +#define A64FX_SDPF_IMP_PF_STREAM_DETECT_CTRL_EL0 sys_reg(3, 3, 11, 4, 0) +#define A64FX_SDPF_V BIT_ULL(63) +#define A64FX_SDPF_L1PF_DIS BIT_ULL(59) +#define A64FX_SDPF_L2PF_DIS BIT_ULL(58) +#define A64FX_SDPF_L1W BIT_ULL(55) +#define A64FX_SDPF_L2W BIT_ULL(54) +#define A64FX_SDPF_L1_DIST GENMASK_ULL(27, 24) +#define A64FX_SDPF_L2_DIST GENMASK_ULL(19, 16) + +#define PFCTL_MIN_L1_DIST 256 +#define PFCTL_MIN_L2_DIST 1024 +#define PFCTL_DIST_AUTO_VAL 0 +#define PFCTL_STRONG_VAL 0 +#define PFCTL_WEAK_VAL 1 + +/* + * Define bitfield access macros for non-constant mask, because macros such as + * FIELD_FIT defined in include/linux/bitfield.h require constant mask. + */ +#define NC_FIELD_FIT(_mask, _val) \ + !((((typeof(_mask))_val) << __bf_shf(_mask)) & ~(_mask)) + +#define NC_FIELD_PREP(_mask, _val) \ + (((typeof(_mask))(_val) << __bf_shf(_mask)) & (_mask)) + +#define NC_FIELD_GET(_mask, _reg) \ + ((typeof(_mask))(((_reg) & (_mask)) >> __bf_shf(_mask))) + +struct a64fx_pfctl_attr { + struct device_attribute attr; + u64 mask; + void *data; +}; + +static const char strength_strong_string[] = "strong"; +static const char strength_weak_string[] = "weak"; +static const char dist_auto_string[] = "auto"; + +/* + * Returns the cpu number of the cpu_device(/sys/devices/system/cpu/cpuX) + * in the ancestor directory of prefetch_control. + * + * When initializing this driver, it is verified that the cache directory exists + * under cpuX device. Therefore, the third level up from prefetch_control is + * cpuX device as shown below. + * + * /sys/devices/system/cpu/cpuX/cache/indexX/prefetch_control + */ +static inline unsigned int pfctl_dev_get_cpu(struct device *pfctl_dev) +{ + return pfctl_dev->parent->parent->parent->id; +} + +/* Enable the value written to the A64FX_SDPF_IMP_PF_STREAM_DETECT_CTRL_EL0 */ +static inline void enable_verify(u64 *reg) +{ + *reg |= NC_FIELD_PREP(A64FX_SDPF_V, 1); +} + +static void _pfctl_read_mask(void *_reg) +{ + u64 *reg = (u64 *)_reg; + + *reg = read_sysreg_s(A64FX_SDPF_IMP_PF_STREAM_DETECT_CTRL_EL0); +} + +static u64 pfctl_read_mask(unsigned int cpu, u64 mask) +{ + u64 reg; + + smp_call_function_single(cpu, _pfctl_read_mask, ®, true); + + return NC_FIELD_GET(mask, reg); +} + +struct write_info { + u64 mask; + u64 val; +}; + +static void _pfctl_write_mask(void *info) +{ + struct write_info *winfo = info; + u64 reg; + + reg = read_sysreg_s(A64FX_SDPF_IMP_PF_STREAM_DETECT_CTRL_EL0); + + reg &= ~winfo->mask; + reg |= NC_FIELD_PREP(winfo->mask, winfo->val); + + enable_verify(®); + + write_sysreg_s(reg, A64FX_SDPF_IMP_PF_STREAM_DETECT_CTRL_EL0); +} + +static int pfctl_write_mask(unsigned int cpu, u64 mask, u64 val) +{ + struct write_info info = { + .mask = mask, + .val = val, + }; + + if (!NC_FIELD_FIT(mask, val)) + return -EINVAL; + + smp_call_function_single(cpu, _pfctl_write_mask, &info, true); + + return 0; +} + +static ssize_t +pfctl_enable_show(struct device *pfctl_dev, struct device_attribute *attr, + char *buf) +{ + unsigned int cpu = pfctl_dev_get_cpu(pfctl_dev); + struct a64fx_pfctl_attr *aa; + bool val; + + aa = container_of(attr, struct a64fx_pfctl_attr, attr); + + val = (bool)pfctl_read_mask(cpu, aa->mask); + + return sysfs_emit(buf, "%d\n", val ? 0 : 1); +} + +static ssize_t +pfctl_enable_store(struct device *pfctl_dev, struct device_attribute *attr, + const char *buf, size_t size) +{ + unsigned int cpu = pfctl_dev_get_cpu(pfctl_dev); + struct a64fx_pfctl_attr *aa; + bool val; + int ret; + + aa = container_of(attr, struct a64fx_pfctl_attr, attr); + + if (strtobool(buf, &val) < 0) + return -EINVAL; + + ret = pfctl_write_mask(cpu, aa->mask, val ? 0 : 1); + if (ret < 0) + return ret; + + return size; +} + +#define A64FX_PFCTL_ENABLE_ATTR(_level, _mask) \ + struct a64fx_pfctl_attr attr_l##_level##_enable = { \ + .attr = __ATTR(stream_detect_prefetcher_enable, 0600, \ + pfctl_enable_show, pfctl_enable_store), \ + .mask = _mask, } + + +static A64FX_PFCTL_ENABLE_ATTR(1, A64FX_SDPF_L1PF_DIS); +static A64FX_PFCTL_ENABLE_ATTR(2, A64FX_SDPF_L2PF_DIS); + +static ssize_t +pfctl_dist_show(struct device *pfctl_dev, struct device_attribute *attr, + char *buf) +{ + unsigned int cpu = pfctl_dev_get_cpu(pfctl_dev); + struct a64fx_pfctl_attr *aa; + u64 val, min_dist; + + aa = container_of(attr, struct a64fx_pfctl_attr, attr); + min_dist = (u64)aa->data; + + val = pfctl_read_mask(cpu, aa->mask); + + if (val == PFCTL_DIST_AUTO_VAL) + return sysfs_emit(buf, "%s\n", dist_auto_string); + else + return sysfs_emit(buf, "%llu\n", val * min_dist); +} + +static ssize_t +pfctl_dist_store(struct device *pfctl_dev, struct device_attribute *attr, + const char *buf, size_t size) +{ + unsigned int cpu = pfctl_dev_get_cpu(pfctl_dev); + struct a64fx_pfctl_attr *aa; + u64 min_dist, val; + int ret; + + aa = container_of(attr, struct a64fx_pfctl_attr, attr); + min_dist = (u64)aa->data; + + if (sysfs_streq(buf, dist_auto_string)) { + val = PFCTL_DIST_AUTO_VAL; + } else { + ret = kstrtoull(buf, 10, &val); + if (ret < 0 || val < 1) + return -EINVAL; + } + + val = roundup(val, min_dist) / min_dist; + + if (!NC_FIELD_FIT(aa->mask, val)) + return -EINVAL; + + ret = pfctl_write_mask(cpu, aa->mask, val); + if (ret < 0) + return ret; + + return size; +} + +#define PFCTL_DIST_ATTR(_level, _mask, _min_dist) \ + struct a64fx_pfctl_attr attr_l##_level##_dist = { \ + .attr = __ATTR(stream_detect_prefetcher_dist, 0600, \ + pfctl_dist_show, pfctl_dist_store), \ + .mask = _mask, \ + .data = (void *)(u64)_min_dist, } + +static PFCTL_DIST_ATTR(1, A64FX_SDPF_L1_DIST, PFCTL_MIN_L1_DIST); +static PFCTL_DIST_ATTR(2, A64FX_SDPF_L2_DIST, PFCTL_MIN_L2_DIST); + +static ssize_t +pfctl_strength_show(struct device *pfctl_dev, struct device_attribute *attr, + char *buf) +{ + unsigned int cpu = pfctl_dev_get_cpu(pfctl_dev); + struct a64fx_pfctl_attr *aa; + u64 val; + + aa = container_of(attr, struct a64fx_pfctl_attr, attr); + + val = pfctl_read_mask(cpu, aa->mask); + + switch (val) { + case PFCTL_STRONG_VAL: + return sysfs_emit(buf, "%s\n", strength_strong_string); + case PFCTL_WEAK_VAL: + return sysfs_emit(buf, "%s\n", strength_weak_string); + default: + return -EINVAL; + } +} + +static ssize_t +pfctl_strength_store(struct device *pfctl_dev, struct device_attribute *attr, + const char *buf, size_t size) +{ + unsigned int cpu = pfctl_dev_get_cpu(pfctl_dev); + struct a64fx_pfctl_attr *aa; + u64 val; + int ret; + + aa = container_of(attr, struct a64fx_pfctl_attr, attr); + + if (sysfs_streq(buf, strength_strong_string)) + val = PFCTL_STRONG_VAL; + else if (sysfs_streq(buf, strength_weak_string)) + val = PFCTL_WEAK_VAL; + else + return -EINVAL; + + ret = pfctl_write_mask(cpu, aa->mask, val); + if (ret < 0) + return ret; + + return size; +} + +#define PFCTL_STRENGTH_ATTR(_level, _mask) \ + struct a64fx_pfctl_attr attr_l##_level##_strength = { \ + .attr = __ATTR(stream_detect_prefetcher_strength, 0600, \ + pfctl_strength_show, \ + pfctl_strength_store), \ + .mask = _mask, } + +static PFCTL_STRENGTH_ATTR(1, A64FX_SDPF_L1W); +static PFCTL_STRENGTH_ATTR(2, A64FX_SDPF_L2W); + +static ssize_t +pfctl_strength_available_show(struct device *dev, struct device_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%s %s\n", strength_strong_string, + strength_weak_string); +} + +/* + * A64FX has same kind of available strength for any caches, so define only one + * attribute. + */ +struct a64fx_pfctl_attr attr_strength_available = { + .attr = __ATTR(stream_detect_prefetcher_strength_available, 0400, + pfctl_strength_available_show, NULL), }; + +static struct device_attribute *l1_attrs[] __ro_after_init = { + &attr_l1_enable.attr, + &attr_l1_dist.attr, + &attr_l1_strength.attr, + &attr_strength_available.attr, + NULL, +}; + +static struct device_attribute *l2_attrs[] __ro_after_init = { + &attr_l2_enable.attr, + &attr_l2_dist.attr, + &attr_l2_strength.attr, + &attr_strength_available.attr, + NULL, +}; + +static const struct pfctl_group a64fx_pfctl_groups[] = { + { + .level = 1, + .type = CACHE_TYPE_DATA, + .attrs = l1_attrs, + }, + { + .level = 2, + .type = CACHE_TYPE_UNIFIED, + .attrs = l2_attrs, + }, + { + .attrs = NULL, + }, +}; + +/* + * This driver returns a negative value if it does not support the Hardware + * Prefetch Control or if it is running on a VM guest. + */ +static int __init a64fx_pfctl_init(void) +{ + unsigned long implementor = read_cpuid_implementor(); + unsigned long part_number = read_cpuid_part_number(); + + if (!is_kernel_in_hyp_mode()) + return -EINVAL; + + if ((implementor != ARM_CPU_IMP_FUJITSU) || + (part_number != FUJITSU_CPU_PART_A64FX)) + return -ENODEV; + + return pfctl_register_attrs(a64fx_pfctl_groups); +} + +static void __exit a64fx_pfctl_exit(void) +{ + return pfctl_unregister_attrs(a64fx_pfctl_groups); +} + +late_initcall(a64fx_pfctl_init); +module_exit(a64fx_pfctl_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("FUJITSU LIMITED"); +MODULE_DESCRIPTION("A64FX Hardware Prefetch Control Driver"); -- 2.27.0