From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-29.0 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4572FC07E99 for ; Fri, 9 Jul 2021 18:58:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 23BF7613BE for ; Fri, 9 Jul 2021 18:58:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229606AbhGITBK (ORCPT ); Fri, 9 Jul 2021 15:01:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36816 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229459AbhGITBJ (ORCPT ); Fri, 9 Jul 2021 15:01:09 -0400 Received: from mail-qk1-x72c.google.com (mail-qk1-x72c.google.com [IPv6:2607:f8b0:4864:20::72c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A2455C0613DD for ; Fri, 9 Jul 2021 11:58:24 -0700 (PDT) Received: by mail-qk1-x72c.google.com with SMTP id e14so10324683qkl.9 for ; Fri, 09 Jul 2021 11:58:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=1Rq09e4FzKVmC8coozmBf2Y4P9tsiDVwZ27Wjq/A92M=; b=DXZ7xOOrb+im+tg33i8mbTwsKlZOJPAfSLVS1lRsLDqp2+R0vdcuS+0zSmf1hozWid h8N38TsVQ1QPBClHpe5emLu45mi3w5tBd5vkYjiI6beSJjrXTQS9Lis5m+m60gBYrsC3 Jtaz1JParBBvM4YHj99L6tC2DD0jJWHj3QIQZ+3nIZDUcW68SbAYvmvI8sPsQ8rrBf9m DIbU5xPKHNZtJodbP2/6ps301q+e0hoyKM6Fse04usXf1IDGbmbscr/M14PPeovJ8vNW Z0jX0S9BplBgXLHzIJNTXSkc81ZwJBu7AYVTyYRZxK2MfH9BfbKUzQx/G8a7Ik/pB8aU o1CA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=1Rq09e4FzKVmC8coozmBf2Y4P9tsiDVwZ27Wjq/A92M=; b=JLZ4BoZK8Qt1VrceCKEKZ6WIkZsfO5z2qm1NYdNC7oD2LDCzUjvzPbrDQMQqQRbo/0 3viOtrU66g+J+mHinDVa0sZxIS3VG5r7ONSikwkcUKyF9Yy+6ufOlwM/H7q//Y9AJIEB dgPHFvRhLeEjq2jlg20bwmBAkUjN97YOrTkxY06tTGU0C8gYNLmRZnDvhJWNZz4TISxx 6qzB3hKAp5IWAeuaHtSxUI+HL8TEbBp3VSBC1XgN4BwEZNZSN3Su98aJhJCA9Pu+Q1j3 IvjyXHixZ/x85ngTde/TJBhyTAhqTqL8hHJUD5yD/6GFN++uesla+j10sm9sX8LPCzor 01hg== X-Gm-Message-State: AOAM5333Y20Mk4fDsasRbfWv7H6h8cyf1ImMdS6Nx1F365z9Nqnce6xc GDhisUbOS2ZLuu9X6JVabJPdYy+cIoVJJQyDhXjQfw== X-Google-Smtp-Source: ABdhPJx38EbV1FTTGhhuDjA8CTzpk9q6mzakZBUDCGz1zQ1TzLXHZcfuZU3Lo65MlCEwO1gJjRbol4iuazm9D4flAsc= X-Received: by 2002:ae9:dd43:: with SMTP id r64mr38651885qkf.216.1625857103581; Fri, 09 Jul 2021 11:58:23 -0700 (PDT) MIME-Version: 1.0 References: <20210709043713.887098-1-senozhatsky@chromium.org> <20210709043713.887098-3-senozhatsky@chromium.org> In-Reply-To: <20210709043713.887098-3-senozhatsky@chromium.org> From: Joel Fernandes Date: Fri, 9 Jul 2021 14:58:11 -0400 Message-ID: Subject: Re: [PATCHv2 2/4] arm64: add guest pvstate support To: Sergey Senozhatsky Cc: Marc Zyngier , Will Deacon , Suleiman Souhlal , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi, Just few nits, patch itself LGTM: On Fri, Jul 9, 2021 at 12:37 AM Sergey Senozhatsky wrote: > > PV-vcpu-state is a per-CPU struct, which, for the time being, > holds boolean `preempted' vCPU state. During the startup, > given that host supports PV-state, each guest vCPU sends > a pointer to its per-CPU variable to the host as a payload > with the SMCCC HV call, so that host can update vCPU state > when it puts or loads vCPU. > > This has impact on the guest's scheduler: > > [..] > wake_up_process() > try_to_wake_up() > select_task_rq_fair() > available_idle_cpu() > vcpu_is_preempted() > > Some sched benchmarks data is available on the github page [0]. > > [0] https://github.com/sergey-senozhatsky/arm64-vcpu_is_preempted > > Signed-off-by: Sergey Senozhatsky > --- > arch/arm64/include/asm/paravirt.h | 19 +++++++ > arch/arm64/kernel/paravirt.c | 94 +++++++++++++++++++++++++++++++ > arch/arm64/kernel/smp.c | 4 ++ > 3 files changed, 117 insertions(+) > > diff --git a/arch/arm64/include/asm/paravirt.h b/arch/arm64/include/asm/paravirt.h > index 9aa193e0e8f2..a3f7665dff38 100644 > --- a/arch/arm64/include/asm/paravirt.h > +++ b/arch/arm64/include/asm/paravirt.h > @@ -2,6 +2,11 @@ > #ifndef _ASM_ARM64_PARAVIRT_H > #define _ASM_ARM64_PARAVIRT_H > > +struct vcpu_state { > + bool preempted; > + u8 reserved[63]; > +}; > + > #ifdef CONFIG_PARAVIRT > #include > > @@ -20,8 +25,22 @@ static inline u64 paravirt_steal_clock(int cpu) > > int __init pv_time_init(void); > > +bool dummy_vcpu_is_preempted(unsigned int cpu); > + > +extern struct static_key pv_vcpu_is_preempted_enabled;. pv_vcpu_is_preempted_enabled static_key is not used in any patch. Maybe it is stale? > +DECLARE_STATIC_CALL(pv_vcpu_is_preempted, dummy_vcpu_is_preempted); > + > +static inline bool paravirt_vcpu_is_preempted(unsigned int cpu) > +{ > + return static_call(pv_vcpu_is_preempted)(cpu); > +} > + > +int __init pv_vcpu_state_init(void); > + > #else > > +#define pv_vcpu_state_init() do {} while (0) > + > #define pv_time_init() do {} while (0) > > #endif // CONFIG_PARAVIRT > diff --git a/arch/arm64/kernel/paravirt.c b/arch/arm64/kernel/paravirt.c > index 75fed4460407..d8fc46795d94 100644 > --- a/arch/arm64/kernel/paravirt.c > +++ b/arch/arm64/kernel/paravirt.c > @@ -40,6 +40,11 @@ struct pv_time_stolen_time_region { > > static DEFINE_PER_CPU(struct pv_time_stolen_time_region, stolen_time_region); > > +static DEFINE_PER_CPU(struct vcpu_state, vcpus_states); > +struct static_key pv_vcpu_is_preempted_enabled; > + > +DEFINE_STATIC_CALL(pv_vcpu_is_preempted, dummy_vcpu_is_preempted); Could we use DEFINE_STATIC_CALL_NULL and get rid of the dummy function? I believe that makes the function trampoline as return instruction, till it is updated. > + > static bool steal_acc = true; > static int __init parse_no_stealacc(char *arg) > { > @@ -165,3 +170,92 @@ int __init pv_time_init(void) > > return 0; > } > + > +bool dummy_vcpu_is_preempted(unsigned int cpu) > +{ > + return false; > +} > + > +static bool __vcpu_is_preempted(unsigned int cpu) > +{ > + struct vcpu_state *st; > + > + st = &per_cpu(vcpus_states, cpu); > + return READ_ONCE(st->preempted); I guess you could just do: { return READ_ONCE(per_cpu(vcpus_states, cpu).preempted); } > +} > + > +static bool has_pv_vcpu_state(void) > +{ > + struct arm_smccc_res res; > + > + /* To detect the presence of PV time support we require SMCCC 1.1+ */ > + if (arm_smccc_1_1_get_conduit() == SMCCC_CONDUIT_NONE) > + return false; > + > + arm_smccc_1_1_invoke(ARM_SMCCC_ARCH_FEATURES_FUNC_ID, > + ARM_SMCCC_HV_PV_VCPU_STATE_FEATURES, > + &res); > + > + if (res.a0 != SMCCC_RET_SUCCESS) > + return false; > + return true; > +} > + > +static int __pv_vcpu_state_hook(unsigned int cpu, int event) > +{ > + struct arm_smccc_res res; > + struct vcpu_state *st; > + > + st = &per_cpu(vcpus_states, cpu); > + arm_smccc_1_1_invoke(event, virt_to_phys(st), &res); > + if (res.a0 != SMCCC_RET_SUCCESS) > + return -EINVAL; > + return 0; > +} > + > +static int vcpu_state_init(unsigned int cpu) > +{ > + int ret = __pv_vcpu_state_hook(cpu, ARM_SMCCC_HV_PV_VCPU_STATE_INIT); > + > + if (ret) > + pr_warn("Unable to ARM_SMCCC_HV_PV_STATE_INIT\n"); > + return ret; > +} > + > +static int vcpu_state_release(unsigned int cpu) > +{ > + int ret = __pv_vcpu_state_hook(cpu, ARM_SMCCC_HV_PV_VCPU_STATE_RELEASE); > + > + if (ret) > + pr_warn("Unable to ARM_SMCCC_HV_PV_STATE_RELEASE\n"); > + return ret; > +} > + > +static int pv_vcpu_state_register_hooks(void) > +{ > + int ret; > + > + ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, > + "hypervisor/arm/pvstate:starting", > + vcpu_state_init, > + vcpu_state_release); > + if (ret < 0) > + pr_warn("Failed to register CPU hooks\n"); > + return 0; > +} > + > +int __init pv_vcpu_state_init(void) > +{ > + int ret; > + > + if (!has_pv_vcpu_state()) > + return 0; > + > + ret = pv_vcpu_state_register_hooks(); > + if (ret) > + return ret; > + > + static_call_update(pv_vcpu_is_preempted, __vcpu_is_preempted); > + static_key_slow_inc(&pv_vcpu_is_preempted_enabled); I think this static key inc is also stale. thanks, -Joel From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 277D0C07E95 for ; Sat, 10 Jul 2021 10:18:43 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id 8B5EC613C3 for ; Sat, 10 Jul 2021 10:18:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8B5EC613C3 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvmarm-bounces@lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id E3C0F4086D; Sat, 10 Jul 2021 06:18:41 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@google.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MH26xMhkm18Q; Sat, 10 Jul 2021 06:18:40 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 99CB240C88; Sat, 10 Jul 2021 06:18:40 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 8EC7940870 for ; Fri, 9 Jul 2021 14:58:25 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qsxO6u5+Nrl4 for ; Fri, 9 Jul 2021 14:58:24 -0400 (EDT) Received: from mail-qk1-f181.google.com (mail-qk1-f181.google.com [209.85.222.181]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 474774086F for ; Fri, 9 Jul 2021 14:58:24 -0400 (EDT) Received: by mail-qk1-f181.google.com with SMTP id i125so10313399qke.12 for ; Fri, 09 Jul 2021 11:58:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=1Rq09e4FzKVmC8coozmBf2Y4P9tsiDVwZ27Wjq/A92M=; b=DXZ7xOOrb+im+tg33i8mbTwsKlZOJPAfSLVS1lRsLDqp2+R0vdcuS+0zSmf1hozWid h8N38TsVQ1QPBClHpe5emLu45mi3w5tBd5vkYjiI6beSJjrXTQS9Lis5m+m60gBYrsC3 Jtaz1JParBBvM4YHj99L6tC2DD0jJWHj3QIQZ+3nIZDUcW68SbAYvmvI8sPsQ8rrBf9m DIbU5xPKHNZtJodbP2/6ps301q+e0hoyKM6Fse04usXf1IDGbmbscr/M14PPeovJ8vNW Z0jX0S9BplBgXLHzIJNTXSkc81ZwJBu7AYVTyYRZxK2MfH9BfbKUzQx/G8a7Ik/pB8aU o1CA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=1Rq09e4FzKVmC8coozmBf2Y4P9tsiDVwZ27Wjq/A92M=; b=R45zziUmKp6FEdWkHyztizaOHK9K8gaTKzbtp9Gb9DNpHeDKY98CXWOa5z9NjzPLpC rz6xjyXxcWcEmEC6F8aexvCH9et5ntYOJ625AWYyHO+c/ZJHlc4EcQvgWee9bVctksUV 5SntL4FXz9nygzsGWW8ytIN8w+AazGCzp7IqaxpYjOOfnZP2fCtWshTKLWU+hhlwr1zw AUfU6SfNEUNOjqov4YR9p0quZokuxw5fuAxYMbEMgxzJ+H1ysHn1CqOaQ2lZVnKcLug7 FmLw6pmVdLCVuTtdGm3CU9+f7n0pDMaruWPj4U/KmAVWmxtlOQ/CIl3lyOjgsSbbBOUr gO8g== X-Gm-Message-State: AOAM531962O1rRu/DlIApjX7zKgPH/cJ/UMr437e/MdtPspAsfQiz2sY n8jfYeaW9Q/sM8ZX2ok7qy5ka/D9FvaKBzo9+Li3jg== X-Google-Smtp-Source: ABdhPJx38EbV1FTTGhhuDjA8CTzpk9q6mzakZBUDCGz1zQ1TzLXHZcfuZU3Lo65MlCEwO1gJjRbol4iuazm9D4flAsc= X-Received: by 2002:ae9:dd43:: with SMTP id r64mr38651885qkf.216.1625857103581; Fri, 09 Jul 2021 11:58:23 -0700 (PDT) MIME-Version: 1.0 References: <20210709043713.887098-1-senozhatsky@chromium.org> <20210709043713.887098-3-senozhatsky@chromium.org> In-Reply-To: <20210709043713.887098-3-senozhatsky@chromium.org> From: Joel Fernandes Date: Fri, 9 Jul 2021 14:58:11 -0400 Message-ID: Subject: Re: [PATCHv2 2/4] arm64: add guest pvstate support To: Sergey Senozhatsky X-Mailman-Approved-At: Sat, 10 Jul 2021 06:18:40 -0400 Cc: Marc Zyngier , linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, Suleiman Souhlal , Will Deacon , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu Hi, Just few nits, patch itself LGTM: On Fri, Jul 9, 2021 at 12:37 AM Sergey Senozhatsky wrote: > > PV-vcpu-state is a per-CPU struct, which, for the time being, > holds boolean `preempted' vCPU state. During the startup, > given that host supports PV-state, each guest vCPU sends > a pointer to its per-CPU variable to the host as a payload > with the SMCCC HV call, so that host can update vCPU state > when it puts or loads vCPU. > > This has impact on the guest's scheduler: > > [..] > wake_up_process() > try_to_wake_up() > select_task_rq_fair() > available_idle_cpu() > vcpu_is_preempted() > > Some sched benchmarks data is available on the github page [0]. > > [0] https://github.com/sergey-senozhatsky/arm64-vcpu_is_preempted > > Signed-off-by: Sergey Senozhatsky > --- > arch/arm64/include/asm/paravirt.h | 19 +++++++ > arch/arm64/kernel/paravirt.c | 94 +++++++++++++++++++++++++++++++ > arch/arm64/kernel/smp.c | 4 ++ > 3 files changed, 117 insertions(+) > > diff --git a/arch/arm64/include/asm/paravirt.h b/arch/arm64/include/asm/paravirt.h > index 9aa193e0e8f2..a3f7665dff38 100644 > --- a/arch/arm64/include/asm/paravirt.h > +++ b/arch/arm64/include/asm/paravirt.h > @@ -2,6 +2,11 @@ > #ifndef _ASM_ARM64_PARAVIRT_H > #define _ASM_ARM64_PARAVIRT_H > > +struct vcpu_state { > + bool preempted; > + u8 reserved[63]; > +}; > + > #ifdef CONFIG_PARAVIRT > #include > > @@ -20,8 +25,22 @@ static inline u64 paravirt_steal_clock(int cpu) > > int __init pv_time_init(void); > > +bool dummy_vcpu_is_preempted(unsigned int cpu); > + > +extern struct static_key pv_vcpu_is_preempted_enabled;. pv_vcpu_is_preempted_enabled static_key is not used in any patch. Maybe it is stale? > +DECLARE_STATIC_CALL(pv_vcpu_is_preempted, dummy_vcpu_is_preempted); > + > +static inline bool paravirt_vcpu_is_preempted(unsigned int cpu) > +{ > + return static_call(pv_vcpu_is_preempted)(cpu); > +} > + > +int __init pv_vcpu_state_init(void); > + > #else > > +#define pv_vcpu_state_init() do {} while (0) > + > #define pv_time_init() do {} while (0) > > #endif // CONFIG_PARAVIRT > diff --git a/arch/arm64/kernel/paravirt.c b/arch/arm64/kernel/paravirt.c > index 75fed4460407..d8fc46795d94 100644 > --- a/arch/arm64/kernel/paravirt.c > +++ b/arch/arm64/kernel/paravirt.c > @@ -40,6 +40,11 @@ struct pv_time_stolen_time_region { > > static DEFINE_PER_CPU(struct pv_time_stolen_time_region, stolen_time_region); > > +static DEFINE_PER_CPU(struct vcpu_state, vcpus_states); > +struct static_key pv_vcpu_is_preempted_enabled; > + > +DEFINE_STATIC_CALL(pv_vcpu_is_preempted, dummy_vcpu_is_preempted); Could we use DEFINE_STATIC_CALL_NULL and get rid of the dummy function? I believe that makes the function trampoline as return instruction, till it is updated. > + > static bool steal_acc = true; > static int __init parse_no_stealacc(char *arg) > { > @@ -165,3 +170,92 @@ int __init pv_time_init(void) > > return 0; > } > + > +bool dummy_vcpu_is_preempted(unsigned int cpu) > +{ > + return false; > +} > + > +static bool __vcpu_is_preempted(unsigned int cpu) > +{ > + struct vcpu_state *st; > + > + st = &per_cpu(vcpus_states, cpu); > + return READ_ONCE(st->preempted); I guess you could just do: { return READ_ONCE(per_cpu(vcpus_states, cpu).preempted); } > +} > + > +static bool has_pv_vcpu_state(void) > +{ > + struct arm_smccc_res res; > + > + /* To detect the presence of PV time support we require SMCCC 1.1+ */ > + if (arm_smccc_1_1_get_conduit() == SMCCC_CONDUIT_NONE) > + return false; > + > + arm_smccc_1_1_invoke(ARM_SMCCC_ARCH_FEATURES_FUNC_ID, > + ARM_SMCCC_HV_PV_VCPU_STATE_FEATURES, > + &res); > + > + if (res.a0 != SMCCC_RET_SUCCESS) > + return false; > + return true; > +} > + > +static int __pv_vcpu_state_hook(unsigned int cpu, int event) > +{ > + struct arm_smccc_res res; > + struct vcpu_state *st; > + > + st = &per_cpu(vcpus_states, cpu); > + arm_smccc_1_1_invoke(event, virt_to_phys(st), &res); > + if (res.a0 != SMCCC_RET_SUCCESS) > + return -EINVAL; > + return 0; > +} > + > +static int vcpu_state_init(unsigned int cpu) > +{ > + int ret = __pv_vcpu_state_hook(cpu, ARM_SMCCC_HV_PV_VCPU_STATE_INIT); > + > + if (ret) > + pr_warn("Unable to ARM_SMCCC_HV_PV_STATE_INIT\n"); > + return ret; > +} > + > +static int vcpu_state_release(unsigned int cpu) > +{ > + int ret = __pv_vcpu_state_hook(cpu, ARM_SMCCC_HV_PV_VCPU_STATE_RELEASE); > + > + if (ret) > + pr_warn("Unable to ARM_SMCCC_HV_PV_STATE_RELEASE\n"); > + return ret; > +} > + > +static int pv_vcpu_state_register_hooks(void) > +{ > + int ret; > + > + ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, > + "hypervisor/arm/pvstate:starting", > + vcpu_state_init, > + vcpu_state_release); > + if (ret < 0) > + pr_warn("Failed to register CPU hooks\n"); > + return 0; > +} > + > +int __init pv_vcpu_state_init(void) > +{ > + int ret; > + > + if (!has_pv_vcpu_state()) > + return 0; > + > + ret = pv_vcpu_state_register_hooks(); > + if (ret) > + return ret; > + > + static_call_update(pv_vcpu_is_preempted, __vcpu_is_preempted); > + static_key_slow_inc(&pv_vcpu_is_preempted_enabled); I think this static key inc is also stale. thanks, -Joel _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0006C07E99 for ; Fri, 9 Jul 2021 18:59:56 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 84193610C7 for ; Fri, 9 Jul 2021 18:59:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 84193610C7 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=fjZ+cXgStm2Irid2qdUX3/p+xDbPEGH1omwOatV+9pE=; b=45g36ScwbK2yRP 32IfMe/ShOe+aZHbOVBPqx0mC6bkKLlTR4qJVmbqdJUKzsUJhOOhsn97FR0taxHKPl8L63lShzPoC Vn2/c0Ez59GlIRD8zRMfbomj0kVVJ5H1HihIaK6Q+R8i9c+ZxOcGyA8noHlU+Qq+PIOeHgSW2BRlI pyDcvF7xqQnNZBI0Qhl3eGQ8qFs82Z6O2ytbUzwMkrf/9h2cDzzSc8y3+SgLNjOrvQAUIQ4ZmRfKN 3EE92BUj+/W8eBv1aDpFRNojxD9hZel1BA/LUkzizpSEd/4wECefC2evTGERz6x4NYNwG4H4/km1A XL8boY+nDZVMN3dlWLjA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1m1vhm-002LzD-So; Fri, 09 Jul 2021 18:58:31 +0000 Received: from mail-qk1-x72f.google.com ([2607:f8b0:4864:20::72f]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1m1vhi-002Lyl-W7 for linux-arm-kernel@lists.infradead.org; Fri, 09 Jul 2021 18:58:28 +0000 Received: by mail-qk1-x72f.google.com with SMTP id a6so10351301qka.4 for ; Fri, 09 Jul 2021 11:58:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=1Rq09e4FzKVmC8coozmBf2Y4P9tsiDVwZ27Wjq/A92M=; b=DXZ7xOOrb+im+tg33i8mbTwsKlZOJPAfSLVS1lRsLDqp2+R0vdcuS+0zSmf1hozWid h8N38TsVQ1QPBClHpe5emLu45mi3w5tBd5vkYjiI6beSJjrXTQS9Lis5m+m60gBYrsC3 Jtaz1JParBBvM4YHj99L6tC2DD0jJWHj3QIQZ+3nIZDUcW68SbAYvmvI8sPsQ8rrBf9m DIbU5xPKHNZtJodbP2/6ps301q+e0hoyKM6Fse04usXf1IDGbmbscr/M14PPeovJ8vNW Z0jX0S9BplBgXLHzIJNTXSkc81ZwJBu7AYVTyYRZxK2MfH9BfbKUzQx/G8a7Ik/pB8aU o1CA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=1Rq09e4FzKVmC8coozmBf2Y4P9tsiDVwZ27Wjq/A92M=; b=aq5Jzj7u47ZMxa9f2zyUuYCwcnY8XXwSt77/czLx8vJ8AIXOQAX9cCNbYIaTB9km12 mjapDMsUXzrWI9mKpUv+nUCK9KaYqNML9zBc8vpe1jOnBOiXEWaoxQURJ822BeoweJC5 1DzQqgLDBJ9KpSMunqoHtJcs5cMBUk3HgiV1oGraXKTCT3pU8wHn61nkW03jNqDTT0jG LTfSFBd5v9pxl4DnDj4n7UgySHSGsIKMaUcQP0mT5XsBZimOMSTQ6PWvyHOzqheR14ql wh0/Xmb38fle1z6881sLiq8esI0nLdOXvIzSy8bMD9MpIEY3TMyFBcV40gJ4QMg2rNyd EURg== X-Gm-Message-State: AOAM532lnnlJ3qtouCIk1aM2/ZmUM+MzH175GTN9vpz3zhyyGNbWqfCN 4/o5uw5DfRIyrWGv//t0QboECV18zD6U2psjoQ6Wwg== X-Google-Smtp-Source: ABdhPJx38EbV1FTTGhhuDjA8CTzpk9q6mzakZBUDCGz1zQ1TzLXHZcfuZU3Lo65MlCEwO1gJjRbol4iuazm9D4flAsc= X-Received: by 2002:ae9:dd43:: with SMTP id r64mr38651885qkf.216.1625857103581; Fri, 09 Jul 2021 11:58:23 -0700 (PDT) MIME-Version: 1.0 References: <20210709043713.887098-1-senozhatsky@chromium.org> <20210709043713.887098-3-senozhatsky@chromium.org> In-Reply-To: <20210709043713.887098-3-senozhatsky@chromium.org> From: Joel Fernandes Date: Fri, 9 Jul 2021 14:58:11 -0400 Message-ID: Subject: Re: [PATCHv2 2/4] arm64: add guest pvstate support To: Sergey Senozhatsky Cc: Marc Zyngier , Will Deacon , Suleiman Souhlal , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210709_115827_103428_A4D6BA7D X-CRM114-Status: GOOD ( 27.31 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi, Just few nits, patch itself LGTM: On Fri, Jul 9, 2021 at 12:37 AM Sergey Senozhatsky wrote: > > PV-vcpu-state is a per-CPU struct, which, for the time being, > holds boolean `preempted' vCPU state. During the startup, > given that host supports PV-state, each guest vCPU sends > a pointer to its per-CPU variable to the host as a payload > with the SMCCC HV call, so that host can update vCPU state > when it puts or loads vCPU. > > This has impact on the guest's scheduler: > > [..] > wake_up_process() > try_to_wake_up() > select_task_rq_fair() > available_idle_cpu() > vcpu_is_preempted() > > Some sched benchmarks data is available on the github page [0]. > > [0] https://github.com/sergey-senozhatsky/arm64-vcpu_is_preempted > > Signed-off-by: Sergey Senozhatsky > --- > arch/arm64/include/asm/paravirt.h | 19 +++++++ > arch/arm64/kernel/paravirt.c | 94 +++++++++++++++++++++++++++++++ > arch/arm64/kernel/smp.c | 4 ++ > 3 files changed, 117 insertions(+) > > diff --git a/arch/arm64/include/asm/paravirt.h b/arch/arm64/include/asm/paravirt.h > index 9aa193e0e8f2..a3f7665dff38 100644 > --- a/arch/arm64/include/asm/paravirt.h > +++ b/arch/arm64/include/asm/paravirt.h > @@ -2,6 +2,11 @@ > #ifndef _ASM_ARM64_PARAVIRT_H > #define _ASM_ARM64_PARAVIRT_H > > +struct vcpu_state { > + bool preempted; > + u8 reserved[63]; > +}; > + > #ifdef CONFIG_PARAVIRT > #include > > @@ -20,8 +25,22 @@ static inline u64 paravirt_steal_clock(int cpu) > > int __init pv_time_init(void); > > +bool dummy_vcpu_is_preempted(unsigned int cpu); > + > +extern struct static_key pv_vcpu_is_preempted_enabled;. pv_vcpu_is_preempted_enabled static_key is not used in any patch. Maybe it is stale? > +DECLARE_STATIC_CALL(pv_vcpu_is_preempted, dummy_vcpu_is_preempted); > + > +static inline bool paravirt_vcpu_is_preempted(unsigned int cpu) > +{ > + return static_call(pv_vcpu_is_preempted)(cpu); > +} > + > +int __init pv_vcpu_state_init(void); > + > #else > > +#define pv_vcpu_state_init() do {} while (0) > + > #define pv_time_init() do {} while (0) > > #endif // CONFIG_PARAVIRT > diff --git a/arch/arm64/kernel/paravirt.c b/arch/arm64/kernel/paravirt.c > index 75fed4460407..d8fc46795d94 100644 > --- a/arch/arm64/kernel/paravirt.c > +++ b/arch/arm64/kernel/paravirt.c > @@ -40,6 +40,11 @@ struct pv_time_stolen_time_region { > > static DEFINE_PER_CPU(struct pv_time_stolen_time_region, stolen_time_region); > > +static DEFINE_PER_CPU(struct vcpu_state, vcpus_states); > +struct static_key pv_vcpu_is_preempted_enabled; > + > +DEFINE_STATIC_CALL(pv_vcpu_is_preempted, dummy_vcpu_is_preempted); Could we use DEFINE_STATIC_CALL_NULL and get rid of the dummy function? I believe that makes the function trampoline as return instruction, till it is updated. > + > static bool steal_acc = true; > static int __init parse_no_stealacc(char *arg) > { > @@ -165,3 +170,92 @@ int __init pv_time_init(void) > > return 0; > } > + > +bool dummy_vcpu_is_preempted(unsigned int cpu) > +{ > + return false; > +} > + > +static bool __vcpu_is_preempted(unsigned int cpu) > +{ > + struct vcpu_state *st; > + > + st = &per_cpu(vcpus_states, cpu); > + return READ_ONCE(st->preempted); I guess you could just do: { return READ_ONCE(per_cpu(vcpus_states, cpu).preempted); } > +} > + > +static bool has_pv_vcpu_state(void) > +{ > + struct arm_smccc_res res; > + > + /* To detect the presence of PV time support we require SMCCC 1.1+ */ > + if (arm_smccc_1_1_get_conduit() == SMCCC_CONDUIT_NONE) > + return false; > + > + arm_smccc_1_1_invoke(ARM_SMCCC_ARCH_FEATURES_FUNC_ID, > + ARM_SMCCC_HV_PV_VCPU_STATE_FEATURES, > + &res); > + > + if (res.a0 != SMCCC_RET_SUCCESS) > + return false; > + return true; > +} > + > +static int __pv_vcpu_state_hook(unsigned int cpu, int event) > +{ > + struct arm_smccc_res res; > + struct vcpu_state *st; > + > + st = &per_cpu(vcpus_states, cpu); > + arm_smccc_1_1_invoke(event, virt_to_phys(st), &res); > + if (res.a0 != SMCCC_RET_SUCCESS) > + return -EINVAL; > + return 0; > +} > + > +static int vcpu_state_init(unsigned int cpu) > +{ > + int ret = __pv_vcpu_state_hook(cpu, ARM_SMCCC_HV_PV_VCPU_STATE_INIT); > + > + if (ret) > + pr_warn("Unable to ARM_SMCCC_HV_PV_STATE_INIT\n"); > + return ret; > +} > + > +static int vcpu_state_release(unsigned int cpu) > +{ > + int ret = __pv_vcpu_state_hook(cpu, ARM_SMCCC_HV_PV_VCPU_STATE_RELEASE); > + > + if (ret) > + pr_warn("Unable to ARM_SMCCC_HV_PV_STATE_RELEASE\n"); > + return ret; > +} > + > +static int pv_vcpu_state_register_hooks(void) > +{ > + int ret; > + > + ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, > + "hypervisor/arm/pvstate:starting", > + vcpu_state_init, > + vcpu_state_release); > + if (ret < 0) > + pr_warn("Failed to register CPU hooks\n"); > + return 0; > +} > + > +int __init pv_vcpu_state_init(void) > +{ > + int ret; > + > + if (!has_pv_vcpu_state()) > + return 0; > + > + ret = pv_vcpu_state_register_hooks(); > + if (ret) > + return ret; > + > + static_call_update(pv_vcpu_is_preempted, __vcpu_is_preempted); > + static_key_slow_inc(&pv_vcpu_is_preempted_enabled); I think this static key inc is also stale. thanks, -Joel _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel