From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.3 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D89BC433ED for ; Tue, 20 Apr 2021 10:24:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 01AF861164 for ; Tue, 20 Apr 2021 10:23:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231573AbhDTKYZ (ORCPT ); Tue, 20 Apr 2021 06:24:25 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:46177 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231473AbhDTKYV (ORCPT ); Tue, 20 Apr 2021 06:24:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1618914229; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=b7a+2NIHct7ey/wtfhecbqVWOV2x5OoBwHuqA9oFCX0=; b=XIGscTCSkpkE7hak/7hH7wpclhsG7GZ5QgfQgUG3bqg10mxdbWmh+08PTrv9AMFz+M0phZ fpSrmPXbPg/QQJ47xcJYaBJ0zsLKeyUcNxYRuBpnQhZRoUan3vR5rNYSf6USgcMbrJGw/Q q0pgUSyT819zAY3wW0DidD5wRZVOWFQ= Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-471-cXFioYEyN02HXgocc0E-Cg-1; Tue, 20 Apr 2021 06:23:47 -0400 X-MC-Unique: cXFioYEyN02HXgocc0E-Cg-1 Received: by mail-ed1-f69.google.com with SMTP id i18-20020aa7c7120000b02903853032ef71so3924584edq.22 for ; Tue, 20 Apr 2021 03:23:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=b7a+2NIHct7ey/wtfhecbqVWOV2x5OoBwHuqA9oFCX0=; b=GjLgzOlxHpsbbwYFap2Mi2OVESuMk53IaCuTrWFg2t5FXI148VkFJp9Z1onHOLpKKG Vkcmp3Z4oj3ABCkrQ2cLtDC6knJDteJDYn18i6j3GXJ85yCoSgi76spwyLkppLAy6yFo iq7f7LvWXzL7PO+mQSkh/Vn8ZhUbaM0Fy2DG1eQbzwurViWLOv7sN2j8JeS2k4h+HC1o o/6n6+Iu/gw4UYxfafYjqP6i3pJ4A7DjQ0IoDHb+hhMgBqpa67goy16fqldasFpfg+6a JUsU3d/QEgdcuQpa++7xdW1K/eiGFeG6CKZdk35VN5B8u2Sq+BBhqeYB09VnE12clNkP mL2g== X-Gm-Message-State: AOAM532c4YMpzDHR/tQaPV2cbDdMC7oKOEs3X73Ju3WUatIZskjLr7BR Z3ogI0CI8z1LJDbdYCXbpN7i9YWwI1XRXlYXESlUYqPidTb+hVI7vb8HYDAaMywTxD9CS+iYx+j kK8Mits3wOkJ6UrRT6RZcnS5q X-Received: by 2002:a17:907:2151:: with SMTP id rk17mr26644958ejb.203.1618914226686; Tue, 20 Apr 2021 03:23:46 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzodp7RMnFhaEX9D/NoF30lFskpxX7NtEUD3yP5k2w2nOq60l9HMWrqSyGFWJM4H4SbP3QJkA== X-Received: by 2002:a17:907:2151:: with SMTP id rk17mr26644944ejb.203.1618914226491; Tue, 20 Apr 2021 03:23:46 -0700 (PDT) Received: from ?IPv6:2001:b07:6468:f312:c8dd:75d4:99ab:290a? ([2001:b07:6468:f312:c8dd:75d4:99ab:290a]) by smtp.gmail.com with ESMTPSA id f20sm9015003ejw.36.2021.04.20.03.23.44 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 20 Apr 2021 03:23:45 -0700 (PDT) Subject: Re: [PATCH] KVM: Boost vCPU candidiate in user mode which is delivering interrupt To: Wanpeng Li Cc: Sean Christopherson , LKML , kvm , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel References: <1618542490-14756-1-git-send-email-wanpengli@tencent.com> <9c49c6ff-d896-e6a5-c051-b6707f6ec58a@redhat.com> From: Paolo Bonzini Message-ID: Date: Tue, 20 Apr 2021 12:23:44 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 20/04/21 10:48, Wanpeng Li wrote: >> I was thinking of something simpler: >> >> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c >> index 9b8e30dd5b9b..455c648f9adc 100644 >> --- a/virt/kvm/kvm_main.c >> +++ b/virt/kvm/kvm_main.c >> @@ -3198,10 +3198,9 @@ void kvm_vcpu_on_spin(struct kvm_vcpu *me, bool yield_to_kernel_mode) >> { >> struct kvm *kvm = me->kvm; >> struct kvm_vcpu *vcpu; >> - int last_boosted_vcpu = me->kvm->last_boosted_vcpu; >> int yielded = 0; >> int try = 3; >> - int pass; >> + int pass, num_passes = 1; >> int i; >> >> kvm_vcpu_set_in_spin_loop(me, true); >> @@ -3212,13 +3211,14 @@ void kvm_vcpu_on_spin(struct kvm_vcpu *me, bool yield_to_kernel_mode) >> * VCPU is holding the lock that we need and will release it. >> * We approximate round-robin by starting at the last boosted VCPU. >> */ >> - for (pass = 0; pass < 2 && !yielded && try; pass++) { >> - kvm_for_each_vcpu(i, vcpu, kvm) { >> - if (!pass && i <= last_boosted_vcpu) { >> - i = last_boosted_vcpu; >> - continue; >> - } else if (pass && i > last_boosted_vcpu) >> - break; >> + for (pass = 0; pass < num_passes; pass++) { >> + int idx = me->kvm->last_boosted_vcpu; >> + int n = atomic_read(&kvm->online_vcpus); >> + for (i = 0; i < n; i++, idx++) { >> + if (idx == n) >> + idx = 0; >> + >> + vcpu = kvm_get_vcpu(kvm, idx); >> if (!READ_ONCE(vcpu->ready)) >> continue; >> if (vcpu == me) >> @@ -3226,23 +3226,36 @@ void kvm_vcpu_on_spin(struct kvm_vcpu *me, bool yield_to_kernel_mode) >> if (rcuwait_active(&vcpu->wait) && >> !vcpu_dy_runnable(vcpu)) >> continue; >> - if (READ_ONCE(vcpu->preempted) && yield_to_kernel_mode && >> - !kvm_arch_vcpu_in_kernel(vcpu)) >> - continue; >> if (!kvm_vcpu_eligible_for_directed_yield(vcpu)) >> continue; >> >> + if (READ_ONCE(vcpu->preempted) && yield_to_kernel_mode && >> + !kvm_arch_vcpu_in_kernel(vcpu)) { >> + /* >> + * A vCPU running in userspace can get to kernel mode via >> + * an interrupt. That's a worse choice than a CPU already >> + * in kernel mode so only do it on a second pass. >> + */ >> + if (!vcpu_dy_runnable(vcpu)) >> + continue; >> + if (pass == 0) { >> + num_passes = 2; >> + continue; >> + } >> + } >> + >> yielded = kvm_vcpu_yield_to(vcpu); >> if (yielded > 0) { >> kvm->last_boosted_vcpu = i; >> - break; >> + goto done; >> } else if (yielded < 0) { >> try--; >> if (!try) >> - break; >> + goto done; >> } >> } >> } >> +done: > > We just tested the above post against 96 vCPUs VM in an over-subscribe > scenario, the score of pbzip2 fluctuated drastically. Sometimes it is > worse than vanilla, but the average improvement is around 2.2%. The > new version of my post is around 9.3%，the origial posted patch is > around 10% which is totally as expected since now both IPI receivers > in user-mode and lock-waiters are second class citizens. Fair enough. Of the two patches you posted I prefer the original, so I'll go with that one. Paolo