From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3151AC433F5 for ; Thu, 31 Mar 2022 21:31:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241545AbiCaVdi (ORCPT ); Thu, 31 Mar 2022 17:33:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43738 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241891AbiCaVc5 (ORCPT ); Thu, 31 Mar 2022 17:32:57 -0400 Received: from mail-pj1-x102d.google.com (mail-pj1-x102d.google.com [IPv6:2607:f8b0:4864:20::102d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4D2C9228D25 for ; Thu, 31 Mar 2022 14:31:06 -0700 (PDT) Received: by mail-pj1-x102d.google.com with SMTP id c15-20020a17090a8d0f00b001c9c81d9648so673845pjo.2 for ; Thu, 31 Mar 2022 14:31:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=UL/PnnUMFBU9oeLgviWivIjBCm6kEvnLQU0vrlKd5dA=; b=fYjBlIX+0NIShJ64f+lcRsE5KdkM+nYLs85jTdUufPPG/2G9PVnkJLdmEq/yaO7jI/ 4hwcxsvV9/7UlykD8aD2qqCderteCl+jWXP4ibB6jS8vuk9Njgv31Gzm5b65HJba+f05 +2xL1D0bAykFDhvXHmtVeDsLWGwqVmuLd4S3MbYPno3VRYDqDEY4q1AiblRD0tnLnft9 sIE2/YF2uCpdh0Wv9IBemeHWnSODTcSlXeeOaPlgwsIkAL6Dd4alI9RHvHxZs8yiog3B bzmEoPH3rSQ43LetFjOCgtdtheFx9w83GhqvxWJ/Jm6z19nLdSshRgfk+Vx4AdrR0gZS QUUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=UL/PnnUMFBU9oeLgviWivIjBCm6kEvnLQU0vrlKd5dA=; b=k2sE+1qKYAjvReSbnYt/qV5RDKlcnGHprPEHzYstHTTdJDVZE8aEi3TXVlJIua8vxl 2m0o+pcRTzlcbvoLiAtISuWlMEqOSR2EkjD/C6Bfbxdrt6sf6194/7B7rKRttK05FPOr /AE9cDuqIxFZ4JCxrXMiGjTcCXaOBkvc8HX78KcuWDt21o/GWRR43sPXuQM7eP4EVEdM KAOs4N/u4rE2M2xwrmTsfPbOTzDEwa4G8Au2A3/PXM75tv6i9tx4wUIiBDXZefnOqtDr V81DlCkXOVgV89t++8OW1L1L1zFAVP3+nP+T4oWkMuNWKhbYDNAuDVGJGH7aCxj+3lg8 Mw8g== X-Gm-Message-State: AOAM530k+GfXPBwwiMkVxEUHrl+KDvHaTZ6QmGGJ1pQ9f7S1QMwdoVcL vQcXwesXtQ3Hnknb2ukpZ2acwA== X-Google-Smtp-Source: ABdhPJxbHbftSGYZ3hVn3/5INO+wSwQxMtdJL5IHoSlq6UXrjCvK/QfReT9Xb+VGBauLz8rRR8Xtjg== X-Received: by 2002:a17:902:768c:b0:155:e4a2:1f09 with SMTP id m12-20020a170902768c00b00155e4a21f09mr7148034pll.43.1648762265581; Thu, 31 Mar 2022 14:31:05 -0700 (PDT) Received: from google.com (157.214.185.35.bc.googleusercontent.com. [35.185.214.157]) by smtp.gmail.com with ESMTPSA id q10-20020a056a00088a00b004f7ceff389esm404697pfj.152.2022.03.31.14.31.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 31 Mar 2022 14:31:04 -0700 (PDT) Date: Thu, 31 Mar 2022 21:31:01 +0000 From: Sean Christopherson To: Mingwei Zhang Cc: Peter Gonda , kvm , LKML Subject: Re: [PATCH] KVM: SEV: Add cond_resched() to loop in sev_clflush_pages() Message-ID: References: <20220330164306.2376085-1-pgonda@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Mar 30, 2022, Mingwei Zhang wrote: > On Wed, Mar 30, 2022 at 9:43 AM Peter Gonda wrote: > > > > Add resched to avoid warning from sev_clflush_pages() with large number > > of pages. > > > > Signed-off-by: Peter Gonda > > Cc: Sean Christopherson > > Cc: kvm@vger.kernel.org > > Cc: linux-kernel@vger.kernel.org > > > > --- > > Here is a warning similar to what I've seen many times running large SEV > > VMs: > > [ 357.714051] CPU 15: need_resched set for > 52000222 ns (52 ticks) without schedule > > [ 357.721623] WARNING: CPU: 15 PID: 35848 at kernel/sched/core.c:3733 scheduler_tick+0x2f9/0x3f0 > > [ 357.730222] Modules linked in: kvm_amd uhaul vfat fat hdi2_standard_ftl hdi2_megablocks hdi2_pmc hdi2_pmc_eeprom hdi2 stg elephant_dev_num ccp i2c_mux_ltc4306 i2c_mux i2c_via_ipmi i2c_piix4 google_bmc_usb google_bmc_gpioi2c_mb_common google_bmc_mailbox cdc_acm xhci_pci xhci_hcd sha3_generic gq nv_p2p_glue accel_class > > [ 357.758261] CPU: 15 PID: 35848 Comm: switchto-defaul Not tainted 4.15.0-smp-DEV #11 > > [ 357.765912] Hardware name: Google, Inc. Arcadia_IT_80/Arcadia_IT_80, BIOS 30.20.2-gce 11/05/2021 > > [ 357.779372] RIP: 0010:scheduler_tick+0x2f9/0x3f0 > > [ 357.783988] RSP: 0018:ffff98558d1c3dd8 EFLAGS: 00010046 > > [ 357.789207] RAX: 741f23206aa8dc00 RBX: 0000005349236a42 RCX: 0000000000000007 > > [ 357.796339] RDX: 0000000000000006 RSI: 0000000000000002 RDI: ffff98558d1d5a98 > > [ 357.803463] RBP: ffff98558d1c3ea0 R08: 0000000000100ceb R09: 0000000000000000 > > [ 357.810597] R10: ffff98558c958c00 R11: ffffffff94850740 R12: 00000000031975de > > [ 357.817729] R13: 0000000000000000 R14: ffff98558d1e2640 R15: ffff98525739ea40 > > [ 357.824862] FS: 00007f87503eb700(0000) GS:ffff98558d1c0000(0000) knlGS:0000000000000000 > > [ 357.832948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 357.838695] CR2: 00005572fe74b080 CR3: 0000007bea706006 CR4: 0000000000360ef0 > > [ 357.845828] Call Trace: > > [ 357.848277] > > [ 357.850294] [] ? tick_setup_sched_timer+0x130/0x130 > > [ 357.856818] [] ? rcu_sched_clock_irq+0x6ed/0x850 > > [ 357.863084] [] ? __run_timers+0x42/0x260 > > [ 357.868654] [] ? tick_setup_sched_timer+0x130/0x130 > > [ 357.875182] [] update_process_times+0x7b/0x90 > > [ 357.881188] [] tick_sched_timer+0x82/0xd0 > > [ 357.886845] [] __run_hrtimer+0x81/0x200 > > [ 357.892331] [] hrtimer_interrupt+0x192/0x450 > > [ 357.898252] [] ? __do_softirq+0x2fa/0x33e > > [ 357.903911] [] smp_apic_timer_interrupt+0xac/0x1d0 > > [ 357.910349] [] apic_timer_interrupt+0x86/0x90 > > [ 357.916347] > > [ 357.918452] RIP: 0010:clflush_cache_range+0x3f/0x50 > > [ 357.923324] RSP: 0018:ffff98529af89cc0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff12 > > [ 357.930889] RAX: 0000000000000040 RBX: 0000000000038135 RCX: ffff985233d36000 > > [ 357.938013] RDX: ffff985233d36000 RSI: 0000000000001000 RDI: ffff985233d35000 > > [ 357.945145] RBP: ffff98529af89cc0 R08: 0000000000000001 R09: ffffb5753fb23000 > > [ 357.952271] R10: 000000000003fe00 R11: 0000000000000008 R12: 0000000000040000 > > [ 357.959401] R13: ffff98525739ea40 R14: ffffb5753fb22000 R15: ffff98532a58dd80 > > [ 357.966536] [] svm_register_enc_region+0xd1/0x170 [kvm_amd] > > [ 357.973758] [] kvm_arch_vm_ioctl+0x84c/0xb00 > > [ 357.979677] [] ? handle_mm_fault+0x6ff/0x1370 > > [ 357.985683] [] kvm_vm_ioctl+0x69b/0x720 > > [ 357.991167] [] do_vfs_ioctl+0x47d/0x680 > > [ 357.996654] [] SyS_ioctl+0x68/0x90 > > [ 358.001706] [] do_syscall_64+0x71/0x110 > > [ 358.007192] [] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 > > > > Tested by running a large 256gib SEV VM several times, saw no warnings. > > Without the change warnings are seen. Clean up the splat (remove timestamps, everything with a ?, etc... I believe there is a kernel scripts/ to do this...) and throw it in the changelog. Documenting the exact problem is very helpful, e.g. future readers may wonder "what warning?". > > --- > > arch/x86/kvm/svm/sev.c | 1 + > > 1 file changed, 1 insertion(+) > > > > diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c > > index 75fa6dd268f0..c2fe89ecdb2d 100644 > > --- a/arch/x86/kvm/svm/sev.c > > +++ b/arch/x86/kvm/svm/sev.c > > @@ -465,6 +465,7 @@ static void sev_clflush_pages(struct page *pages[], unsigned long npages) > > page_virtual = kmap_atomic(pages[i]); > > clflush_cache_range(page_virtual, PAGE_SIZE); > > kunmap_atomic(page_virtual); > > + cond_resched(); > > If you add cond_resched() here, the frequency (once per 4K) might be > too high. You may want to do it once per X pages, where X could be > something like 1G/4K? No, every iteration is perfectly ok. The "cond"itional part means that this will reschedule if and only if it actually needs to be rescheduled, e.g. if the task's timeslice as expired. The check for a needed reschedule is cheap, using cond_resched() in tight-ish loops is ok and intended, e.g. KVM does a reched check prior to enterring the guest.