From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9022AC4338F for ; Fri, 23 Jul 2021 22:20:44 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1086A60EB4 for ; Fri, 23 Jul 2021 22:20:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 1086A60EB4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:50478 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m73X7-0008Mu-6H for qemu-devel@archiver.kernel.org; Fri, 23 Jul 2021 18:20:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34042) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m73WA-0007gr-Hk for qemu-devel@nongnu.org; Fri, 23 Jul 2021 18:19:42 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:26426) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m73W8-0002tO-A7 for qemu-devel@nongnu.org; Fri, 23 Jul 2021 18:19:41 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1627078779; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=c124M8BTnQQYvZckXLTZ9axGbH9li65kvr1RL4kJyRg=; b=KQUFe00CJeWsg78L4kc3M1MyxDCDBT+SDmLm22EQGm9FCAz+6CzeKYorg6RgMRjg5AGBqr RwgC8z5h5z8zfz0zNNqSl3ObksiM1dXH3atCJnmIAT46cM4hCdrjWVRFwA2PrFlPcJisFF WU5uWy+hRA5GmlhuRjLplyFriptJcsw= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-3-OANSFev9MsSfK1BnSPNjGw-1; Fri, 23 Jul 2021 18:19:38 -0400 X-MC-Unique: OANSFev9MsSfK1BnSPNjGw-1 Received: by mail-qk1-f198.google.com with SMTP id p123-20020a378d810000b02903ad5730c883so2142595qkd.22 for ; Fri, 23 Jul 2021 15:19:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=c124M8BTnQQYvZckXLTZ9axGbH9li65kvr1RL4kJyRg=; b=HFNSIH8ydtBluwdOx94UmchxmoArbj4bMCTR0u/GDfR347AXz1abySb7u7YJsB2mOE rb/o0WF3PSItrUOjsD4cTrgZbU/tbWf1yFzczKiWNC9Mmdmb+y57BMFhgQGA+tjnoS5s gu4JQid+AHeCTIr6SUgWYW61T2fWYA5+qTI1Q79BCMG9ZZhZFAbSV9Czt5RlXOhpOfM7 BS8vxoq6qzpexECawSJJDza6DD/QuUNy2oUEaAYctomUJWJDuOTZ+1jVel92lezGiXwG sGwWIQp7RRfXcZuNni7++4c3g4beZXN17GTsvGf2nXd+MAe4rAweb4Oym9No2+wcdQw7 igVg== X-Gm-Message-State: AOAM532+BflYSEz4m2NMUsAvNcz+Np1Wrb7+Bl1ehOLfabpF3kjVr+PG /z2fYA0d0hDn/WA1OvevE1jUTiqw68QQwQaR+YnJfDz9cTCGjQBjc0Hg0LcD5PxUFYApFjmak/V DEPTSIkkwOE0XAvo= X-Received: by 2002:a37:ad0a:: with SMTP id f10mr7061079qkm.284.1627078777656; Fri, 23 Jul 2021 15:19:37 -0700 (PDT) X-Google-Smtp-Source: ABdhPJznPWIJkInfz3AklLqfshs8nFAZD06Pfz7hKe3xMLz6e0P579In2JO271ihUCeaeri2W7hssQ== X-Received: by 2002:a37:ad0a:: with SMTP id f10mr7061055qkm.284.1627078777443; Fri, 23 Jul 2021 15:19:37 -0700 (PDT) Received: from t490s (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id d79sm10524891qke.45.2021.07.23.15.19.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jul 2021 15:19:36 -0700 (PDT) Date: Fri, 23 Jul 2021 18:19:35 -0400 From: Peter Xu To: David Hildenbrand Subject: Re: [PATCH v2 0/6] migration/ram: Optimize for virtio-mem via RamDiscardManager Message-ID: References: <20210721092759.21368-1-david@redhat.com> <800e421c-70b8-1ef2-56f7-cdbce7a7706b@redhat.com> MIME-Version: 1.0 In-Reply-To: Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -35 X-Spam_score: -3.6 X-Spam_bar: --- X-Spam_report: (-3.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.472, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , "Michael S. Tsirkin" , Pankaj Gupta , Juan Quintela , teawater , "Dr. David Alan Gilbert" , qemu-devel@nongnu.org, Alex Williamson , Marek Kedzierski , Paolo Bonzini , Andrey Gruzdev , Wei Yang Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" On Fri, Jul 23, 2021 at 08:41:40PM +0200, David Hildenbrand wrote: > On 23.07.21 18:12, Peter Xu wrote: > > On Thu, Jul 22, 2021 at 01:43:41PM +0200, David Hildenbrand wrote: > > > > > a) In precopy code, always clearing all dirty bits from the bitmap that > > > > > correspond to discarded range, whenever we update the dirty bitmap. This > > > > > results in logically unplugged memory to never get migrated. > > > > > > > > Have you seen cases where discarded areas are being marked as dirty? > > > > That suggests something somewhere is writing to them and shouldn't be. > > > > > > I have due to sub-optimal clear_bmap handling to be sorted out by > > > > > > https://lkml.kernel.org/r/20210722083055.23352-1-wei.w.wang@intel.com > > > > > > Whereby the issue is rather that initially dirty bits don't get cleared in > > > lower layers and keep popping up as dirty. > > > > > > The issue with postcopy recovery code setting discarded ranges dirty in > > > the dirty bitmap, I did not try reproducing. But from looking at the > > > code, it's pretty clear that it would happen. > > > > > > Apart from that, nothing should dirty that memory. Of course, > > > malicious guests could trigger it for now, in which case we wouldn't catch it > > > and migrate such pages with postcopy, because the final bitmap sync in > > > ram_postcopy_send_discard_bitmap() is performed without calling notifiers > > > right now. > > > > I have the same concern with Dave: does it mean that we don't need to touch at > > least ramblock_sync_dirty_bitmap in patch 3? > > Yes, see the comment in patch #3: > > " > Note: If discarded ranges span complete clear_bmap chunks, we'll never > clear the corresponding bits from clear_bmap and consequently never call > memory_region_clear_dirty_bitmap on the affected regions. While this is > perfectly fine, we're still synchronizing the bitmap of discarded ranges, > for example, in > ramblock_sync_dirty_bitmap()->cpu_physical_memory_sync_dirty_bitmap() > but also during memory_global_dirty_log_sync(). > > In the future, it might make sense to never even synchronize the dirty log > of these ranges, for example in KVM code, skipping discarded ranges > completely. > " > > The KVM path might be even more interesting (with !dirty ring IIRC). > > So that might certainly be worth looking into if we find it to be a real > performance problem. OK; hmm then I feel like what's missing is we didn't have the dirty bmap and the clear map synced - say, what if we do memory_region_clear_dirty_bitmap() when dropping the virtio-mem unplugged ranges too? If disgarded ranges are static during migration, the clear dirty log should happen once for them at bitmap init time. Then IIUC when sync we don't need to worry about unplugged memory anymore. > > > > > Doing that for bitmap init and postcopy recovery looks right. > > > > One other trivial comment is instead of touching up ram_dirty_bitmap_reload(), > > IMHO it's simpler to set all 1's to disgarded memories on dst receivedmap; > > imagine multiple postcopy recovery happened, then with that we walk the disgard > > memory list only once for each migration. Not a big deal, though. > > Right, but I decided to reuse > ramblock_dirty_bitmap_exclude_discarded_pages() such that I can avoid yet > another helper. Yeah, that's okay. Thanks, -- Peter Xu