From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89EE5C4338F for ; Wed, 18 Aug 2021 02:49:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6F14160C3F for ; Wed, 18 Aug 2021 02:49:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234754AbhHRCuJ (ORCPT ); Tue, 17 Aug 2021 22:50:09 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:61008 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231449AbhHRCuE (ORCPT ); Tue, 17 Aug 2021 22:50:04 -0400 Received: from pps.filterd (m0098416.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17I2Xbkh178896; Tue, 17 Aug 2021 22:49:26 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=message-id : subject : from : reply-to : to : cc : date : in-reply-to : references : content-type : mime-version : content-transfer-encoding; s=pp1; bh=+6BQ94yJVmaI2Fu+VmqdYXcvuuOZRpOs2FcOeXGN+RA=; b=C7WtOP1xHf7tbQQAW/aFLzpm781/s3tJTiSZGoiTUhNZwBERASeYnSGVinhfjWi7XVNm xowlEP5jRmRrgMPN6rUK04Cl1yvanbodrVLTJwKJRHzkDWG45Y8r5peNVdRDmq85SObw VGKX3zqzHtern3QtnZLUSgXAXxx5NdAjNaCcUHt+ShM5o8ghu+F9d39ZpHztwdYzp0sQ u/kHISQRl5BTRhNnDPivLLJVmkbm2KDFpLTpbI0n72Z4IOZMjAuOKTaLMXMmJFBFfDmt FMTdNwTAKh6Oqie1KV4DPWBGPm/iCLwwjGOBRCF+bJCgzeRFsgm8OwugI44uLo6lT2QK IQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 3agf0e9wbr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 17 Aug 2021 22:49:26 -0400 Received: from m0098416.ppops.net (m0098416.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 17I2ZRU8182823; Tue, 17 Aug 2021 22:49:25 -0400 Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0b-001b2d01.pphosted.com with ESMTP id 3agf0e9wbf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 17 Aug 2021 22:49:25 -0400 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 17I2gFO1023498; Wed, 18 Aug 2021 02:49:24 GMT Received: from b03cxnp07029.gho.boulder.ibm.com (b03cxnp07029.gho.boulder.ibm.com [9.17.130.16]) by ppma03dal.us.ibm.com with ESMTP id 3ae5fe99h4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 18 Aug 2021 02:49:24 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp07029.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 17I2nNTc44958168 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 18 Aug 2021 02:49:23 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 49E1C78064; Wed, 18 Aug 2021 02:49:23 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E4DC67805C; Wed, 18 Aug 2021 02:49:19 +0000 (GMT) Received: from jarvis.lan (unknown [9.160.128.138]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Wed, 18 Aug 2021 02:49:19 +0000 (GMT) Message-ID: Subject: Re: [RFC PATCH 00/13] Add support for Mirror VM. From: James Bottomley Reply-To: jejb@linux.ibm.com To: Steve Rutherford Cc: Paolo Bonzini , Ashish Kalra , qemu-devel , Thomas Lendacky , Brijesh Singh , "Habkost, Eduardo" , "S. Tsirkin, Michael" , Richard Henderson , Tobin Feldman-Fitzthum , Dov Murik , Hubertus Franke , David Gilbert , kvm Date: Tue, 17 Aug 2021 22:49:18 -0400 In-Reply-To: References: <0fcfafde-a690-f53a-01fc-542054948bb2@redhat.com> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.34.4 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: SfOHr1ARLf-i_TefRs9cqp3jSg-SmtUK X-Proofpoint-GUID: azzm65j96tps6cIfZhlnNAG2rT6H6lOX X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.790 definitions=2021-08-17_09:2021-08-17,2021-08-17 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 lowpriorityscore=0 bulkscore=0 mlxlogscore=535 priorityscore=1501 malwarescore=0 mlxscore=0 impostorscore=0 clxscore=1015 suspectscore=0 adultscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108180014 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On Tue, 2021-08-17 at 16:10 -0700, Steve Rutherford wrote: > On Tue, Aug 17, 2021 at 3:57 PM James Bottomley > wrote: > > Realistically, migration is becoming a royal pain, not just for > > confidential computing, but for virtual functions in general. I > > really think we should look at S3 suspend, where we shut down the > > drivers and then reattach on S3 resume as the potential pathway to > > getting migration working both for virtual functions and this use > > case. > > This type of migration seems a little bit less "live", which makes me > concerned about its performance characteristics. Well, there are too many scenarios we just fail at migration today. We need help from the guest to quiesce or shut down the interior devices, and S3 suspend seems to be the machine signal for that. I think in most clouds guests would accept some loss of "liveness" for a gain in reliability as long as we keep them within the SLA ... which is 5 minutes a year for 5 nines. Most failed migrations also instantly fail SLAs because of the recovery times involved so I don't see what's to be achieved by keeping the current "we can migrate sometimes" approach. James From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 871E7C4320A for ; Wed, 18 Aug 2021 02:50:17 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2060C60FE6 for ; Wed, 18 Aug 2021 02:50:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 2060C60FE6 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:56642 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mGBei-0004gd-6j for qemu-devel@archiver.kernel.org; Tue, 17 Aug 2021 22:50:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39722) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mGBdy-000404-84 for qemu-devel@nongnu.org; Tue, 17 Aug 2021 22:49:30 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:50200 helo=mx0a-001b2d01.pphosted.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mGBdw-000290-HI for qemu-devel@nongnu.org; Tue, 17 Aug 2021 22:49:30 -0400 Received: from pps.filterd (m0098416.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17I2Xbkh178896; Tue, 17 Aug 2021 22:49:26 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=message-id : subject : from : reply-to : to : cc : date : in-reply-to : references : content-type : mime-version : content-transfer-encoding; s=pp1; bh=+6BQ94yJVmaI2Fu+VmqdYXcvuuOZRpOs2FcOeXGN+RA=; b=C7WtOP1xHf7tbQQAW/aFLzpm781/s3tJTiSZGoiTUhNZwBERASeYnSGVinhfjWi7XVNm xowlEP5jRmRrgMPN6rUK04Cl1yvanbodrVLTJwKJRHzkDWG45Y8r5peNVdRDmq85SObw VGKX3zqzHtern3QtnZLUSgXAXxx5NdAjNaCcUHt+ShM5o8ghu+F9d39ZpHztwdYzp0sQ u/kHISQRl5BTRhNnDPivLLJVmkbm2KDFpLTpbI0n72Z4IOZMjAuOKTaLMXMmJFBFfDmt FMTdNwTAKh6Oqie1KV4DPWBGPm/iCLwwjGOBRCF+bJCgzeRFsgm8OwugI44uLo6lT2QK IQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 3agf0e9wbr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 17 Aug 2021 22:49:26 -0400 Received: from m0098416.ppops.net (m0098416.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 17I2ZRU8182823; Tue, 17 Aug 2021 22:49:25 -0400 Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0b-001b2d01.pphosted.com with ESMTP id 3agf0e9wbf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 17 Aug 2021 22:49:25 -0400 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 17I2gFO1023498; Wed, 18 Aug 2021 02:49:24 GMT Received: from b03cxnp07029.gho.boulder.ibm.com (b03cxnp07029.gho.boulder.ibm.com [9.17.130.16]) by ppma03dal.us.ibm.com with ESMTP id 3ae5fe99h4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 18 Aug 2021 02:49:24 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp07029.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 17I2nNTc44958168 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 18 Aug 2021 02:49:23 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 49E1C78064; Wed, 18 Aug 2021 02:49:23 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E4DC67805C; Wed, 18 Aug 2021 02:49:19 +0000 (GMT) Received: from jarvis.lan (unknown [9.160.128.138]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Wed, 18 Aug 2021 02:49:19 +0000 (GMT) Message-ID: Subject: Re: [RFC PATCH 00/13] Add support for Mirror VM. From: James Bottomley To: Steve Rutherford Date: Tue, 17 Aug 2021 22:49:18 -0400 In-Reply-To: References: <0fcfafde-a690-f53a-01fc-542054948bb2@redhat.com> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.34.4 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: SfOHr1ARLf-i_TefRs9cqp3jSg-SmtUK X-Proofpoint-GUID: azzm65j96tps6cIfZhlnNAG2rT6H6lOX X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-08-17_09:2021-08-17, 2021-08-17 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 lowpriorityscore=0 bulkscore=0 mlxlogscore=535 priorityscore=1501 malwarescore=0 mlxscore=0 impostorscore=0 clxscore=1015 suspectscore=0 adultscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108180014 Received-SPF: pass client-ip=148.163.158.5; envelope-from=jejb@linux.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -19 X-Spam_score: -2.0 X-Spam_bar: -- X-Spam_report: (-2.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: jejb@linux.ibm.com Cc: Thomas Lendacky , Ashish Kalra , Brijesh Singh , "Habkost, Eduardo" , kvm , "S. Tsirkin, Michael" , Tobin Feldman-Fitzthum , Richard Henderson , qemu-devel , David Gilbert , Hubertus Franke , Dov Murik , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" On Tue, 2021-08-17 at 16:10 -0700, Steve Rutherford wrote: > On Tue, Aug 17, 2021 at 3:57 PM James Bottomley > wrote: > > Realistically, migration is becoming a royal pain, not just for > > confidential computing, but for virtual functions in general. I > > really think we should look at S3 suspend, where we shut down the > > drivers and then reattach on S3 resume as the potential pathway to > > getting migration working both for virtual functions and this use > > case. > > This type of migration seems a little bit less "live", which makes me > concerned about its performance characteristics. Well, there are too many scenarios we just fail at migration today. We need help from the guest to quiesce or shut down the interior devices, and S3 suspend seems to be the machine signal for that. I think in most clouds guests would accept some loss of "liveness" for a gain in reliability as long as we keep them within the SLA ... which is 5 minutes a year for 5 nines. Most failed migrations also instantly fail SLAs because of the recovery times involved so I don't see what's to be achieved by keeping the current "we can migrate sometimes" approach. James