From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C4F17C433F5 for ; Mon, 28 Feb 2022 02:32:52 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 4AC80818E6; Mon, 28 Feb 2022 02:32:52 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PReRctzFzAZ3; Mon, 28 Feb 2022 02:32:51 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id 6D273818C2; Mon, 28 Feb 2022 02:32:50 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 24171C0077; Mon, 28 Feb 2022 02:32:50 +0000 (UTC) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 45F56C001A for ; Mon, 28 Feb 2022 02:32:48 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 1C9A6818DC for ; Mon, 28 Feb 2022 02:32:48 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ia-Z8iM4TYEX for ; Mon, 28 Feb 2022 02:32:46 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id B4147818C2 for ; Mon, 28 Feb 2022 02:32:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1646015565; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Xrd1TskTNZt+2NHwTSaGAJFC8Qbcqr4kfERKct4YaoY=; b=BzAfnlI0c5s2XB8iU6/WVXDKAFe37kl3wFw6JgVrlXAZemiyaQftDkoUZTMbgCDnMak8MV hekP4FT4mXjmGwiVCqK9QELBg5U4hsRJzAJBWNfuSnhfTcPFOVAbD3zsRa78ihVVro0Z3R d1zAW0mGCW9aRkeQL6zN6ZgOtTLTzq4= Received: from mail-lj1-f198.google.com (mail-lj1-f198.google.com [209.85.208.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-25-kBxc3wrxNA2LBX4GobxfhA-1; Sun, 27 Feb 2022 21:32:41 -0500 X-MC-Unique: kBxc3wrxNA2LBX4GobxfhA-1 Received: by mail-lj1-f198.google.com with SMTP id x8-20020a2ea7c8000000b00246215e0fc3so4979497ljp.8 for ; Sun, 27 Feb 2022 18:32:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=Xrd1TskTNZt+2NHwTSaGAJFC8Qbcqr4kfERKct4YaoY=; b=LBDlm4BrlLaotcke59DJ5ozPmatpoybluIVwZtpQnGEkRK2I1k7FWBrZhpx9HFmTbU qkG1Vc+IuP4UQDDa5VHUCCbWIWW76CgS4QQFpiwkQgRLSdHDY3dqQBgYOSOcnmJKI+nP iqoPBETBUMiarYxsTtcpHRWRRltp17CCAgn+kLcrHcwbCn7h1H8ti16Ld7/19iOjK6KW dEmp+48nuiUntVNkWGlTfNS9uP2mo1AWu2t3EGNsL9mBgOrCV9hVdKtN6VymrcmWfDwq /0LKDA9ETk230jMlcmOMQZghHkfVilImFOHWrH/k26XO3zxZwNXpqOzRO+C0xJ5s3KR9 IYsQ== X-Gm-Message-State: AOAM5339Onv0YnZltIKcgFa6iEN3froVYTOkFP8e2tyVRJcRHbL+kBbb yxj5o/H+msfzmaSE9psJWsb3aopFQ/mCZZVw3qJlLKg47QD1XtxxobdKSWyRc2ulZw7hMBFkb68 aUOe7sWxdNzxYyIlPPvHZNYhMsJXXgx+1//clZKCqngdu1/UJCUbLM/1xWw== X-Received: by 2002:a05:6512:3d08:b0:43f:8f45:d670 with SMTP id d8-20020a0565123d0800b0043f8f45d670mr11761011lfv.587.1646015559252; Sun, 27 Feb 2022 18:32:39 -0800 (PST) X-Google-Smtp-Source: ABdhPJzJbERaRpVPuscRuwppVxrJVxBmLAxFbF5aRlwvmomeYFYfX+szdthw510q7eTiobfN1DFqzqMCx1YKtrZnhKc= X-Received: by 2002:a05:6512:3d08:b0:43f:8f45:d670 with SMTP id d8-20020a0565123d0800b0043f8f45d670mr11760986lfv.587.1646015558994; Sun, 27 Feb 2022 18:32:38 -0800 (PST) MIME-Version: 1.0 References: <20220227134111.3254066-1-eperezma@redhat.com> In-Reply-To: <20220227134111.3254066-1-eperezma@redhat.com> From: Jason Wang Date: Mon, 28 Feb 2022 10:32:27 +0800 Message-ID: Subject: Re: [PATCH v2 00/14] vDPA shadow virtqueue To: =?UTF-8?Q?Eugenio_P=C3=A9rez?= Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jasowang@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: "Michael S. Tsirkin" , qemu-devel , virtualization , Eli Cohen , Eric Blake , Parav Pandit , Cindy Lu , "Fangyi \(Eric\)" , Markus Armbruster , yebiaoxiang@huawei.com, Liuxiangdong , Laurent Vivier , Eduardo Habkost , Richard Henderson , Gautam Dawar , Xiao W Wang , Stefan Hajnoczi , Harpreet Singh Anand , Paolo Bonzini , Lingshan X-BeenThere: virtualization@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Linux virtualization List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: virtualization-bounces@lists.linux-foundation.org Sender: "Virtualization" T24gU3VuLCBGZWIgMjcsIDIwMjIgYXQgOTo0MiBQTSBFdWdlbmlvIFDDqXJleiA8ZXBlcmV6bWFA cmVkaGF0LmNvbT4gd3JvdGU6Cj4KPiBUaGlzIHNlcmllcyBlbmFibGUgc2hhZG93IHZpcnRxdWV1 ZSAoU1ZRKSBmb3Igdmhvc3QtdmRwYSBkZXZpY2VzLiBUaGlzCj4gaXMgaW50ZW5kZWQgYXMgYSBu ZXcgbWV0aG9kIG9mIHRyYWNraW5nIHRoZSBtZW1vcnkgdGhlIGRldmljZXMgdG91Y2gKPiBkdXJp bmcgYSBtaWdyYXRpb24gcHJvY2VzczogSW5zdGVhZCBvZiByZWxheSBvbiB2aG9zdCBkZXZpY2Un cyBkaXJ0eQo+IGxvZ2dpbmcgY2FwYWJpbGl0eSwgU1ZRIGludGVyY2VwdHMgdGhlIFZRIGRhdGFw bGFuZSBmb3J3YXJkaW5nIHRoZQo+IGRlc2NyaXB0b3JzIGJldHdlZW4gVk0gYW5kIGRldmljZS4g VGhpcyB3YXkgcWVtdSBpcyB0aGUgZWZmZWN0aXZlCj4gd3JpdGVyIG9mIGd1ZXN0cyBtZW1vcnks IGxpa2UgaW4gcWVtdSdzIHZpcnRpbyBkZXZpY2Ugb3BlcmF0aW9uLgo+Cj4gV2hlbiBTVlEgaXMg ZW5hYmxlZCBxZW11IG9mZmVycyBhIG5ldyB2aXJ0dWFsIGFkZHJlc3Mgc3BhY2UgdG8gdGhlCj4g ZGV2aWNlIHRvIHJlYWQgYW5kIHdyaXRlIGludG8sIGFuZCBpdCBtYXBzIG5ldyB2cmluZ3MgYW5k IHRoZSBndWVzdAo+IG1lbW9yeSBpbiBpdC4gU1ZRIGFsc28gaW50ZXJjZXB0cyBraWNrcyBhbmQg Y2FsbHMgYmV0d2VlbiB0aGUgZGV2aWNlCj4gYW5kIHRoZSBndWVzdC4gVXNlZCBidWZmZXJzIHJl bGF5IHdvdWxkIGNhdXNlIGRpcnR5IG1lbW9yeSBiZWluZwo+IHRyYWNrZWQuCj4KPiBUaGlzIGVm ZmVjdGl2ZWx5IG1lYW5zIHRoYXQgdkRQQSBkZXZpY2UgcGFzc3Rocm91Z2ggaXMgaW50ZXJjZXB0 ZWQgYnkKPiBxZW11LiBXaGlsZSBTVlEgc2hvdWxkIG9ubHkgYmUgZW5hYmxlZCBhdCBtaWdyYXRp b24gdGltZSwgdGhlIHN3aXRjaGluZwo+IGZyb20gcmVndWxhciBtb2RlIHRvIFNWUSBtb2RlIGlz IGxlZnQgZm9yIGEgZnV0dXJlIHNlcmllcy4KPgo+IEl0IGlzIGJhc2VkIG9uIHRoZSBpZGVhcyBv ZiBEUERLIFNXIGFzc2lzdGVkIExNLCBpbiB0aGUgc2VyaWVzIG9mCj4gRFBESydzIGh0dHBzOi8v cGF0Y2h3b3JrLmRwZGsub3JnL2NvdmVyLzQ4MzcwLyAuIEhvd2V2ZXIsIHRoZXNlIGRvZXMKPiBu b3QgbWFwIHRoZSBzaGFkb3cgdnEgaW4gZ3Vlc3QncyBWQSwgYnV0IGluIHFlbXUncy4KPgo+IEZv ciBxZW11IHRvIHVzZSBzaGFkb3cgdmlydHF1ZXVlcyB0aGUgZ3Vlc3QgdmlydGlvIGRyaXZlciBt dXN0IG5vdCB1c2UKPiBmZWF0dXJlcyBsaWtlIGV2ZW50X2lkeCwgaW5kaXJlY3QgZGVzY3JpcHRv cnMsIHBhY2tlZCBhbmQgaW5fb3JkZXIuCj4gVGhlc2UgZmVhdHVyZXMgYXJlIGVhc3kgdG8gaW1w bGVtZW50IG9uIHRvcCBvZiB0aGlzIGJhc2UsIGJ1dCBpcyBsZWZ0Cj4gZm9yIGEgZnV0dXJlIHNl cmllcyBmb3Igc2ltcGxpY2l0eS4KPgo+IFNWUSBuZWVkcyB0byBiZSBlbmFibGVkIGF0IHFlbXUg c3RhcnQgdGltZSB3aXRoIHZkcGEgY21kbGluZSBwYXJhbWV0ZXI6Cj4KPiAtbmV0ZGV2IHR5cGU9 dmhvc3QtdmRwYSx2aG9zdGRldj12aG9zdC12ZHBhLTAsaWQ9dmhvc3QtdmRwYTAseC1zdnE9b2Zm Cj4KPiBUaGUgZmlyc3QgdGhyZWUgcGF0Y2hlcyBlbmFibGVzIG5vdGlmaWNhdGlvbnMgZm9yd2Fy ZGluZyB3aXRoCj4gYXNzaXN0YW5jZSBvZiBxZW11LiBJdCdzIGVhc3kgdG8gZW5hYmxlIG9ubHkg dGhpcyBpZiB0aGUgcmVsZXZhbnQKPiBjbWRsaW5lIHBhcnQgb2YgdGhlIGxhc3QgcGF0Y2ggaXMg YXBwbGllZCBvbiB0b3Agb2YgdGhlc2UuCj4KPiBOZXh0IGZvdXIgcGF0Y2hlcyBpbXBsZW1lbnQg dGhlIGFjdHVhbCBidWZmZXIgZm9yd2FyZGluZy4gSG93ZXZlciwKPiBhZGRyZXNzIGFyZSBub3Qg dHJhbnNsYXRlZCBmcm9tIEhWQSBzbyB0aGV5IHdpbGwgbmVlZCBhIGhvc3QgZGV2aWNlIHdpdGgK PiBhbiBpb21tdSBhbGxvd2luZyB0aGVtIHRvIGFjY2VzcyBhbGwgb2YgdGhlIEhWQSByYW5nZS4K Pgo+IFRoZSBsYXN0IHBhcnQgb2YgdGhlIHNlcmllcyB1c2VzIHByb3Blcmx5IHRoZSBob3N0IGlv bW11LCBzbyBxZW11Cj4gY3JlYXRlcyBhIG5ldyBpb3ZhIGFkZHJlc3Mgc3BhY2UgaW4gdGhlIGRl dmljZSdzIHJhbmdlIGFuZCB0cmFuc2xhdGVzCj4gdGhlIGJ1ZmZlcnMgaW4gaXQuIEZpbmFsbHks IGl0IGFkZHMgdGhlIGNtZGxpbmUgcGFyYW1ldGVyLgo+Cj4gU29tZSBzaW1wbGUgcGVyZm9ybWFu Y2UgdGVzdHMgd2l0aCBuZXRwZXJmIHdlcmUgZG9uZS4gVGhleSB1c2VkIGEgbmVzdGVkCj4gZ3Vl c3Qgd2l0aCB2cF92ZHBhLCB2aG9zdC1rZXJuZWwgYXQgTDAgaG9zdC4gU3RhcnRpbmcgd2l0aCBu byBzdnEgYW5kIGEKPiBiYXNlbGluZSBhdmVyYWdlIG9mIH45OTgwLjEzTWJwczoKPiBSZWN2ICAg U2VuZCAgICBTZW5kCj4gU29ja2V0IFNvY2tldCAgTWVzc2FnZSAgRWxhcHNlZAo+IFNpemUgICBT aXplICAgIFNpemUgICAgIFRpbWUgICAgIFRocm91Z2hwdXQKPiBieXRlcyAgYnl0ZXMgICBieXRl cyAgICBzZWNzLiAgICAxMF42Yml0cy9zZWMKPgo+IDEzMTA3MiAgMTYzODQgIDE2Mzg0ICAgIDMw LjAxICAgIDk5MTAuNjEKPiAxMzEwNzIgIDE2Mzg0ICAxNjM4NCAgICAzMC4wMCAgICAxMDAzMC45 NAo+IDEzMTA3MiAgMTYzODQgIDE2Mzg0ICAgIDMwLjAxICAgIDk5OTguODQKPgo+IFRvIGVuYWJs ZSB0aGUgbm90aWZpY2F0aW9ucyBpbnRlcmNlcHRpb24gcmVkdWNlZCBwZXJmb3JtYW5jZSB0byBh bgo+IGF2ZXJhZ2Ugb2Ygfjk1NzcuNzNNYml0L3M6Cj4gUmVjdiAgIFNlbmQgICAgU2VuZAo+IFNv Y2tldCBTb2NrZXQgIE1lc3NhZ2UgIEVsYXBzZWQKPiBTaXplICAgU2l6ZSAgICBTaXplICAgICBU aW1lICAgICBUaHJvdWdocHV0Cj4gYnl0ZXMgIGJ5dGVzICAgYnl0ZXMgICAgc2Vjcy4gICAgMTBe NmJpdHMvc2VjCj4KPiAxMzEwNzIgIDE2Mzg0ICAxNjM4NCAgICAzMC4wMCAgICA5NTYzLjAzCj4g MTMxMDcyICAxNjM4NCAgMTYzODQgICAgMzAuMDEgICAgOTYyNi42NQo+IDEzMTA3MiAgMTYzODQg IDE2Mzg0ICAgIDMwLjAxICAgIDk1NDMuNTEKPgo+IEZpbmFsbHksIHRvIGVuYWJsZSBidWZmZXJz IGZvcndhcmRpbmcgcmVkdWNlZCB0aGUgdGhyb3VnaHB1dCBhZ2FpbiB0bwo+IH44OTAyLjkyTWJp dC9zOgo+IFJlY3YgICBTZW5kICAgIFNlbmQKPiBTb2NrZXQgU29ja2V0ICBNZXNzYWdlICBFbGFw c2VkCj4gU2l6ZSAgIFNpemUgICAgU2l6ZSAgICAgVGltZSAgICAgVGhyb3VnaHB1dAo+IGJ5dGVz ICBieXRlcyAgIGJ5dGVzICAgIHNlY3MuICAgIDEwXjZiaXRzL3NlYwo+Cj4gMTMxMDcyICAxNjM4 NCAgMTYzODQgICAgMzAuMDEgICAgODY0My4xOQo+IDEzMTA3MiAgMTYzODQgIDE2Mzg0ICAgIDMw LjAxICAgIDkwMzMuNTYKPiAxMzEwNzIgIDE2Mzg0ICAxNjM4NCAgICAzMC4wMSAgICA5MDMyLjAy Cj4KPiBIb3dldmVyLCBtYW55IHBlcmZvcm1hbmNlIGltcHJvdmVtZW50cyB3ZXJlIGxlZnQgb3V0 IG9mIHRoaXMgc2VyaWVzIGZvcgo+IHNpbXBsaWNpdHksIHNvIGRpZmZlcmVuY2UgaWYgcGVyZm9y bWFuY2Ugc2hvdWxkIHNocmluayBpbiB0aGUgZnV0dXJlLgoKSSB0aGluayB0aGUgcGVyZm9ybWFu Y2Ugc2hvdWxkIGJlIGFjY2VwdGFibGUgYXMgYSBzdGFydC4KCj4KPiBDb21tZW50cyBhcmUgd2Vs Y29tZS4KPgo+IFRPRE8gaW4gZnV0dXJlIHNlcmllczoKPiAqIEV2ZW50LCBpbmRpcmVjdCwgcGFj a2VkLCBhbmQgb3RoZXJzIGZlYXR1cmVzIG9mIHZpcnRpby4KPiAqIFRvIHN1cHBvcnQgZGlmZmVy ZW50IHNldCBvZiBmZWF0dXJlcyBiZXR3ZWVuIHRoZSBkZXZpY2U8LT5TVlEgYW5kIHRoZQo+ICAg U1ZRPC0+Z3Vlc3QgY29tbXVuaWNhdGlvbi4KPiAqIFN1cHBvcnQgb2YgZGV2aWNlIGhvc3Qgbm90 aWZpZXIgbWVtb3J5IHJlZ2lvbnMuCj4gKiBUbyBzZXBwYXJhdGUgYnVmZmVycyBmb3J3YXJkaW5n IGluIGl0cyBvd24gQUlPIGNvbnRleHQsIHNvIHdlIGNhbgo+ICAgdGhyb3cgbW9yZSB0aHJlYWRz IHRvIHRoYXQgdGFzayBhbmQgd2UgZG9uJ3QgbmVlZCB0byBzdG9wIHRoZSBtYWluCj4gICBldmVu dCBsb29wLgo+ICogU3VwcG9ydCBtdWx0aXF1ZXVlIHZpcnRpby1uZXQgdmRwYS4KPiAqIFByb3Bl ciBkb2N1bWVudGF0aW9uLgo+Cj4gQ2hhbmdlcyBmcm9tIHYxOgo+ICogRmVhdHVyZSBzZXQgYXQg ZGV2aWNlLT5TVlEgaXMgbm93IHRoZSBzYW1lIGFzIFNWUS0+Z3Vlc3QuCj4gKiBTaXplIG9mIFNW USBpcyBub3QgbWF4IGF2YWlsYWJsZSBkZXZpY2Ugc2l6ZSBhbnltb3JlLCBidXQgZ3Vlc3Qncwo+ ICAgbmVnb3RpYXRlZC4KPiAqIEFkZCBWSE9TVF9GSUxFX1VOQklORCBraWNrIGFuZCBjYWxsIGZk IHRyZWF0bWVudC4KPiAqIE1ha2UgU1ZRIGEgcHVibGljIHN0cnVjdAo+ICogQ29tZSBiYWNrIHRv IHByZXZpb3VzIGFwcHJvYWNoIHRvIGlvdmEtdHJlZQo+ICogU29tZSBhc3NlcnRpb25zIGFyZSBu b3cgZmFpbCBwYXRocy4gU29tZSBlcnJvcnMgYXJlIG5vdyBsb2dfZ3Vlc3QuCj4gKiBPbmx5IG1h c2sgX0ZfTE9HIGZlYXR1cmUgYXQgdmRwYV9zZXRfZmVhdHVyZXMgc3ZxIGVuYWJsZSBwYXRoLgo+ ICogUmVmYWN0b3Igc29tZSBlcnJvcnMgYW5kIG1lc3NhZ2VzLiBBZGQgbWlzc2luZyBlcnJvciB1 bndpbmRpbmdzLgo+ICogQWRkIG1lbW9yeSBiYXJyaWVyIGF0IF9GX05PX05PVElGWSBzZXQuCj4g KiBTdG9wIGNoZWNraW5nIGZvciBmZWF0dXJlcyBmbGFncyBvdXQgb2YgdHJhbnNwb3J0IHJhbmdl Lgo+IHYxIGxpbms6Cj4gaHR0cHM6Ly9sb3JlLmtlcm5lbC5vcmcvdmlydHVhbGl6YXRpb24vN2Q4 NmM3MTUtNmQ3MS04YTI3LTkxZjUtOGQ0N2I3MWUzMjAxQHJlZGhhdC5jb20vCj4KPiBDaGFuZ2Vz IGZyb20gdjQgUkZDOgo+ICogU3VwcG9ydCBvZiBhbGxvY2F0aW5nIC8gZnJlZWluZyBpb3ZhIHJh bmdlcyBpbiBJT1ZBIHRyZWUuIEV4dGVuZGluZwo+ICAgYWxyZWFkeSBwcmVzZW50IGlvdmEtdHJl ZSBmb3IgdGhhdC4KPiAqIFByb3BlciB2YWxpZGF0aW9uIG9mIGd1ZXN0IGZlYXR1cmVzLiBOb3cg U1ZRIGNhbiBuZWdvdGlhdGUgYQo+ICAgZGlmZmVyZW50IHNldCBvZiBmZWF0dXJlcyB3aXRoIHRo ZSBkZXZpY2Ugd2hlbiBlbmFibGVkLgo+ICogU3VwcG9ydCBvZiBob3N0IG5vdGlmaWVycyBtZW1v cnkgcmVnaW9ucwo+ICogSGFuZGxpbmcgb2YgU1ZRIGZ1bGwgcXVldWUgaW4gY2FzZSBndWVzdCdz IGRlc2NyaXB0b3JzIHNwYW4gdG8KPiAgIGRpZmZlcmVudCBtZW1vcnkgcmVnaW9ucyAocWVtdSdz IFZBIGNodW5rcykuCj4gKiBGbHVzaCBwZW5kaW5nIHVzZWQgYnVmZmVycyBhdCBlbmQgb2YgU1ZR IG9wZXJhdGlvbi4KPiAqIFFNUCBjb21tYW5kIG5vdyBsb29rcyBieSBOZXRDbGllbnRTdGF0ZSBu YW1lLiBPdGhlciBkZXZpY2VzIHdpbGwgbmVlZAo+ICAgdG8gaW1wbGVtZW50IGl0J3Mgd2F5IHRv IGVuYWJsZSB2ZHBhLgo+ICogUmVuYW1lIFFNUCBjb21tYW5kIHRvIHNldCwgc28gaXQgbG9va3Mg bW9yZSBsaWtlIGEgd2F5IG9mIHdvcmtpbmcKPiAqIEJldHRlciB1c2Ugb2YgcWVtdSBlcnJvciBz eXN0ZW0KPiAqIE1ha2UgYSBmZXcgYXNzZXJ0aW9ucyBwcm9wZXIgZXJyb3ItaGFuZGxpbmcgcGF0 aHMuCj4gKiBBZGQgbW9yZSBkb2N1bWVudGF0aW9uCj4gKiBMZXNzIGNvdXBsaW5nIG9mIHZpcnRp byAvIHZob3N0LCB0aGF0IGNvdWxkIGNhdXNlIGZyaWN0aW9uIG9uIGNoYW5nZXMKPiAqIEFkZHJl c3NlZCBtYW55IG90aGVyIHNtYWxsIGNvbW1lbnRzIGFuZCBzbWFsbCBmaXhlcy4KPgo+IENoYW5n ZXMgZnJvbSB2MyBSRkM6Cj4gICAqIE1vdmUgZXZlcnl0aGluZyB0byB2aG9zdC12ZHBhIGJhY2tl bmQuIEEgYmlnIGNoYW5nZSwgdGhpcyBhbGxvd2VkCj4gICAgIHNvbWUgY2xlYW51cCBidXQgbW9y ZSBjb2RlIGhhcyBiZWVuIGFkZGVkIGluIG90aGVyIHBsYWNlcy4KPiAgICogTW9yZSB1c2Ugb2Yg Z2xpYiB1dGlsaXRpZXMsIGVzcGVjaWFsbHkgdG8gbWFuYWdlIG1lbW9yeS4KPiB2MyBsaW5rOgo+ IGh0dHBzOi8vbGlzdHMubm9uZ251Lm9yZy9hcmNoaXZlL2h0bWwvcWVtdS1kZXZlbC8yMDIxLTA1 L21zZzA2MDMyLmh0bWwKPgo+IENoYW5nZXMgZnJvbSB2MiBSRkM6Cj4gICAqIEFkZGluZyB2aG9z dC12ZHBhIGRldmljZXMgc3VwcG9ydAo+ICAgKiBGaXhlZCBzb21lIG1lbW9yeSBsZWFrcyBwb2lu dGVkIGJ5IGRpZmZlcmVudCBjb21tZW50cwo+IHYyIGxpbms6Cj4gaHR0cHM6Ly9saXN0cy5ub25n bnUub3JnL2FyY2hpdmUvaHRtbC9xZW11LWRldmVsLzIwMjEtMDMvbXNnMDU2MDAuaHRtbAo+Cj4g Q2hhbmdlcyBmcm9tIHYxIFJGQzoKPiAgICogVXNlIFFNUCBpbnN0ZWFkIG9mIG1pZ3JhdGlvbiB0 byBzdGFydCBTVlEgbW9kZS4KPiAgICogT25seSBhY2NlcHRpbmcgSU9NTVUgZGV2aWNlcywgY2xv c2VyIGJlaGF2aW9yIHdpdGggdGFyZ2V0IGRldmljZXMKPiAgICAgKHZEUEEpCj4gICAqIEZpeCBp bnZhbGlkIG1hc2tpbmcvdW5tYXNraW5nIG9mIHZob3N0IGNhbGwgZmQuCj4gICAqIFVzZSBvZiBw cm9wZXIgbWV0aG9kcyBmb3Igc3luY2hyb25pemF0aW9uLgo+ICAgKiBObyBuZWVkIHRvIG1vZGlm eSBWaXJ0SU8gZGV2aWNlIGNvZGUsIGFsbCBvZiB0aGUgY2hhbmdlcyBhcmUKPiAgICAgY29udGFp bmVkIGluIHZob3N0IGNvZGUuCj4gICAqIERlbGV0ZSBzdXBlcmZsdW91cyBjb2RlLgo+ICAgKiBB biBpbnRlcm1lZGlhdGUgUkZDIHdhcyBzZW50IHdpdGggb25seSB0aGUgbm90aWZpY2F0aW9ucyBm b3J3YXJkaW5nCj4gICAgIGNoYW5nZXMuIEl0IGNhbiBiZSBzZWVuIGluCj4gICAgIGh0dHBzOi8v cGF0Y2hldy5vcmcvUUVNVS8yMDIxMDEyOTIwNTQxNS44NzYyOTAtMS1lcGVyZXptYUByZWRoYXQu Y29tLwo+IHYxIGxpbms6Cj4gaHR0cHM6Ly9saXN0cy5nbnUub3JnL2FyY2hpdmUvaHRtbC9xZW11 LWRldmVsLzIwMjAtMTEvbXNnMDUzNzIuaHRtbAo+Cj4gRXVnZW5pbyBQw6lyZXogKDIwKToKPiAg ICAgICB2aXJ0aW86IEFkZCBWSVJUSU9fRl9RVUVVRV9TVEFURQo+ICAgICAgIHZpcnRpby1uZXQ6 IEhvbm9yIFZJUlRJT19DT05GSUdfU19ERVZJQ0VfU1RPUFBFRAo+ICAgICAgIHZpcnRpbzogQWRk IHZpcnRpb19xdWV1ZV9pc19ob3N0X25vdGlmaWVyX2VuYWJsZWQKPiAgICAgICB2aG9zdDogTWFr ZSB2aG9zdF92aXJ0cXVldWVfe3N0YXJ0LHN0b3B9IHB1YmxpYwo+ICAgICAgIHZob3N0OiBBZGQg eC12aG9zdC1lbmFibGUtc2hhZG93LXZxIHFtcAo+ICAgICAgIHZob3N0OiBBZGQgVmhvc3RTaGFk b3dWaXJ0cXVldWUKPiAgICAgICB2ZHBhOiBSZWdpc3RlciB2ZHBhIGRldmljZXMgaW4gYSBsaXN0 Cj4gICAgICAgdmhvc3Q6IFJvdXRlIGd1ZXN0LT5ob3N0IG5vdGlmaWNhdGlvbiB0aHJvdWdoIHNo YWRvdyB2aXJ0cXVldWUKPiAgICAgICBBZGQgdmhvc3Rfc3ZxX2dldF9zdnFfY2FsbF9ub3RpZmll cgo+ICAgICAgIEFkZCB2aG9zdF9zdnFfc2V0X2d1ZXN0X2NhbGxfbm90aWZpZXIKPiAgICAgICB2 ZHBhOiBTYXZlIGNhbGxfZmQgaW4gdmhvc3QtdmRwYQo+ICAgICAgIHZob3N0LXZkcGE6IFRha2Ug aW50byBhY2NvdW50IFNWUSBpbiB2aG9zdF92ZHBhX3NldF92cmluZ19jYWxsCj4gICAgICAgdmhv c3Q6IFJvdXRlIGhvc3QtPmd1ZXN0IG5vdGlmaWNhdGlvbiB0aHJvdWdoIHNoYWRvdyB2aXJ0cXVl dWUKPiAgICAgICB2aXJ0aW86IEFkZCB2aG9zdF9zaGFkb3dfdnFfZ2V0X3ZyaW5nX2FkZHIKPiAg ICAgICB2ZHBhOiBTYXZlIGhvc3QgYW5kIGd1ZXN0IGZlYXR1cmVzCj4gICAgICAgdmhvc3Q6IEFk ZCB2aG9zdF9zdnFfdmFsaWRfZGV2aWNlX2ZlYXR1cmVzIHRvIHNoYWRvdyB2cQo+ICAgICAgIHZo b3N0OiBTaGFkb3cgdmlydHF1ZXVlIGJ1ZmZlcnMgZm9yd2FyZGluZwo+ICAgICAgIHZob3N0OiBB ZGQgVmhvc3RJT1ZBVHJlZQo+ICAgICAgIHZob3N0OiBVc2UgYSB0cmVlIHRvIHN0b3JlIG1lbW9y eSBtYXBwaW5ncwo+ICAgICAgIHZkcGE6IEFkZCBjdXN0b20gSU9UTEIgdHJhbnNsYXRpb25zIHRv IFNWUQoKVGhpcyBsaXN0IHNlZW1zIHdyb25nIGJ0dyA6KQoKVGhhbmtzCgo+Cj4gRXVnZW5pbyBQ w6lyZXogKDE0KToKPiAgIHZob3N0OiBBZGQgVmhvc3RTaGFkb3dWaXJ0cXVldWUKPiAgIHZob3N0 OiBBZGQgU2hhZG93IFZpcnRRdWV1ZSBraWNrIGZvcndhcmRpbmcgY2FwYWJpbGl0aWVzCj4gICB2 aG9zdDogQWRkIFNoYWRvdyBWaXJ0UXVldWUgY2FsbCBmb3J3YXJkaW5nIGNhcGFiaWxpdGllcwo+ ICAgdmhvc3Q6IEFkZCB2aG9zdF9zdnFfdmFsaWRfZmVhdHVyZXMgdG8gc2hhZG93IHZxCj4gICB2 aXJ0aW86IEFkZCB2aG9zdF9zaGFkb3dfdnFfZ2V0X3ZyaW5nX2FkZHIKPiAgIHZkcGE6IGFkYXB0 IHZob3N0X29wcyBjYWxsYmFja3MgdG8gc3ZxCj4gICB2aG9zdDogU2hhZG93IHZpcnRxdWV1ZSBi dWZmZXJzIGZvcndhcmRpbmcKPiAgIHV0aWw6IEFkZCBpb3ZhX3RyZWVfYWxsb2MKPiAgIHZob3N0 OiBBZGQgVmhvc3RJT1ZBVHJlZQo+ICAgdmRwYTogQWRkIGN1c3RvbSBJT1RMQiB0cmFuc2xhdGlv bnMgdG8gU1ZRCj4gICB2ZHBhOiBBZGFwdCB2aG9zdF92ZHBhX2dldF92cmluZ19iYXNlIHRvIFNW UQo+ICAgdmRwYTogTmV2ZXIgc2V0IGxvZ19iYXNlIGFkZHIgaWYgU1ZRIGlzIGVuYWJsZWQKPiAg IHZkcGE6IEV4cG9zZSBWSE9TVF9GX0xPR19BTEwgb24gU1ZRCj4gICB2ZHBhOiBBZGQgeC1zdnEg dG8gTmV0ZGV2Vmhvc3RWRFBBT3B0aW9ucwo+Cj4gIHFhcGkvbmV0Lmpzb24gICAgICAgICAgICAg ICAgICAgICAgfCAgIDUgKy0KPiAgaHcvdmlydGlvL3Zob3N0LWlvdmEtdHJlZS5oICAgICAgICB8 ICAyNyArKwo+ICBody92aXJ0aW8vdmhvc3Qtc2hhZG93LXZpcnRxdWV1ZS5oIHwgIDkwICsrKysK PiAgaW5jbHVkZS9ody92aXJ0aW8vdmhvc3QtdmRwYS5oICAgICB8ICAgOCArCj4gIGluY2x1ZGUv cWVtdS9pb3ZhLXRyZWUuaCAgICAgICAgICAgfCAgMTggKwo+ICBody92aXJ0aW8vdmhvc3QtaW92 YS10cmVlLmMgICAgICAgIHwgMTU1ICsrKysrKysKPiAgaHcvdmlydGlvL3Zob3N0LXNoYWRvdy12 aXJ0cXVldWUuYyB8IDYzMiArKysrKysrKysrKysrKysrKysrKysrKysrKysrKwo+ICBody92aXJ0 aW8vdmhvc3QtdmRwYS5jICAgICAgICAgICAgIHwgNTUxICsrKysrKysrKysrKysrKysrKysrKysr Ky0KPiAgbmV0L3Zob3N0LXZkcGEuYyAgICAgICAgICAgICAgICAgICB8ICA0OCArKy0KPiAgdXRp bC9pb3ZhLXRyZWUuYyAgICAgICAgICAgICAgICAgICB8IDEzMyArKysrKysKPiAgaHcvdmlydGlv L21lc29uLmJ1aWxkICAgICAgICAgICAgICB8ICAgMiArLQo+ICAxMSBmaWxlcyBjaGFuZ2VkLCAx NjQ0IGluc2VydGlvbnMoKyksIDI1IGRlbGV0aW9ucygtKQo+ICBjcmVhdGUgbW9kZSAxMDA2NDQg aHcvdmlydGlvL3Zob3N0LWlvdmEtdHJlZS5oCj4gIGNyZWF0ZSBtb2RlIDEwMDY0NCBody92aXJ0 aW8vdmhvc3Qtc2hhZG93LXZpcnRxdWV1ZS5oCj4gIGNyZWF0ZSBtb2RlIDEwMDY0NCBody92aXJ0 aW8vdmhvc3QtaW92YS10cmVlLmMKPiAgY3JlYXRlIG1vZGUgMTAwNjQ0IGh3L3ZpcnRpby92aG9z dC1zaGFkb3ctdmlydHF1ZXVlLmMKPgo+IC0tCj4gMi4yNy4wCj4KPgoKX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KVmlydHVhbGl6YXRpb24gbWFpbGluZyBs aXN0ClZpcnR1YWxpemF0aW9uQGxpc3RzLmxpbnV4LWZvdW5kYXRpb24ub3JnCmh0dHBzOi8vbGlz dHMubGludXhmb3VuZGF0aW9uLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3ZpcnR1YWxpemF0aW9u From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BA2E7C433F5 for ; Mon, 28 Feb 2022 02:33:43 +0000 (UTC) Received: from localhost ([::1]:48480 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nOVr4-0000Uk-HP for qemu-devel@archiver.kernel.org; Sun, 27 Feb 2022 21:33:42 -0500 Received: from eggs.gnu.org ([209.51.188.92]:35850) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nOVqF-0008Dz-So for qemu-devel@nongnu.org; Sun, 27 Feb 2022 21:32:51 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:47153) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nOVqB-0003qF-Gm for qemu-devel@nongnu.org; Sun, 27 Feb 2022 21:32:50 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1646015565; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Xrd1TskTNZt+2NHwTSaGAJFC8Qbcqr4kfERKct4YaoY=; b=BzAfnlI0c5s2XB8iU6/WVXDKAFe37kl3wFw6JgVrlXAZemiyaQftDkoUZTMbgCDnMak8MV hekP4FT4mXjmGwiVCqK9QELBg5U4hsRJzAJBWNfuSnhfTcPFOVAbD3zsRa78ihVVro0Z3R d1zAW0mGCW9aRkeQL6zN6ZgOtTLTzq4= Received: from mail-lj1-f200.google.com (mail-lj1-f200.google.com [209.85.208.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-450-j68NmgDgMoO_QAkp17bG6Q-1; Sun, 27 Feb 2022 21:32:41 -0500 X-MC-Unique: j68NmgDgMoO_QAkp17bG6Q-1 Received: by mail-lj1-f200.google.com with SMTP id n9-20020a2e82c9000000b002435af2e8b9so4938995ljh.20 for ; Sun, 27 Feb 2022 18:32:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=Xrd1TskTNZt+2NHwTSaGAJFC8Qbcqr4kfERKct4YaoY=; b=66bTl3QHQE3QzQYiHxwCIoOgZdLsPr+/gw5aSBAMamlET+NjAAVo+ad2FOr1Ma+HBS rEEu0QOJQdaKqF4e8OghpWvzF4s+tFfs6ANBccUsVtCrbMEEfCaFoqR83s45jIFY8fYL f1rMb4wV7IO42xkzEAtvnJnzUgUoP/TNgAcsNylpN8Akj41zBTTNjme7Ao42EzT3I+JG 9uGC0upyWF4uIy4CVZZfdzi3wGpOaoXABwmUd/+qZbY8Hk6sean/ckyQwAArZW2MbBtd g3iPXjB7mw79NgvjPI2C0SvZW0Rz0F1kIiEss5VcSP1tixFdd3S/r82/s2IR7rBiFX48 zL9g== X-Gm-Message-State: AOAM533P3lHrMxRWJi4vSLJgFEoS9Zz9nL/0R3uqwKjpJhKycrAOVzaJ Dbz2d7Edzh6llkUYD43m+X/eT6FJM8hlDnixOkvsyEM1804dUOmBhBnJCtqx75n50VJ7OwyfeXx 137vO+fZak2OlN9vhijP0LVDzImrWyGA= X-Received: by 2002:a05:6512:3d08:b0:43f:8f45:d670 with SMTP id d8-20020a0565123d0800b0043f8f45d670mr11761024lfv.587.1646015559358; Sun, 27 Feb 2022 18:32:39 -0800 (PST) X-Google-Smtp-Source: ABdhPJzJbERaRpVPuscRuwppVxrJVxBmLAxFbF5aRlwvmomeYFYfX+szdthw510q7eTiobfN1DFqzqMCx1YKtrZnhKc= X-Received: by 2002:a05:6512:3d08:b0:43f:8f45:d670 with SMTP id d8-20020a0565123d0800b0043f8f45d670mr11760986lfv.587.1646015558994; Sun, 27 Feb 2022 18:32:38 -0800 (PST) MIME-Version: 1.0 References: <20220227134111.3254066-1-eperezma@redhat.com> In-Reply-To: <20220227134111.3254066-1-eperezma@redhat.com> From: Jason Wang Date: Mon, 28 Feb 2022 10:32:27 +0800 Message-ID: Subject: Re: [PATCH v2 00/14] vDPA shadow virtqueue To: =?UTF-8?Q?Eugenio_P=C3=A9rez?= Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jasowang@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=170.10.133.124; envelope-from=jasowang@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Michael S. Tsirkin" , qemu-devel , Peter Xu , virtualization , Eli Cohen , Eric Blake , Parav Pandit , Cindy Lu , "Fangyi \(Eric\)" , Markus Armbruster , yebiaoxiang@huawei.com, Liuxiangdong , Stefano Garzarella , Laurent Vivier , Eduardo Habkost , Richard Henderson , Gautam Dawar , Xiao W Wang , Stefan Hajnoczi , Juan Quintela , Harpreet Singh Anand , Paolo Bonzini , Lingshan Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" On Sun, Feb 27, 2022 at 9:42 PM Eugenio P=C3=A9rez wr= ote: > > This series enable shadow virtqueue (SVQ) for vhost-vdpa devices. This > is intended as a new method of tracking the memory the devices touch > during a migration process: Instead of relay on vhost device's dirty > logging capability, SVQ intercepts the VQ dataplane forwarding the > descriptors between VM and device. This way qemu is the effective > writer of guests memory, like in qemu's virtio device operation. > > When SVQ is enabled qemu offers a new virtual address space to the > device to read and write into, and it maps new vrings and the guest > memory in it. SVQ also intercepts kicks and calls between the device > and the guest. Used buffers relay would cause dirty memory being > tracked. > > This effectively means that vDPA device passthrough is intercepted by > qemu. While SVQ should only be enabled at migration time, the switching > from regular mode to SVQ mode is left for a future series. > > It is based on the ideas of DPDK SW assisted LM, in the series of > DPDK's https://patchwork.dpdk.org/cover/48370/ . However, these does > not map the shadow vq in guest's VA, but in qemu's. > > For qemu to use shadow virtqueues the guest virtio driver must not use > features like event_idx, indirect descriptors, packed and in_order. > These features are easy to implement on top of this base, but is left > for a future series for simplicity. > > SVQ needs to be enabled at qemu start time with vdpa cmdline parameter: > > -netdev type=3Dvhost-vdpa,vhostdev=3Dvhost-vdpa-0,id=3Dvhost-vdpa0,x-svq= =3Doff > > The first three patches enables notifications forwarding with > assistance of qemu. It's easy to enable only this if the relevant > cmdline part of the last patch is applied on top of these. > > Next four patches implement the actual buffer forwarding. However, > address are not translated from HVA so they will need a host device with > an iommu allowing them to access all of the HVA range. > > The last part of the series uses properly the host iommu, so qemu > creates a new iova address space in the device's range and translates > the buffers in it. Finally, it adds the cmdline parameter. > > Some simple performance tests with netperf were done. They used a nested > guest with vp_vdpa, vhost-kernel at L0 host. Starting with no svq and a > baseline average of ~9980.13Mbps: > Recv Send Send > Socket Socket Message Elapsed > Size Size Size Time Throughput > bytes bytes bytes secs. 10^6bits/sec > > 131072 16384 16384 30.01 9910.61 > 131072 16384 16384 30.00 10030.94 > 131072 16384 16384 30.01 9998.84 > > To enable the notifications interception reduced performance to an > average of ~9577.73Mbit/s: > Recv Send Send > Socket Socket Message Elapsed > Size Size Size Time Throughput > bytes bytes bytes secs. 10^6bits/sec > > 131072 16384 16384 30.00 9563.03 > 131072 16384 16384 30.01 9626.65 > 131072 16384 16384 30.01 9543.51 > > Finally, to enable buffers forwarding reduced the throughput again to > ~8902.92Mbit/s: > Recv Send Send > Socket Socket Message Elapsed > Size Size Size Time Throughput > bytes bytes bytes secs. 10^6bits/sec > > 131072 16384 16384 30.01 8643.19 > 131072 16384 16384 30.01 9033.56 > 131072 16384 16384 30.01 9032.02 > > However, many performance improvements were left out of this series for > simplicity, so difference if performance should shrink in the future. I think the performance should be acceptable as a start. > > Comments are welcome. > > TODO in future series: > * Event, indirect, packed, and others features of virtio. > * To support different set of features between the device<->SVQ and the > SVQ<->guest communication. > * Support of device host notifier memory regions. > * To sepparate buffers forwarding in its own AIO context, so we can > throw more threads to that task and we don't need to stop the main > event loop. > * Support multiqueue virtio-net vdpa. > * Proper documentation. > > Changes from v1: > * Feature set at device->SVQ is now the same as SVQ->guest. > * Size of SVQ is not max available device size anymore, but guest's > negotiated. > * Add VHOST_FILE_UNBIND kick and call fd treatment. > * Make SVQ a public struct > * Come back to previous approach to iova-tree > * Some assertions are now fail paths. Some errors are now log_guest. > * Only mask _F_LOG feature at vdpa_set_features svq enable path. > * Refactor some errors and messages. Add missing error unwindings. > * Add memory barrier at _F_NO_NOTIFY set. > * Stop checking for features flags out of transport range. > v1 link: > https://lore.kernel.org/virtualization/7d86c715-6d71-8a27-91f5-8d47b71e32= 01@redhat.com/ > > Changes from v4 RFC: > * Support of allocating / freeing iova ranges in IOVA tree. Extending > already present iova-tree for that. > * Proper validation of guest features. Now SVQ can negotiate a > different set of features with the device when enabled. > * Support of host notifiers memory regions > * Handling of SVQ full queue in case guest's descriptors span to > different memory regions (qemu's VA chunks). > * Flush pending used buffers at end of SVQ operation. > * QMP command now looks by NetClientState name. Other devices will need > to implement it's way to enable vdpa. > * Rename QMP command to set, so it looks more like a way of working > * Better use of qemu error system > * Make a few assertions proper error-handling paths. > * Add more documentation > * Less coupling of virtio / vhost, that could cause friction on changes > * Addressed many other small comments and small fixes. > > Changes from v3 RFC: > * Move everything to vhost-vdpa backend. A big change, this allowed > some cleanup but more code has been added in other places. > * More use of glib utilities, especially to manage memory. > v3 link: > https://lists.nongnu.org/archive/html/qemu-devel/2021-05/msg06032.html > > Changes from v2 RFC: > * Adding vhost-vdpa devices support > * Fixed some memory leaks pointed by different comments > v2 link: > https://lists.nongnu.org/archive/html/qemu-devel/2021-03/msg05600.html > > Changes from v1 RFC: > * Use QMP instead of migration to start SVQ mode. > * Only accepting IOMMU devices, closer behavior with target devices > (vDPA) > * Fix invalid masking/unmasking of vhost call fd. > * Use of proper methods for synchronization. > * No need to modify VirtIO device code, all of the changes are > contained in vhost code. > * Delete superfluous code. > * An intermediate RFC was sent with only the notifications forwarding > changes. It can be seen in > https://patchew.org/QEMU/20210129205415.876290-1-eperezma@redhat.com/ > v1 link: > https://lists.gnu.org/archive/html/qemu-devel/2020-11/msg05372.html > > Eugenio P=C3=A9rez (20): > virtio: Add VIRTIO_F_QUEUE_STATE > virtio-net: Honor VIRTIO_CONFIG_S_DEVICE_STOPPED > virtio: Add virtio_queue_is_host_notifier_enabled > vhost: Make vhost_virtqueue_{start,stop} public > vhost: Add x-vhost-enable-shadow-vq qmp > vhost: Add VhostShadowVirtqueue > vdpa: Register vdpa devices in a list > vhost: Route guest->host notification through shadow virtqueue > Add vhost_svq_get_svq_call_notifier > Add vhost_svq_set_guest_call_notifier > vdpa: Save call_fd in vhost-vdpa > vhost-vdpa: Take into account SVQ in vhost_vdpa_set_vring_call > vhost: Route host->guest notification through shadow virtqueue > virtio: Add vhost_shadow_vq_get_vring_addr > vdpa: Save host and guest features > vhost: Add vhost_svq_valid_device_features to shadow vq > vhost: Shadow virtqueue buffers forwarding > vhost: Add VhostIOVATree > vhost: Use a tree to store memory mappings > vdpa: Add custom IOTLB translations to SVQ This list seems wrong btw :) Thanks > > Eugenio P=C3=A9rez (14): > vhost: Add VhostShadowVirtqueue > vhost: Add Shadow VirtQueue kick forwarding capabilities > vhost: Add Shadow VirtQueue call forwarding capabilities > vhost: Add vhost_svq_valid_features to shadow vq > virtio: Add vhost_shadow_vq_get_vring_addr > vdpa: adapt vhost_ops callbacks to svq > vhost: Shadow virtqueue buffers forwarding > util: Add iova_tree_alloc > vhost: Add VhostIOVATree > vdpa: Add custom IOTLB translations to SVQ > vdpa: Adapt vhost_vdpa_get_vring_base to SVQ > vdpa: Never set log_base addr if SVQ is enabled > vdpa: Expose VHOST_F_LOG_ALL on SVQ > vdpa: Add x-svq to NetdevVhostVDPAOptions > > qapi/net.json | 5 +- > hw/virtio/vhost-iova-tree.h | 27 ++ > hw/virtio/vhost-shadow-virtqueue.h | 90 ++++ > include/hw/virtio/vhost-vdpa.h | 8 + > include/qemu/iova-tree.h | 18 + > hw/virtio/vhost-iova-tree.c | 155 +++++++ > hw/virtio/vhost-shadow-virtqueue.c | 632 +++++++++++++++++++++++++++++ > hw/virtio/vhost-vdpa.c | 551 ++++++++++++++++++++++++- > net/vhost-vdpa.c | 48 ++- > util/iova-tree.c | 133 ++++++ > hw/virtio/meson.build | 2 +- > 11 files changed, 1644 insertions(+), 25 deletions(-) > create mode 100644 hw/virtio/vhost-iova-tree.h > create mode 100644 hw/virtio/vhost-shadow-virtqueue.h > create mode 100644 hw/virtio/vhost-iova-tree.c > create mode 100644 hw/virtio/vhost-shadow-virtqueue.c > > -- > 2.27.0 > >