From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D4F6C433FE for ; Wed, 13 Oct 2021 03:27:33 +0000 (UTC) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CFCDD603E8 for ; Wed, 13 Oct 2021 03:27:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org CFCDD603E8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 7C76E407C5; Wed, 13 Oct 2021 03:27:32 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 99-GncgxyM4j; Wed, 13 Oct 2021 03:27:31 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id 50F1C407C3; Wed, 13 Oct 2021 03:27:30 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0BD33C0011; Wed, 13 Oct 2021 03:27:30 +0000 (UTC) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 96C29C000D for ; Wed, 13 Oct 2021 03:27:28 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 90E90401FC for ; Wed, 13 Oct 2021 03:27:28 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eYbErVgMVo0w for ; Wed, 13 Oct 2021 03:27:26 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 573814012C for ; Wed, 13 Oct 2021 03:27:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1634095644; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=s+D8fjAlWfOwVxTjXsqSU9ga2khi1k6Zax2Zs/uECd0=; b=AI4WWBlmI63//p+fAAjX+WKZNjq3GRWYqFuVUG1KxCd4D6Sy7wOaPpiv+rWLeapHieEj99 pek2abkb0bDpeaR9OX+GaNwaE8rRQbeWuas5UuIRDRhbrGpy9b8e3gKxt3U231G+DHUILG DH9Oz6MMmT9zKDuLA6L2Qj3IxjIKYaY= Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-126-MNph110HOlmsToceoSpzsg-1; Tue, 12 Oct 2021 23:27:23 -0400 X-MC-Unique: MNph110HOlmsToceoSpzsg-1 Received: by mail-pl1-f199.google.com with SMTP id h3-20020a170902704300b0013dbfc88e14so379240plt.13 for ; Tue, 12 Oct 2021 20:27:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=s+D8fjAlWfOwVxTjXsqSU9ga2khi1k6Zax2Zs/uECd0=; b=JAi/K9XtZEiB14Litco26Ar1UuK6jxrRpSZIQtv+QvnLtdlP0ggeyCEEtxkaVWa4uW MZJHH2af7h0kN46ZARJQyqEYKJ3VdtaWrnJnX3CfgHIHzyIRMyS/y1f9/G5GQXAwsH2M PBl1u7fbrn7Q3WgiNCEdN/gal1tLf+7kViB2KQifCcFgbPzraXE2EvuwIae66kxdbJGZ 5sj+AfiNW+bow6hSiRZi9PD2pUYcf8UywQON3oCEy2f0bSTq27Pr/Q500/iZNBPxP9aI nhzF4he0cM/69kMWmM0eKIkiXcTWFpFhA9qdPr5ec3OZ2ag2qKEHSL6bMUxRUnBnnsD1 JX8Q== X-Gm-Message-State: AOAM532wYVwaVgggF/YoPKoFSrPnnng5XVTUt+uNflfbcaeu66WY4T0H 6Od4gNnNBr44aDhj0RPZZy0P4AaZfPfeDLs56HgPihB1I3e8OjtJ64OsKj6fQkAtHM7gssPNxLs SPIfiPB1VT2GcGNYVwetLdDiQpIGSHwXZph+fB0o6cQ== X-Received: by 2002:aa7:80d1:0:b029:399:ce3a:d617 with SMTP id a17-20020aa780d10000b0290399ce3ad617mr35233906pfn.16.1634095642301; Tue, 12 Oct 2021 20:27:22 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxpxA2JgacQ9XKMhHV8f1AW8OJ5PyYZKq8Ogru5GxisdlVBWuuRuZvwDoW6ciIRKPadY/Bxcw== X-Received: by 2002:aa7:80d1:0:b029:399:ce3a:d617 with SMTP id a17-20020aa780d10000b0290399ce3ad617mr35233864pfn.16.1634095641839; Tue, 12 Oct 2021 20:27:21 -0700 (PDT) Received: from wangxiaodeMacBook-Air.local ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id d24sm12615715pgv.52.2021.10.12.20.27.13 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 12 Oct 2021 20:27:21 -0700 (PDT) Subject: Re: [RFC PATCH v4 08/20] vhost: Route guest->host notification through shadow virtqueue To: =?UTF-8?Q?Eugenio_P=c3=a9rez?= , qemu-devel@nongnu.org References: <20211001070603.307037-1-eperezma@redhat.com> <20211001070603.307037-9-eperezma@redhat.com> From: Jason Wang Message-ID: <23e17562-719e-44da-3362-97e0ca5ae1ff@redhat.com> Date: Wed, 13 Oct 2021 11:27:11 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 MIME-Version: 1.0 In-Reply-To: <20211001070603.307037-9-eperezma@redhat.com> Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jasowang@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Cc: Parav Pandit , "Michael S. Tsirkin" , Markus Armbruster , virtualization@lists.linux-foundation.org, Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , Eric Blake , Michael Lilja X-BeenThere: virtualization@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Linux virtualization List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Errors-To: virtualization-bounces@lists.linux-foundation.org Sender: "Virtualization" CuWcqCAyMDIxLzEwLzEg5LiL5Y2IMzowNSwgRXVnZW5pbyBQw6lyZXog5YaZ6YGTOgo+IFNoYWRv dyB2aXJ0cXVldWUgbm90aWZpY2F0aW9ucyBmb3J3YXJkaW5nIGlzIGRpc2FibGVkIHdoZW4gdmhv c3RfZGV2Cj4gc3RvcHMsIHNvIGNvZGUgZmxvdyBmb2xsb3dzIHVzdWFsIGNsZWFudXAuCj4KPiBB bHNvLCBob3N0IG5vdGlmaWVycyBtdXN0IGJlIGRpc2FibGVkIGF0IFNWUSBzdGFydCwKCgpBbnkg cmVhc29uIGZvciB0aGlzPwoKCj4gYW5kIHRoZXkgd2lsbCBub3QKPiBzdGFydCBpZiBTVlEgaGFz IGJlZW4gZW5hYmxlZCB3aGVuIGRldmljZSBpcyBzdG9wcGVkLiBUaGlzIGlzIHRyaXZpYWwKPiB0 byBhZGRyZXNzLCBidXQgaXQgaXMgbGVmdCBvdXQgZm9yIHNpbXBsaWNpdHkgYXQgdGhpcyBtb21l bnQuCgoKSXQgbG9va3MgdG8gbWUgdGhpcyBwYXRjaCBhbHNvIGNvbnRhaW5zIHRoZSBmb2xsb3dp bmcgbG9naWNzCgoxKSBjb2RlcyB0byBlbmFibGUgc3ZxCgoyKSBjb2RlcyB0byBsZXQgc3ZxIHRv IGJlIGVuYWJsZWQgZnJvbSBRTVAuCgpJIHRoaW5rIHRoZXkgbmVlZCB0byBiZSBzcGxpdCBvdXQs IHdlIG1heSBlbmR1cCB3aXRoIHRoZSBmb2xsb3dpbmcgCnNlcmllcyBvZiBwYXRjaGVzCgoxKSBz dnEgc2tlbGV0b24gd2l0aCBlbmFibGUvZGlzYWJsZQoyKSByb3V0ZSBob3N0IG5vdGlmaWVyIHRv IHN2cQozKSByb3V0ZSBndWVzdCBub3RpZmllciB0byBzdnEKNCkgY29kZXMgdG8gZW5hYmxlIHN2 cQo1KSBlbmFibGUgc3ZxIHZpYSBRTVAKCgo+Cj4gU2lnbmVkLW9mZi1ieTogRXVnZW5pbyBQw6ly ZXogPGVwZXJlem1hQHJlZGhhdC5jb20+Cj4gLS0tCj4gICBxYXBpL25ldC5qc29uICAgICAgICAg ICAgICAgICAgICAgIHwgICAyICstCj4gICBody92aXJ0aW8vdmhvc3Qtc2hhZG93LXZpcnRxdWV1 ZS5oIHwgICA4ICsrCj4gICBpbmNsdWRlL2h3L3ZpcnRpby92aG9zdC12ZHBhLmggICAgIHwgICA0 ICsKPiAgIGh3L3ZpcnRpby92aG9zdC1zaGFkb3ctdmlydHF1ZXVlLmMgfCAxMzggKysrKysrKysr KysrKysrKysrKysrKysrKysrKy0KPiAgIGh3L3ZpcnRpby92aG9zdC12ZHBhLmMgICAgICAgICAg ICAgfCAxMTYgKysrKysrKysrKysrKysrKysrKysrKystCj4gICA1IGZpbGVzIGNoYW5nZWQsIDI2 NCBpbnNlcnRpb25zKCspLCA0IGRlbGV0aW9ucygtKQo+Cj4gZGlmZiAtLWdpdCBhL3FhcGkvbmV0 Lmpzb24gYi9xYXBpL25ldC5qc29uCj4gaW5kZXggYTJjMzBmZDQ1NS4uZmU1NDZiMGU3YyAxMDA2 NDQKPiAtLS0gYS9xYXBpL25ldC5qc29uCj4gKysrIGIvcWFwaS9uZXQuanNvbgo+IEBAIC04OCw3 ICs4OCw3IEBACj4gICAjCj4gICAjIEBlbmFibGU6IHRydWUgdG8gdXNlIHRoZSBhbHRlcm5hdGUg c2hhZG93IFZRIG5vdGlmaWNhdGlvbnMKPiAgICMKPiAtIyBSZXR1cm5zOiBBbHdheXMgZXJyb3Is IHNpbmNlIFNWUSBpcyBub3QgaW1wbGVtZW50ZWQgYXQgdGhlIG1vbWVudC4KPiArIyBSZXR1cm5z OiBFcnJvciBpZiBmYWlsdXJlLCBvciAnbm8gZXJyb3InIGZvciBzdWNjZXNzLgo+ICAgIwo+ICAg IyBTaW5jZTogNi4yCj4gICAjCj4gZGlmZiAtLWdpdCBhL2h3L3ZpcnRpby92aG9zdC1zaGFkb3ct dmlydHF1ZXVlLmggYi9ody92aXJ0aW8vdmhvc3Qtc2hhZG93LXZpcnRxdWV1ZS5oCj4gaW5kZXgg MjdhYzYzODhmYS4uMjM3Y2ZjZWI5YyAxMDA2NDQKPiAtLS0gYS9ody92aXJ0aW8vdmhvc3Qtc2hh ZG93LXZpcnRxdWV1ZS5oCj4gKysrIGIvaHcvdmlydGlvL3Zob3N0LXNoYWRvdy12aXJ0cXVldWUu aAo+IEBAIC0xNCw2ICsxNCwxNCBAQAo+ICAgCj4gICB0eXBlZGVmIHN0cnVjdCBWaG9zdFNoYWRv d1ZpcnRxdWV1ZSBWaG9zdFNoYWRvd1ZpcnRxdWV1ZTsKPiAgIAo+ICtFdmVudE5vdGlmaWVyICp2 aG9zdF9zdnFfZ2V0X3N2cV9jYWxsX25vdGlmaWVyKFZob3N0U2hhZG93VmlydHF1ZXVlICpzdnEp OwoKCkxldCdzIG1vdmUgdGhpcyBmdW5jdGlvbiB0byBhbm90aGVyIHBhdGNoIHNpbmNlIGl0J3Mg dW5yZWxhdGVkIHRvIHRoZSAKZ3Vlc3QtPmhvc3Qgcm91dGluZy4KCgo+ICt2b2lkIHZob3N0X3N2 cV9zZXRfZ3Vlc3RfY2FsbF9ub3RpZmllcihWaG9zdFNoYWRvd1ZpcnRxdWV1ZSAqc3ZxLCBpbnQg Y2FsbF9mZCk7Cj4gKwo+ICtib29sIHZob3N0X3N2cV9zdGFydChzdHJ1Y3Qgdmhvc3RfZGV2ICpk ZXYsIHVuc2lnbmVkIGlkeCwKPiArICAgICAgICAgICAgICAgICAgICAgVmhvc3RTaGFkb3dWaXJ0 cXVldWUgKnN2cSk7Cj4gK3ZvaWQgdmhvc3Rfc3ZxX3N0b3Aoc3RydWN0IHZob3N0X2RldiAqZGV2 LCB1bnNpZ25lZCBpZHgsCj4gKyAgICAgICAgICAgICAgICAgICAgVmhvc3RTaGFkb3dWaXJ0cXVl dWUgKnN2cSk7Cj4gKwo+ICAgVmhvc3RTaGFkb3dWaXJ0cXVldWUgKnZob3N0X3N2cV9uZXcoc3Ry dWN0IHZob3N0X2RldiAqZGV2LCBpbnQgaWR4KTsKPiAgIAo+ICAgdm9pZCB2aG9zdF9zdnFfZnJl ZShWaG9zdFNoYWRvd1ZpcnRxdWV1ZSAqdnEpOwo+IGRpZmYgLS1naXQgYS9pbmNsdWRlL2h3L3Zp cnRpby92aG9zdC12ZHBhLmggYi9pbmNsdWRlL2h3L3ZpcnRpby92aG9zdC12ZHBhLmgKPiBpbmRl eCAwZDU2NWJiNWJkLi40OGFhZTU5ZDhlIDEwMDY0NAo+IC0tLSBhL2luY2x1ZGUvaHcvdmlydGlv L3Zob3N0LXZkcGEuaAo+ICsrKyBiL2luY2x1ZGUvaHcvdmlydGlvL3Zob3N0LXZkcGEuaAo+IEBA IC0xMiw2ICsxMiw4IEBACj4gICAjaWZuZGVmIEhXX1ZJUlRJT19WSE9TVF9WRFBBX0gKPiAgICNk ZWZpbmUgSFdfVklSVElPX1ZIT1NUX1ZEUEFfSAo+ICAgCj4gKyNpbmNsdWRlIDxnbW9kdWxlLmg+ Cj4gKwo+ICAgI2luY2x1ZGUgInFlbXUvcXVldWUuaCIKPiAgICNpbmNsdWRlICJody92aXJ0aW8v dmlydGlvLmgiCj4gICAKPiBAQCAtMjQsNiArMjYsOCBAQCB0eXBlZGVmIHN0cnVjdCB2aG9zdF92 ZHBhIHsKPiAgICAgICBpbnQgZGV2aWNlX2ZkOwo+ICAgICAgIHVpbnQzMl90IG1zZ190eXBlOwo+ ICAgICAgIE1lbW9yeUxpc3RlbmVyIGxpc3RlbmVyOwo+ICsgICAgYm9vbCBzaGFkb3dfdnFzX2Vu YWJsZWQ7Cj4gKyAgICBHUHRyQXJyYXkgKnNoYWRvd192cXM7Cj4gICAgICAgc3RydWN0IHZob3N0 X2RldiAqZGV2Owo+ICAgICAgIFFMSVNUX0VOVFJZKHZob3N0X3ZkcGEpIGVudHJ5Owo+ICAgICAg IFZob3N0VkRQQUhvc3ROb3RpZmllciBub3RpZmllcltWSVJUSU9fUVVFVUVfTUFYXTsKPiBkaWZm IC0tZ2l0IGEvaHcvdmlydGlvL3Zob3N0LXNoYWRvdy12aXJ0cXVldWUuYyBiL2h3L3ZpcnRpby92 aG9zdC1zaGFkb3ctdmlydHF1ZXVlLmMKPiBpbmRleCBjNDgyNmExYjU2Li4yMWRjOTlhYjVkIDEw MDY0NAo+IC0tLSBhL2h3L3ZpcnRpby92aG9zdC1zaGFkb3ctdmlydHF1ZXVlLmMKPiArKysgYi9o dy92aXJ0aW8vdmhvc3Qtc2hhZG93LXZpcnRxdWV1ZS5jCj4gQEAgLTksOSArOSwxMiBAQAo+ICAg Cj4gICAjaW5jbHVkZSAicWVtdS9vc2RlcC5oIgo+ICAgI2luY2x1ZGUgImh3L3ZpcnRpby92aG9z dC1zaGFkb3ctdmlydHF1ZXVlLmgiCj4gKyNpbmNsdWRlICJody92aXJ0aW8vdmhvc3QuaCIKPiAr Cj4gKyNpbmNsdWRlICJzdGFuZGFyZC1oZWFkZXJzL2xpbnV4L3Zob3N0X3R5cGVzLmgiCj4gICAK PiAgICNpbmNsdWRlICJxZW11L2Vycm9yLXJlcG9ydC5oIgo+IC0jaW5jbHVkZSAicWVtdS9ldmVu dF9ub3RpZmllci5oIgo+ICsjaW5jbHVkZSAicWVtdS9tYWluLWxvb3AuaCIKPiAgIAo+ICAgLyog U2hhZG93IHZpcnRxdWV1ZSB0byByZWxheSBub3RpZmljYXRpb25zICovCj4gICB0eXBlZGVmIHN0 cnVjdCBWaG9zdFNoYWRvd1ZpcnRxdWV1ZSB7Cj4gQEAgLTE5LDE0ICsyMiwxNDYgQEAgdHlwZWRl ZiBzdHJ1Y3QgVmhvc3RTaGFkb3dWaXJ0cXVldWUgewo+ICAgICAgIEV2ZW50Tm90aWZpZXIga2lj a19ub3RpZmllcjsKPiAgICAgICAvKiBTaGFkb3cgY2FsbCBub3RpZmllciwgc2VudCB0byB2aG9z dCAqLwo+ICAgICAgIEV2ZW50Tm90aWZpZXIgY2FsbF9ub3RpZmllcjsKPiArCj4gKyAgICAvKgo+ ICsgICAgICogQm9ycm93ZWQgdmlydHF1ZXVlJ3MgZ3Vlc3QgdG8gaG9zdCBub3RpZmllci4KPiAr ICAgICAqIFRvIGJvcnJvdyBpdCBpbiB0aGlzIGV2ZW50IG5vdGlmaWVyIGFsbG93cyB0byByZWdp c3RlciBvbiB0aGUgZXZlbnQKPiArICAgICAqIGxvb3AgYW5kIGFjY2VzcyB0aGUgYXNzb2NpYXRl ZCBzaGFkb3cgdmlydHF1ZXVlIGVhc2lseS4gSWYgd2UgdXNlIHRoZQo+ICsgICAgICogVmlydFF1 ZXVlLCB3ZSBkb24ndCBoYXZlIGFuIGVhc3kgd2F5IHRvIHJldHJpZXZlIGl0Lgo+ICsgICAgICoK PiArICAgICAqIFNvIHNoYWRvdyB2aXJ0cXVldWUgbXVzdCBub3QgY2xlYW4gaXQsIG9yIHdlIHdv dWxkIGxvc2UgVmlydFF1ZXVlIG9uZS4KPiArICAgICAqLwo+ICsgICAgRXZlbnROb3RpZmllciBo b3N0X25vdGlmaWVyOwo+ICsKPiArICAgIC8qIEd1ZXN0J3MgY2FsbCBub3RpZmllciwgd2hlcmUg U1ZRIGNhbGxzIGd1ZXN0LiAqLwo+ICsgICAgRXZlbnROb3RpZmllciBndWVzdF9jYWxsX25vdGlm aWVyOwoKClRvIGJlIGNvbnNpc3RlbnQsIGxldCdzIHNpbXBseSB1c2UgImd1ZXN0X25vdGlmaWVy IiBoZXJlLgoKCj4gKwo+ICsgICAgLyogVmlydGlvIHF1ZXVlIHNoYWRvd2luZyAqLwo+ICsgICAg VmlydFF1ZXVlICp2cTsKPiAgIH0gVmhvc3RTaGFkb3dWaXJ0cXVldWU7Cj4gICAKPiArLyogRm9y d2FyZCBndWVzdCBub3RpZmljYXRpb25zICovCj4gK3N0YXRpYyB2b2lkIHZob3N0X2hhbmRsZV9n dWVzdF9raWNrKEV2ZW50Tm90aWZpZXIgKm4pCj4gK3sKPiArICAgIFZob3N0U2hhZG93VmlydHF1 ZXVlICpzdnEgPSBjb250YWluZXJfb2YobiwgVmhvc3RTaGFkb3dWaXJ0cXVldWUsCj4gKyAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGhvc3Rfbm90aWZpZXIpOwo+ ICsKPiArICAgIGlmICh1bmxpa2VseSghZXZlbnRfbm90aWZpZXJfdGVzdF9hbmRfY2xlYXIobikp KSB7Cj4gKyAgICAgICAgcmV0dXJuOwo+ICsgICAgfQoKCklzIHRoZXJlIGEgY2hhbmNlIHRoYXQg d2UgbWF5IHN0b3AgdGhlIHByb2Nlc3Npbmcgb2YgYXZhaWxhYmxlIGJ1ZmZlcnMgCmR1cmluZyB0 aGUgc3ZxIGVuYWJsaW5nPyBUaGVyZSBjb3VsZCBiZSBubyBraWNrIGZyb20gdGhlIGd1ZXN0IGlu IHRoaXMgY2FzZS4KCgo+ICsKPiArICAgIGV2ZW50X25vdGlmaWVyX3NldCgmc3ZxLT5raWNrX25v dGlmaWVyKTsKPiArfQo+ICsKPiArLyoKPiArICogT2J0YWluIHRoZSBTVlEgY2FsbCBub3RpZmll ciwgd2hlcmUgdmhvc3QgZGV2aWNlIG5vdGlmaWVzIFNWUSB0aGF0IHRoZXJlCj4gKyAqIGV4aXN0 cyBwZW5kaW5nIHVzZWQgYnVmZmVycy4KPiArICoKPiArICogQHN2cSBTaGFkb3cgVmlydHF1ZXVl Cj4gKyAqLwo+ICtFdmVudE5vdGlmaWVyICp2aG9zdF9zdnFfZ2V0X3N2cV9jYWxsX25vdGlmaWVy KFZob3N0U2hhZG93VmlydHF1ZXVlICpzdnEpCj4gK3sKPiArICAgIHJldHVybiAmc3ZxLT5jYWxs X25vdGlmaWVyOwo+ICt9Cj4gKwo+ICsvKgo+ICsgKiBTZXQgdGhlIGNhbGwgbm90aWZpZXIgZm9y IHRoZSBTVlEgdG8gY2FsbCB0aGUgZ3Vlc3QKPiArICoKPiArICogQHN2cSBTaGFkb3cgdmlydHF1 ZXVlCj4gKyAqIEBjYWxsX2ZkIGNhbGwgbm90aWZpZXIKPiArICoKPiArICogQ2FsbGVkIG9uIEJR TCBjb250ZXh0Lgo+ICsgKi8KPiArdm9pZCB2aG9zdF9zdnFfc2V0X2d1ZXN0X2NhbGxfbm90aWZp ZXIoVmhvc3RTaGFkb3dWaXJ0cXVldWUgKnN2cSwgaW50IGNhbGxfZmQpCj4gK3sKPiArICAgIGV2 ZW50X25vdGlmaWVyX2luaXRfZmQoJnN2cS0+Z3Vlc3RfY2FsbF9ub3RpZmllciwgY2FsbF9mZCk7 Cj4gK30KPiArCj4gKy8qCj4gKyAqIFJlc3RvcmUgdGhlIHZob3N0IGd1ZXN0IHRvIGhvc3Qgbm90 aWZpZXIsIGkuZS4sIGRpc2FibGVzIHN2cSBlZmZlY3QuCj4gKyAqLwo+ICtzdGF0aWMgaW50IHZo b3N0X3N2cV9yZXN0b3JlX3ZkZXZfaG9zdF9ub3RpZmllcihzdHJ1Y3Qgdmhvc3RfZGV2ICpkZXYs Cj4gKyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHVuc2ln bmVkIHZob3N0X2luZGV4LAo+ICsgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICBWaG9zdFNoYWRvd1ZpcnRxdWV1ZSAqc3ZxKQo+ICt7Cj4gKyAgICBFdmVudE5v dGlmaWVyICp2cV9ob3N0X25vdGlmaWVyID0gdmlydGlvX3F1ZXVlX2dldF9ob3N0X25vdGlmaWVy KHN2cS0+dnEpOwo+ICsgICAgc3RydWN0IHZob3N0X3ZyaW5nX2ZpbGUgZmlsZSA9IHsKPiArICAg ICAgICAuaW5kZXggPSB2aG9zdF9pbmRleCwKPiArICAgICAgICAuZmQgPSBldmVudF9ub3RpZmll cl9nZXRfZmQodnFfaG9zdF9ub3RpZmllciksCj4gKyAgICB9Owo+ICsgICAgaW50IHI7Cj4gKwo+ ICsgICAgLyogUmVzdG9yZSB2aG9zdCBraWNrICovCj4gKyAgICByID0gZGV2LT52aG9zdF9vcHMt PnZob3N0X3NldF92cmluZ19raWNrKGRldiwgJmZpbGUpOwoKCkFuZCByZW1hcCB0aGUgbm90aWZp Y2F0aW9uIGFyZWEgaWYgbmVjZXNzYXJ5LgoKCj4gKyAgICByZXR1cm4gciA/IC1lcnJubyA6IDA7 Cj4gK30KPiArCj4gKy8qCj4gKyAqIFN0YXJ0IHNoYWRvdyB2aXJ0cXVldWUgb3BlcmF0aW9uLgo+ ICsgKiBAZGV2IHZob3N0IGRldmljZQo+ICsgKiBAaGlkeCB2aG9zdCB2aXJ0cXVldWUgaW5kZXgK PiArICogQHN2cSBTaGFkb3cgVmlydHF1ZXVlCj4gKyAqLwo+ICtib29sIHZob3N0X3N2cV9zdGFy dChzdHJ1Y3Qgdmhvc3RfZGV2ICpkZXYsIHVuc2lnbmVkIGlkeCwKPiArICAgICAgICAgICAgICAg ICAgICAgVmhvc3RTaGFkb3dWaXJ0cXVldWUgKnN2cSkKPiArewo+ICsgICAgRXZlbnROb3RpZmll ciAqdnFfaG9zdF9ub3RpZmllciA9IHZpcnRpb19xdWV1ZV9nZXRfaG9zdF9ub3RpZmllcihzdnEt PnZxKTsKPiArICAgIHN0cnVjdCB2aG9zdF92cmluZ19maWxlIGZpbGUgPSB7Cj4gKyAgICAgICAg LmluZGV4ID0gZGV2LT52aG9zdF9vcHMtPnZob3N0X2dldF92cV9pbmRleChkZXYsIGRldi0+dnFf aW5kZXggKyBpZHgpLAo+ICsgICAgICAgIC5mZCA9IGV2ZW50X25vdGlmaWVyX2dldF9mZCgmc3Zx LT5raWNrX25vdGlmaWVyKSwKPiArICAgIH07Cj4gKyAgICBpbnQgcjsKPiArCj4gKyAgICAvKiBD aGVjayB0aGF0IG5vdGlmaWNhdGlvbnMgYXJlIHN0aWxsIGdvaW5nIGRpcmVjdGx5IHRvIHZob3N0 IGRldiAqLwo+ICsgICAgYXNzZXJ0KHZpcnRpb19xdWV1ZV9pc19ob3N0X25vdGlmaWVyX2VuYWJs ZWQoc3ZxLT52cSkpOwo+ICsKPiArICAgIC8qCj4gKyAgICAgKiBldmVudF9ub3RpZmllcl9zZXRf aGFuZGxlciBhbHJlYWR5IGNoZWNrcyBmb3IgZ3Vlc3QncyBub3RpZmljYXRpb25zIGlmCj4gKyAg ICAgKiB0aGV5IGFycml2ZSBpbiB0aGUgc3dpdGNoLCBzbyB0aGVyZSBpcyBubyBuZWVkIHRvIGV4 cGxpY2l0ZWx5IGNoZWNrIGZvcgo+ICsgICAgICogdGhlbS4KPiArICAgICAqLwoKCklmIHRoaXMg aXMgdHJ1ZSwgc2hvdWxkbid0IHdlIGNhbGwgdmhvc3Rfc2V0X3ZyaW5nX2tpY2soKSBiZWZvcmUg dGhlIApldmVudF9ub3RpZmllcl9zZXRfaGFuZGxlcigpPwoKQnR3LCBJIHRoaW5rIHdlIHNob3Vs ZCB1cGRhdGUgdGhlIGZkIGlmIHNldF92cmluZ19raWNrKCkgd2FzIGNhbGxlZCAKYWZ0ZXIgdGhp cyBmdW5jdGlvbj8KCgo+ICsgICAgZXZlbnRfbm90aWZpZXJfaW5pdF9mZCgmc3ZxLT5ob3N0X25v dGlmaWVyLAo+ICsgICAgICAgICAgICAgICAgICAgICAgICAgICBldmVudF9ub3RpZmllcl9nZXRf ZmQodnFfaG9zdF9ub3RpZmllcikpOwo+ICsgICAgZXZlbnRfbm90aWZpZXJfc2V0X2hhbmRsZXIo JnN2cS0+aG9zdF9ub3RpZmllciwgdmhvc3RfaGFuZGxlX2d1ZXN0X2tpY2spOwo+ICsKPiArICAg IHIgPSBkZXYtPnZob3N0X29wcy0+dmhvc3Rfc2V0X3ZyaW5nX2tpY2soZGV2LCAmZmlsZSk7CgoK QW5kIHdlIG5lZWQgdG8gc3RvcCB0aGUgbm90aWZpY2F0aW9uIGFyZWEgbW1hcC4KCgo+ICsgICAg aWYgKHVubGlrZWx5KHIgIT0gMCkpIHsKPiArICAgICAgICBlcnJvcl9yZXBvcnQoIkNvdWxkbid0 IHNldCBraWNrIGZkOiAlcyIsIHN0cmVycm9yKGVycm5vKSk7Cj4gKyAgICAgICAgZ290byBlcnJf c2V0X3ZyaW5nX2tpY2s7Cj4gKyAgICB9Cj4gKwo+ICsgICAgcmV0dXJuIHRydWU7Cj4gKwo+ICtl cnJfc2V0X3ZyaW5nX2tpY2s6Cj4gKyAgICBldmVudF9ub3RpZmllcl9zZXRfaGFuZGxlcigmc3Zx LT5ob3N0X25vdGlmaWVyLCBOVUxMKTsKPiArCj4gKyAgICByZXR1cm4gZmFsc2U7Cj4gK30KPiAr Cj4gKy8qCj4gKyAqIFN0b3Agc2hhZG93IHZpcnRxdWV1ZSBvcGVyYXRpb24uCj4gKyAqIEBkZXYg dmhvc3QgZGV2aWNlCj4gKyAqIEBpZHggdmhvc3QgcXVldWUgaW5kZXgKPiArICogQHN2cSBTaGFk b3cgVmlydHF1ZXVlCj4gKyAqLwo+ICt2b2lkIHZob3N0X3N2cV9zdG9wKHN0cnVjdCB2aG9zdF9k ZXYgKmRldiwgdW5zaWduZWQgaWR4LAo+ICsgICAgICAgICAgICAgICAgICAgIFZob3N0U2hhZG93 VmlydHF1ZXVlICpzdnEpCj4gK3sKPiArICAgIGludCByID0gdmhvc3Rfc3ZxX3Jlc3RvcmVfdmRl dl9ob3N0X25vdGlmaWVyKGRldiwgaWR4LCBzdnEpOwo+ICsgICAgaWYgKHVubGlrZWx5KHIgPCAw KSkgewo+ICsgICAgICAgIGVycm9yX3JlcG9ydCgiQ291bGRuJ3QgcmVzdG9yZSB2cSBraWNrIGZk OiAlcyIsIHN0cmVycm9yKC1yKSk7Cj4gKyAgICB9Cj4gKwo+ICsgICAgZXZlbnRfbm90aWZpZXJf c2V0X2hhbmRsZXIoJnN2cS0+aG9zdF9ub3RpZmllciwgTlVMTCk7Cj4gK30KPiArCj4gICAvKgo+ ICAgICogQ3JlYXRlcyB2aG9zdCBzaGFkb3cgdmlydHF1ZXVlLCBhbmQgaW5zdHJ1Y3Qgdmhvc3Qg ZGV2aWNlIHRvIHVzZSB0aGUgc2hhZG93Cj4gICAgKiBtZXRob2RzIGFuZCBmaWxlIGRlc2NyaXB0 b3JzLgo+ICAgICovCj4gICBWaG9zdFNoYWRvd1ZpcnRxdWV1ZSAqdmhvc3Rfc3ZxX25ldyhzdHJ1 Y3Qgdmhvc3RfZGV2ICpkZXYsIGludCBpZHgpCj4gICB7Cj4gKyAgICBpbnQgdnFfaWR4ID0gZGV2 LT52cV9pbmRleCArIGlkeDsKPiAgICAgICBnX2F1dG9mcmVlIFZob3N0U2hhZG93VmlydHF1ZXVl ICpzdnEgPSBnX25ldzAoVmhvc3RTaGFkb3dWaXJ0cXVldWUsIDEpOwo+ICAgICAgIGludCByOwo+ ICAgCj4gQEAgLTQ0LDYgKzE3OSw3IEBAIFZob3N0U2hhZG93VmlydHF1ZXVlICp2aG9zdF9zdnFf bmV3KHN0cnVjdCB2aG9zdF9kZXYgKmRldiwgaW50IGlkeCkKPiAgICAgICAgICAgZ290byBlcnJf aW5pdF9jYWxsX25vdGlmaWVyOwo+ICAgICAgIH0KPiAgIAo+ICsgICAgc3ZxLT52cSA9IHZpcnRp b19nZXRfcXVldWUoZGV2LT52ZGV2LCB2cV9pZHgpOwo+ICAgICAgIHJldHVybiBnX3N0ZWFsX3Bv aW50ZXIoJnN2cSk7Cj4gICAKPiAgIGVycl9pbml0X2NhbGxfbm90aWZpZXI6Cj4gZGlmZiAtLWdp dCBhL2h3L3ZpcnRpby92aG9zdC12ZHBhLmMgYi9ody92aXJ0aW8vdmhvc3QtdmRwYS5jCj4gaW5k ZXggZTBkYzc1MDhjMy4uMzZjOTU0YTc3OSAxMDA2NDQKPiAtLS0gYS9ody92aXJ0aW8vdmhvc3Qt dmRwYS5jCj4gKysrIGIvaHcvdmlydGlvL3Zob3N0LXZkcGEuYwo+IEBAIC0xNyw2ICsxNyw3IEBA Cj4gICAjaW5jbHVkZSAiaHcvdmlydGlvL3Zob3N0LmgiCj4gICAjaW5jbHVkZSAiaHcvdmlydGlv L3Zob3N0LWJhY2tlbmQuaCIKPiAgICNpbmNsdWRlICJody92aXJ0aW8vdmlydGlvLW5ldC5oIgo+ ICsjaW5jbHVkZSAiaHcvdmlydGlvL3Zob3N0LXNoYWRvdy12aXJ0cXVldWUuaCIKPiAgICNpbmNs dWRlICJody92aXJ0aW8vdmhvc3QtdmRwYS5oIgo+ICAgI2luY2x1ZGUgImV4ZWMvYWRkcmVzcy1z cGFjZXMuaCIKPiAgICNpbmNsdWRlICJxZW11L21haW4tbG9vcC5oIgo+IEBAIC0yNzIsNiArMjcz LDE2IEBAIHN0YXRpYyB2b2lkIHZob3N0X3ZkcGFfYWRkX3N0YXR1cyhzdHJ1Y3Qgdmhvc3RfZGV2 ICpkZXYsIHVpbnQ4X3Qgc3RhdHVzKQo+ICAgICAgIHZob3N0X3ZkcGFfY2FsbChkZXYsIFZIT1NU X1ZEUEFfU0VUX1NUQVRVUywgJnMpOwo+ICAgfQo+ICAgCj4gKy8qKgo+ICsgKiBBZGFwdG9yIGZ1 bmN0aW9uIHRvIGZyZWUgc2hhZG93IHZpcnRxdWV1ZSB0aHJvdWdoIGdwb2ludGVyCj4gKyAqCj4g KyAqIEBzdnEgICBUaGUgU2hhZG93IFZpcnRxdWV1ZQo+ICsgKi8KPiArc3RhdGljIHZvaWQgdmhv c3RfcHN2cV9mcmVlKGdwb2ludGVyIHN2cSkKPiArewo+ICsgICAgdmhvc3Rfc3ZxX2ZyZWUoc3Zx KTsKPiArfQo+ICsKPiAgIHN0YXRpYyBpbnQgdmhvc3RfdmRwYV9pbml0KHN0cnVjdCB2aG9zdF9k ZXYgKmRldiwgdm9pZCAqb3BhcXVlLCBFcnJvciAqKmVycnApCj4gICB7Cj4gICAgICAgc3RydWN0 IHZob3N0X3ZkcGEgKnY7Cj4gQEAgLTI4Myw2ICsyOTQsNyBAQCBzdGF0aWMgaW50IHZob3N0X3Zk cGFfaW5pdChzdHJ1Y3Qgdmhvc3RfZGV2ICpkZXYsIHZvaWQgKm9wYXF1ZSwgRXJyb3IgKiplcnJw KQo+ICAgICAgIGRldi0+b3BhcXVlID0gIG9wYXF1ZSA7Cj4gICAgICAgdi0+bGlzdGVuZXIgPSB2 aG9zdF92ZHBhX21lbW9yeV9saXN0ZW5lcjsKPiAgICAgICB2LT5tc2dfdHlwZSA9IFZIT1NUX0lP VExCX01TR19WMjsKPiArICAgIHYtPnNoYWRvd192cXMgPSBnX3B0cl9hcnJheV9uZXdfZnVsbChk ZXYtPm52cXMsIHZob3N0X3BzdnFfZnJlZSk7Cj4gICAgICAgUUxJU1RfSU5TRVJUX0hFQUQoJnZo b3N0X3ZkcGFfZGV2aWNlcywgdiwgZW50cnkpOwo+ICAgCj4gICAgICAgdmhvc3RfdmRwYV9hZGRf c3RhdHVzKGRldiwgVklSVElPX0NPTkZJR19TX0FDS05PV0xFREdFIHwKPiBAQCAtMzczLDYgKzM4 NSwxNyBAQCBlcnI6Cj4gICAgICAgcmV0dXJuOwo+ICAgfQo+ICAgCj4gK3N0YXRpYyB2b2lkIHZo b3N0X3ZkcGFfc3ZxX2NsZWFudXAoc3RydWN0IHZob3N0X2RldiAqZGV2KQo+ICt7Cj4gKyAgICBz dHJ1Y3Qgdmhvc3RfdmRwYSAqdiA9IGRldi0+b3BhcXVlOwo+ICsgICAgc2l6ZV90IGlkeDsKPiAr Cj4gKyAgICBmb3IgKGlkeCA9IDA7IGlkeCA8IHYtPnNoYWRvd192cXMtPmxlbjsgKytpZHgpIHsK PiArICAgICAgICB2aG9zdF9zdnFfc3RvcChkZXYsIGlkeCwgZ19wdHJfYXJyYXlfaW5kZXgodi0+ c2hhZG93X3ZxcywgaWR4KSk7Cj4gKyAgICB9Cj4gKyAgICBnX3B0cl9hcnJheV9mcmVlKHYtPnNo YWRvd192cXMsIHRydWUpOwo+ICt9Cj4gKwo+ICAgc3RhdGljIGludCB2aG9zdF92ZHBhX2NsZWFu dXAoc3RydWN0IHZob3N0X2RldiAqZGV2KQo+ICAgewo+ICAgICAgIHN0cnVjdCB2aG9zdF92ZHBh ICp2Owo+IEBAIC0zODEsNiArNDA0LDcgQEAgc3RhdGljIGludCB2aG9zdF92ZHBhX2NsZWFudXAo c3RydWN0IHZob3N0X2RldiAqZGV2KQo+ICAgICAgIHRyYWNlX3Zob3N0X3ZkcGFfY2xlYW51cChk ZXYsIHYpOwo+ICAgICAgIHZob3N0X3ZkcGFfaG9zdF9ub3RpZmllcnNfdW5pbml0KGRldiwgZGV2 LT5udnFzKTsKPiAgICAgICBtZW1vcnlfbGlzdGVuZXJfdW5yZWdpc3Rlcigmdi0+bGlzdGVuZXIp Owo+ICsgICAgdmhvc3RfdmRwYV9zdnFfY2xlYW51cChkZXYpOwo+ICAgICAgIFFMSVNUX1JFTU9W RSh2LCBlbnRyeSk7Cj4gICAKPiAgICAgICBkZXYtPm9wYXF1ZSA9IE5VTEw7Cj4gQEAgLTU1Nyw3 ICs1ODEsOSBAQCBzdGF0aWMgaW50IHZob3N0X3ZkcGFfZGV2X3N0YXJ0KHN0cnVjdCB2aG9zdF9k ZXYgKmRldiwgYm9vbCBzdGFydGVkKQo+ICAgICAgIGlmIChzdGFydGVkKSB7Cj4gICAgICAgICAg IHVpbnQ4X3Qgc3RhdHVzID0gMDsKPiAgICAgICAgICAgbWVtb3J5X2xpc3RlbmVyX3JlZ2lzdGVy KCZ2LT5saXN0ZW5lciwgJmFkZHJlc3Nfc3BhY2VfbWVtb3J5KTsKPiAtICAgICAgICB2aG9zdF92 ZHBhX2hvc3Rfbm90aWZpZXJzX2luaXQoZGV2KTsKPiArICAgICAgICBpZiAoIXYtPnNoYWRvd192 cXNfZW5hYmxlZCkgewo+ICsgICAgICAgICAgICB2aG9zdF92ZHBhX2hvc3Rfbm90aWZpZXJzX2lu aXQoZGV2KTsKPiArICAgICAgICB9CgoKVGhpcyBsb29rcyBsaWtlIGEgdHJpY2ssIHdoeSBub3Qg Y2hlY2sgYW5kIHNldHVwIHNoYWRvd192cXMgaW5zaWRlOgoKMSkgdmhvc3RfdmRwYV9ob3N0X25v dGlmaWVyc19pbml0KCkKCmFuZAoKMikgdmhvc3RfdmRwYV9zZXRfdnJpbmdfa2ljaygpCgoKPiAg ICAgICAgICAgdmhvc3RfdmRwYV9zZXRfdnJpbmdfcmVhZHkoZGV2KTsKPiAgICAgICAgICAgdmhv c3RfdmRwYV9hZGRfc3RhdHVzKGRldiwgVklSVElPX0NPTkZJR19TX0RSSVZFUl9PSyk7Cj4gICAg ICAgICAgIHZob3N0X3ZkcGFfY2FsbChkZXYsIFZIT1NUX1ZEUEFfR0VUX1NUQVRVUywgJnN0YXR1 cyk7Cj4gQEAgLTY2MywxMCArNjg5LDk2IEBAIHN0YXRpYyBib29sICB2aG9zdF92ZHBhX2ZvcmNl X2lvbW11KHN0cnVjdCB2aG9zdF9kZXYgKmRldikKPiAgICAgICByZXR1cm4gdHJ1ZTsKPiAgIH0K PiAgIAo+ICsvKgo+ICsgKiBTdGFydCBzaGFkb3cgdmlydHF1ZXVlLgo+ICsgKi8KPiArc3RhdGlj IGJvb2wgdmhvc3RfdmRwYV9zdnFfc3RhcnRfdnEoc3RydWN0IHZob3N0X2RldiAqZGV2LCB1bnNp Z25lZCBpZHgpCj4gK3sKPiArICAgIHN0cnVjdCB2aG9zdF92ZHBhICp2ID0gZGV2LT5vcGFxdWU7 Cj4gKyAgICBWaG9zdFNoYWRvd1ZpcnRxdWV1ZSAqc3ZxID0gZ19wdHJfYXJyYXlfaW5kZXgodi0+ c2hhZG93X3ZxcywgaWR4KTsKPiArICAgIHJldHVybiB2aG9zdF9zdnFfc3RhcnQoZGV2LCBpZHgs IHN2cSk7Cj4gK30KPiArCj4gK3N0YXRpYyB1bnNpZ25lZCB2aG9zdF92ZHBhX2VuYWJsZV9zdnEo c3RydWN0IHZob3N0X3ZkcGEgKnYsIGJvb2wgZW5hYmxlKQo+ICt7Cj4gKyAgICBzdHJ1Y3Qgdmhv c3RfZGV2ICpoZGV2ID0gdi0+ZGV2Owo+ICsgICAgdW5zaWduZWQgbjsKPiArCj4gKyAgICBpZiAo ZW5hYmxlID09IHYtPnNoYWRvd192cXNfZW5hYmxlZCkgewo+ICsgICAgICAgIHJldHVybiBoZGV2 LT5udnFzOwo+ICsgICAgfQo+ICsKPiArICAgIGlmIChlbmFibGUpIHsKPiArICAgICAgICAvKiBB bGxvY2F0ZSByZXNvdXJjZXMgKi8KPiArICAgICAgICBhc3NlcnQodi0+c2hhZG93X3Zxcy0+bGVu ID09IDApOwo+ICsgICAgICAgIGZvciAobiA9IDA7IG4gPCBoZGV2LT5udnFzOyArK24pIHsKPiAr ICAgICAgICAgICAgVmhvc3RTaGFkb3dWaXJ0cXVldWUgKnN2cSA9IHZob3N0X3N2cV9uZXcoaGRl diwgbik7Cj4gKyAgICAgICAgICAgIGJvb2wgb2s7Cj4gKwo+ICsgICAgICAgICAgICBpZiAodW5s aWtlbHkoIXN2cSkpIHsKPiArICAgICAgICAgICAgICAgIGdfcHRyX2FycmF5X3NldF9zaXplKHYt PnNoYWRvd192cXMsIDApOwo+ICsgICAgICAgICAgICAgICAgcmV0dXJuIDA7Cj4gKyAgICAgICAg ICAgIH0KPiArICAgICAgICAgICAgZ19wdHJfYXJyYXlfYWRkKHYtPnNoYWRvd192cXMsIHN2cSk7 Cj4gKwo+ICsgICAgICAgICAgICBvayA9IHZob3N0X3ZkcGFfc3ZxX3N0YXJ0X3ZxKGhkZXYsIG4p Owo+ICsgICAgICAgICAgICBpZiAodW5saWtlbHkoIW9rKSkgewo+ICsgICAgICAgICAgICAgICAg LyogRnJlZSBzdGlsbCBub3Qgc3RhcnRlZCBzdnFzICovCj4gKyAgICAgICAgICAgICAgICBnX3B0 cl9hcnJheV9zZXRfc2l6ZSh2LT5zaGFkb3dfdnFzLCBuKTsKPiArICAgICAgICAgICAgICAgIGVu YWJsZSA9IGZhbHNlOwo+ICsgICAgICAgICAgICAgICAgYnJlYWs7Cj4gKyAgICAgICAgICAgIH0K PiArICAgICAgICB9CgoKU2luY2UgdGhlcmUncyBhbG1vc3Qgbm8gbG9naWMgY291bGQgYmUgc2hh cmVkIGJldHdlZW4gZW5hYmxlIGFuZCAKZGlzYWJsZS4gTGV0J3Mgc3BsaXQgdGhvc2UgbG9naWMg b3V0IGludG8gZGVkaWNhdGVkIGZ1bmN0aW9ucyB3aGVyZSB0aGUgCmNvZGVzIGxvb2tzIG1vcmUg ZWFzeSB0byBiZSByZXZpZXdlZCAoZS5nIGhhdmUgYSBiZXR0ZXIgZXJyb3IgaGFuZGxpbmcgZXRj KS4KCgo+ICsgICAgfQo+ICsKPiArICAgIHYtPnNoYWRvd192cXNfZW5hYmxlZCA9IGVuYWJsZTsK PiArCj4gKyAgICBpZiAoIWVuYWJsZSkgewo+ICsgICAgICAgIC8qIERpc2FibGUgYWxsIHF1ZXVl cyBvciBjbGVhbiB1cCBmYWlsZWQgc3RhcnQgKi8KPiArICAgICAgICBmb3IgKG4gPSAwOyBuIDwg di0+c2hhZG93X3Zxcy0+bGVuOyArK24pIHsKPiArICAgICAgICAgICAgdW5zaWduZWQgdnFfaWR4 ID0gdmhvc3RfdmRwYV9nZXRfdnFfaW5kZXgoaGRldiwgbik7Cj4gKyAgICAgICAgICAgIFZob3N0 U2hhZG93VmlydHF1ZXVlICpzdnEgPSBnX3B0cl9hcnJheV9pbmRleCh2LT5zaGFkb3dfdnFzLCBu KTsKPiArICAgICAgICAgICAgdmhvc3Rfc3ZxX3N0b3AoaGRldiwgbiwgc3ZxKTsKPiArICAgICAg ICAgICAgdmhvc3RfdmlydHF1ZXVlX3N0YXJ0KGhkZXYsIGhkZXYtPnZkZXYsICZoZGV2LT52cXNb bl0sIHZxX2lkeCk7Cj4gKyAgICAgICAgfQo+ICsKPiArICAgICAgICAvKiBSZXNvdXJjZXMgY2xl YW51cCAqLwo+ICsgICAgICAgIGdfcHRyX2FycmF5X3NldF9zaXplKHYtPnNoYWRvd192cXMsIDAp Owo+ICsgICAgfQo+ICsKPiArICAgIHJldHVybiBuOwo+ICt9Cj4gICAKPiAgIHZvaWQgcW1wX3hf dmhvc3RfZW5hYmxlX3NoYWRvd192cShjb25zdCBjaGFyICpuYW1lLCBib29sIGVuYWJsZSwgRXJy b3IgKiplcnJwKQo+ICAgewo+IC0gICAgZXJyb3Jfc2V0ZyhlcnJwLCAiU2hhZG93IHZpcnRxdWV1 ZSBzdGlsbCBub3QgaW1wbGVtZW50ZWQiKTsKPiArICAgIHN0cnVjdCB2aG9zdF92ZHBhICp2Owo+ ICsgICAgY29uc3QgY2hhciAqZXJyX2NhdXNlID0gTlVMTDsKPiArICAgIGJvb2wgcjsKPiArCj4g KyAgICBRTElTVF9GT1JFQUNIKHYsICZ2aG9zdF92ZHBhX2RldmljZXMsIGVudHJ5KSB7Cj4gKyAg ICAgICAgaWYgKHYtPmRldi0+dmRldiAmJiAwID09IHN0cmNtcCh2LT5kZXYtPnZkZXYtPm5hbWUs IG5hbWUpKSB7Cj4gKyAgICAgICAgICAgIGJyZWFrOwo+ICsgICAgICAgIH0KPiArICAgIH0KCgpJ IHRoaW5rIHlvdSBjYW4gaXRlcmF0ZSB0aGUgTmV0Q2xpZW50U3RhdGVzIHRvIGdlIHR0aGUgdmhv c3QtdmRwYSBiYWNrZW5kcy4KCgo+ICsKPiArICAgIGlmICghdikgewo+ICsgICAgICAgIGVycl9j YXVzZSA9ICJEZXZpY2Ugbm90IGZvdW5kIjsKPiArICAgICAgICBnb3RvIGVycjsKPiArICAgIH0g ZWxzZSBpZiAodi0+bm90aWZpZXJbMF0uYWRkcikgewo+ICsgICAgICAgIGVycl9jYXVzZSA9ICJE ZXZpY2UgaGFzIGhvc3Qgbm90aWZpZXJzIGVuYWJsZWQiOwoKCkkgZG9uJ3QgZ2V0IHRoaXMuCgpC dHcgdGhpcyBmdW5jdGlvbiBzaG91bGQgYmUgaW1wbGVtZW50ZWQgaW4gYW4gaW5kZXBlbmRlbnQg cGF0Y2ggYWZ0ZXIgCnN2cSBpcyBmdWxseSBmdW5jdGlvbmFsLgoKVGhhbmtzCgoKPiArICAgICAg ICBnb3RvIGVycjsKPiArICAgIH0KPiArCj4gKyAgICByID0gdmhvc3RfdmRwYV9lbmFibGVfc3Zx KHYsIGVuYWJsZSk7Cj4gKyAgICBpZiAodW5saWtlbHkoIXIpKSB7Cj4gKyAgICAgICAgZXJyX2Nh dXNlID0gIkVycm9yIGVuYWJsaW5nIChzZWUgbW9uaXRvcikiOwo+ICsgICAgICAgIGdvdG8gZXJy Owo+ICsgICAgfQo+ICsKPiArZXJyOgo+ICsgICAgaWYgKGVycl9jYXVzZSkgewo+ICsgICAgICAg IGVycm9yX3NldGcoZXJycCwgIkNhbid0IGVuYWJsZSBzaGFkb3cgdnEgb24gJXM6ICVzIiwgbmFt ZSwgZXJyX2NhdXNlKTsKPiArICAgIH0KPiAgIH0KPiAgIAo+ICAgY29uc3QgVmhvc3RPcHMgdmRw YV9vcHMgPSB7CgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f XwpWaXJ0dWFsaXphdGlvbiBtYWlsaW5nIGxpc3QKVmlydHVhbGl6YXRpb25AbGlzdHMubGludXgt Zm91bmRhdGlvbi5vcmcKaHR0cHM6Ly9saXN0cy5saW51eGZvdW5kYXRpb24ub3JnL21haWxtYW4v bGlzdGluZm8vdmlydHVhbGl6YXRpb24= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9970C433EF for ; Wed, 13 Oct 2021 03:29:34 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3318C60D43 for ; Wed, 13 Oct 2021 03:29:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 3318C60D43 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:44156 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1maUxR-0000Md-3Q for qemu-devel@archiver.kernel.org; Tue, 12 Oct 2021 23:29:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34296) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1maUvY-0007Iw-7e for qemu-devel@nongnu.org; Tue, 12 Oct 2021 23:27:36 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:45606) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1maUvT-00073h-6r for qemu-devel@nongnu.org; Tue, 12 Oct 2021 23:27:34 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1634095646; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=s+D8fjAlWfOwVxTjXsqSU9ga2khi1k6Zax2Zs/uECd0=; b=LFZ+nd0tsdH1JtwCjxfr6mUjjDC+vHbrmpMQqKQo2kEL2TAk57EJpXbU2pmySL6YUzFbij Oc/BHDIXTTffLfTWIF5YhjG+HgsFmcFG1CHqIm+csvPS+p6LUFkWHbKuTyplUtiug/suVF UCvuzMaCUvJ+MvfWXWGfdm4qexlTAXs= Received: from mail-pj1-f69.google.com (mail-pj1-f69.google.com [209.85.216.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-327-4qxFCCimPPq4ueRf_wvGXA-1; Tue, 12 Oct 2021 23:27:23 -0400 X-MC-Unique: 4qxFCCimPPq4ueRf_wvGXA-1 Received: by mail-pj1-f69.google.com with SMTP id nl15-20020a17090b384f00b001a0d49be015so633431pjb.1 for ; Tue, 12 Oct 2021 20:27:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=s+D8fjAlWfOwVxTjXsqSU9ga2khi1k6Zax2Zs/uECd0=; b=T5W75ldEKCJ7K3daBWuplHiZeVpYuZyu6bRluCTHmRpfS7/RKaq13oOZdbdCKS6ZHk uL7UktKjDlbPGrvTIlM43TLJgX3DicQLiuzBGDCjExKJ3AaFRrZqHQkF+r5GBrvHddJ0 gLgV8G6/ptx14Le36q+ktxZR9OaATCRSR8pulxAqX63sSG396sBMvlCzMzo5PB/C3aOe E0/nleE27Sh2XnADUojLGUCQGEsEYscIq6ke9BAAW/tQkbztcvLCzhK32CKplf5Rpxoj WQ4ZQ9GQt6If+g2kOjUN8vpQXGRmdgN4aOXZpvrlE7qF8C1wsEtOwbRzLl1b4LJIOQvL MEag== X-Gm-Message-State: AOAM532eDG4okQoG3nOtmAzTicn7/af/oZsWVneNN5Syh90AXBj5w+C+ zy5GCK+coima2EVTrCAbekOpPfm1mM9PhMSMkKjSzRJ5hfYToZXA+elOsiqz4ex//yoP5CzWe+d BSpLl8yvRa2Ox0o0= X-Received: by 2002:aa7:80d1:0:b029:399:ce3a:d617 with SMTP id a17-20020aa780d10000b0290399ce3ad617mr35233900pfn.16.1634095642300; Tue, 12 Oct 2021 20:27:22 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxpxA2JgacQ9XKMhHV8f1AW8OJ5PyYZKq8Ogru5GxisdlVBWuuRuZvwDoW6ciIRKPadY/Bxcw== X-Received: by 2002:aa7:80d1:0:b029:399:ce3a:d617 with SMTP id a17-20020aa780d10000b0290399ce3ad617mr35233864pfn.16.1634095641839; Tue, 12 Oct 2021 20:27:21 -0700 (PDT) Received: from wangxiaodeMacBook-Air.local ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id d24sm12615715pgv.52.2021.10.12.20.27.13 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 12 Oct 2021 20:27:21 -0700 (PDT) Subject: Re: [RFC PATCH v4 08/20] vhost: Route guest->host notification through shadow virtqueue To: =?UTF-8?Q?Eugenio_P=c3=a9rez?= , qemu-devel@nongnu.org References: <20211001070603.307037-1-eperezma@redhat.com> <20211001070603.307037-9-eperezma@redhat.com> From: Jason Wang Message-ID: <23e17562-719e-44da-3362-97e0ca5ae1ff@redhat.com> Date: Wed, 13 Oct 2021 11:27:11 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 MIME-Version: 1.0 In-Reply-To: <20211001070603.307037-9-eperezma@redhat.com> Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jasowang@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US Received-SPF: pass client-ip=216.205.24.124; envelope-from=jasowang@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.049, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, NICE_REPLY_A=-0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Parav Pandit , Juan Quintela , "Michael S. Tsirkin" , Markus Armbruster , virtualization@lists.linux-foundation.org, Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , Eric Blake , Michael Lilja , Stefano Garzarella Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" 在 2021/10/1 下午3:05, Eugenio Pérez 写道: > Shadow virtqueue notifications forwarding is disabled when vhost_dev > stops, so code flow follows usual cleanup. > > Also, host notifiers must be disabled at SVQ start, Any reason for this? > and they will not > start if SVQ has been enabled when device is stopped. This is trivial > to address, but it is left out for simplicity at this moment. It looks to me this patch also contains the following logics 1) codes to enable svq 2) codes to let svq to be enabled from QMP. I think they need to be split out, we may endup with the following series of patches 1) svq skeleton with enable/disable 2) route host notifier to svq 3) route guest notifier to svq 4) codes to enable svq 5) enable svq via QMP > > Signed-off-by: Eugenio Pérez > --- > qapi/net.json | 2 +- > hw/virtio/vhost-shadow-virtqueue.h | 8 ++ > include/hw/virtio/vhost-vdpa.h | 4 + > hw/virtio/vhost-shadow-virtqueue.c | 138 ++++++++++++++++++++++++++++- > hw/virtio/vhost-vdpa.c | 116 +++++++++++++++++++++++- > 5 files changed, 264 insertions(+), 4 deletions(-) > > diff --git a/qapi/net.json b/qapi/net.json > index a2c30fd455..fe546b0e7c 100644 > --- a/qapi/net.json > +++ b/qapi/net.json > @@ -88,7 +88,7 @@ > # > # @enable: true to use the alternate shadow VQ notifications > # > -# Returns: Always error, since SVQ is not implemented at the moment. > +# Returns: Error if failure, or 'no error' for success. > # > # Since: 6.2 > # > diff --git a/hw/virtio/vhost-shadow-virtqueue.h b/hw/virtio/vhost-shadow-virtqueue.h > index 27ac6388fa..237cfceb9c 100644 > --- a/hw/virtio/vhost-shadow-virtqueue.h > +++ b/hw/virtio/vhost-shadow-virtqueue.h > @@ -14,6 +14,14 @@ > > typedef struct VhostShadowVirtqueue VhostShadowVirtqueue; > > +EventNotifier *vhost_svq_get_svq_call_notifier(VhostShadowVirtqueue *svq); Let's move this function to another patch since it's unrelated to the guest->host routing. > +void vhost_svq_set_guest_call_notifier(VhostShadowVirtqueue *svq, int call_fd); > + > +bool vhost_svq_start(struct vhost_dev *dev, unsigned idx, > + VhostShadowVirtqueue *svq); > +void vhost_svq_stop(struct vhost_dev *dev, unsigned idx, > + VhostShadowVirtqueue *svq); > + > VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx); > > void vhost_svq_free(VhostShadowVirtqueue *vq); > diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h > index 0d565bb5bd..48aae59d8e 100644 > --- a/include/hw/virtio/vhost-vdpa.h > +++ b/include/hw/virtio/vhost-vdpa.h > @@ -12,6 +12,8 @@ > #ifndef HW_VIRTIO_VHOST_VDPA_H > #define HW_VIRTIO_VHOST_VDPA_H > > +#include > + > #include "qemu/queue.h" > #include "hw/virtio/virtio.h" > > @@ -24,6 +26,8 @@ typedef struct vhost_vdpa { > int device_fd; > uint32_t msg_type; > MemoryListener listener; > + bool shadow_vqs_enabled; > + GPtrArray *shadow_vqs; > struct vhost_dev *dev; > QLIST_ENTRY(vhost_vdpa) entry; > VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX]; > diff --git a/hw/virtio/vhost-shadow-virtqueue.c b/hw/virtio/vhost-shadow-virtqueue.c > index c4826a1b56..21dc99ab5d 100644 > --- a/hw/virtio/vhost-shadow-virtqueue.c > +++ b/hw/virtio/vhost-shadow-virtqueue.c > @@ -9,9 +9,12 @@ > > #include "qemu/osdep.h" > #include "hw/virtio/vhost-shadow-virtqueue.h" > +#include "hw/virtio/vhost.h" > + > +#include "standard-headers/linux/vhost_types.h" > > #include "qemu/error-report.h" > -#include "qemu/event_notifier.h" > +#include "qemu/main-loop.h" > > /* Shadow virtqueue to relay notifications */ > typedef struct VhostShadowVirtqueue { > @@ -19,14 +22,146 @@ typedef struct VhostShadowVirtqueue { > EventNotifier kick_notifier; > /* Shadow call notifier, sent to vhost */ > EventNotifier call_notifier; > + > + /* > + * Borrowed virtqueue's guest to host notifier. > + * To borrow it in this event notifier allows to register on the event > + * loop and access the associated shadow virtqueue easily. If we use the > + * VirtQueue, we don't have an easy way to retrieve it. > + * > + * So shadow virtqueue must not clean it, or we would lose VirtQueue one. > + */ > + EventNotifier host_notifier; > + > + /* Guest's call notifier, where SVQ calls guest. */ > + EventNotifier guest_call_notifier; To be consistent, let's simply use "guest_notifier" here. > + > + /* Virtio queue shadowing */ > + VirtQueue *vq; > } VhostShadowVirtqueue; > > +/* Forward guest notifications */ > +static void vhost_handle_guest_kick(EventNotifier *n) > +{ > + VhostShadowVirtqueue *svq = container_of(n, VhostShadowVirtqueue, > + host_notifier); > + > + if (unlikely(!event_notifier_test_and_clear(n))) { > + return; > + } Is there a chance that we may stop the processing of available buffers during the svq enabling? There could be no kick from the guest in this case. > + > + event_notifier_set(&svq->kick_notifier); > +} > + > +/* > + * Obtain the SVQ call notifier, where vhost device notifies SVQ that there > + * exists pending used buffers. > + * > + * @svq Shadow Virtqueue > + */ > +EventNotifier *vhost_svq_get_svq_call_notifier(VhostShadowVirtqueue *svq) > +{ > + return &svq->call_notifier; > +} > + > +/* > + * Set the call notifier for the SVQ to call the guest > + * > + * @svq Shadow virtqueue > + * @call_fd call notifier > + * > + * Called on BQL context. > + */ > +void vhost_svq_set_guest_call_notifier(VhostShadowVirtqueue *svq, int call_fd) > +{ > + event_notifier_init_fd(&svq->guest_call_notifier, call_fd); > +} > + > +/* > + * Restore the vhost guest to host notifier, i.e., disables svq effect. > + */ > +static int vhost_svq_restore_vdev_host_notifier(struct vhost_dev *dev, > + unsigned vhost_index, > + VhostShadowVirtqueue *svq) > +{ > + EventNotifier *vq_host_notifier = virtio_queue_get_host_notifier(svq->vq); > + struct vhost_vring_file file = { > + .index = vhost_index, > + .fd = event_notifier_get_fd(vq_host_notifier), > + }; > + int r; > + > + /* Restore vhost kick */ > + r = dev->vhost_ops->vhost_set_vring_kick(dev, &file); And remap the notification area if necessary. > + return r ? -errno : 0; > +} > + > +/* > + * Start shadow virtqueue operation. > + * @dev vhost device > + * @hidx vhost virtqueue index > + * @svq Shadow Virtqueue > + */ > +bool vhost_svq_start(struct vhost_dev *dev, unsigned idx, > + VhostShadowVirtqueue *svq) > +{ > + EventNotifier *vq_host_notifier = virtio_queue_get_host_notifier(svq->vq); > + struct vhost_vring_file file = { > + .index = dev->vhost_ops->vhost_get_vq_index(dev, dev->vq_index + idx), > + .fd = event_notifier_get_fd(&svq->kick_notifier), > + }; > + int r; > + > + /* Check that notifications are still going directly to vhost dev */ > + assert(virtio_queue_is_host_notifier_enabled(svq->vq)); > + > + /* > + * event_notifier_set_handler already checks for guest's notifications if > + * they arrive in the switch, so there is no need to explicitely check for > + * them. > + */ If this is true, shouldn't we call vhost_set_vring_kick() before the event_notifier_set_handler()? Btw, I think we should update the fd if set_vring_kick() was called after this function? > + event_notifier_init_fd(&svq->host_notifier, > + event_notifier_get_fd(vq_host_notifier)); > + event_notifier_set_handler(&svq->host_notifier, vhost_handle_guest_kick); > + > + r = dev->vhost_ops->vhost_set_vring_kick(dev, &file); And we need to stop the notification area mmap. > + if (unlikely(r != 0)) { > + error_report("Couldn't set kick fd: %s", strerror(errno)); > + goto err_set_vring_kick; > + } > + > + return true; > + > +err_set_vring_kick: > + event_notifier_set_handler(&svq->host_notifier, NULL); > + > + return false; > +} > + > +/* > + * Stop shadow virtqueue operation. > + * @dev vhost device > + * @idx vhost queue index > + * @svq Shadow Virtqueue > + */ > +void vhost_svq_stop(struct vhost_dev *dev, unsigned idx, > + VhostShadowVirtqueue *svq) > +{ > + int r = vhost_svq_restore_vdev_host_notifier(dev, idx, svq); > + if (unlikely(r < 0)) { > + error_report("Couldn't restore vq kick fd: %s", strerror(-r)); > + } > + > + event_notifier_set_handler(&svq->host_notifier, NULL); > +} > + > /* > * Creates vhost shadow virtqueue, and instruct vhost device to use the shadow > * methods and file descriptors. > */ > VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx) > { > + int vq_idx = dev->vq_index + idx; > g_autofree VhostShadowVirtqueue *svq = g_new0(VhostShadowVirtqueue, 1); > int r; > > @@ -44,6 +179,7 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx) > goto err_init_call_notifier; > } > > + svq->vq = virtio_get_queue(dev->vdev, vq_idx); > return g_steal_pointer(&svq); > > err_init_call_notifier: > diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c > index e0dc7508c3..36c954a779 100644 > --- a/hw/virtio/vhost-vdpa.c > +++ b/hw/virtio/vhost-vdpa.c > @@ -17,6 +17,7 @@ > #include "hw/virtio/vhost.h" > #include "hw/virtio/vhost-backend.h" > #include "hw/virtio/virtio-net.h" > +#include "hw/virtio/vhost-shadow-virtqueue.h" > #include "hw/virtio/vhost-vdpa.h" > #include "exec/address-spaces.h" > #include "qemu/main-loop.h" > @@ -272,6 +273,16 @@ static void vhost_vdpa_add_status(struct vhost_dev *dev, uint8_t status) > vhost_vdpa_call(dev, VHOST_VDPA_SET_STATUS, &s); > } > > +/** > + * Adaptor function to free shadow virtqueue through gpointer > + * > + * @svq The Shadow Virtqueue > + */ > +static void vhost_psvq_free(gpointer svq) > +{ > + vhost_svq_free(svq); > +} > + > static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) > { > struct vhost_vdpa *v; > @@ -283,6 +294,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) > dev->opaque = opaque ; > v->listener = vhost_vdpa_memory_listener; > v->msg_type = VHOST_IOTLB_MSG_V2; > + v->shadow_vqs = g_ptr_array_new_full(dev->nvqs, vhost_psvq_free); > QLIST_INSERT_HEAD(&vhost_vdpa_devices, v, entry); > > vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE | > @@ -373,6 +385,17 @@ err: > return; > } > > +static void vhost_vdpa_svq_cleanup(struct vhost_dev *dev) > +{ > + struct vhost_vdpa *v = dev->opaque; > + size_t idx; > + > + for (idx = 0; idx < v->shadow_vqs->len; ++idx) { > + vhost_svq_stop(dev, idx, g_ptr_array_index(v->shadow_vqs, idx)); > + } > + g_ptr_array_free(v->shadow_vqs, true); > +} > + > static int vhost_vdpa_cleanup(struct vhost_dev *dev) > { > struct vhost_vdpa *v; > @@ -381,6 +404,7 @@ static int vhost_vdpa_cleanup(struct vhost_dev *dev) > trace_vhost_vdpa_cleanup(dev, v); > vhost_vdpa_host_notifiers_uninit(dev, dev->nvqs); > memory_listener_unregister(&v->listener); > + vhost_vdpa_svq_cleanup(dev); > QLIST_REMOVE(v, entry); > > dev->opaque = NULL; > @@ -557,7 +581,9 @@ static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started) > if (started) { > uint8_t status = 0; > memory_listener_register(&v->listener, &address_space_memory); > - vhost_vdpa_host_notifiers_init(dev); > + if (!v->shadow_vqs_enabled) { > + vhost_vdpa_host_notifiers_init(dev); > + } This looks like a trick, why not check and setup shadow_vqs inside: 1) vhost_vdpa_host_notifiers_init() and 2) vhost_vdpa_set_vring_kick() > vhost_vdpa_set_vring_ready(dev); > vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_DRIVER_OK); > vhost_vdpa_call(dev, VHOST_VDPA_GET_STATUS, &status); > @@ -663,10 +689,96 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev *dev) > return true; > } > > +/* > + * Start shadow virtqueue. > + */ > +static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) > +{ > + struct vhost_vdpa *v = dev->opaque; > + VhostShadowVirtqueue *svq = g_ptr_array_index(v->shadow_vqs, idx); > + return vhost_svq_start(dev, idx, svq); > +} > + > +static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) > +{ > + struct vhost_dev *hdev = v->dev; > + unsigned n; > + > + if (enable == v->shadow_vqs_enabled) { > + return hdev->nvqs; > + } > + > + if (enable) { > + /* Allocate resources */ > + assert(v->shadow_vqs->len == 0); > + for (n = 0; n < hdev->nvqs; ++n) { > + VhostShadowVirtqueue *svq = vhost_svq_new(hdev, n); > + bool ok; > + > + if (unlikely(!svq)) { > + g_ptr_array_set_size(v->shadow_vqs, 0); > + return 0; > + } > + g_ptr_array_add(v->shadow_vqs, svq); > + > + ok = vhost_vdpa_svq_start_vq(hdev, n); > + if (unlikely(!ok)) { > + /* Free still not started svqs */ > + g_ptr_array_set_size(v->shadow_vqs, n); > + enable = false; > + break; > + } > + } Since there's almost no logic could be shared between enable and disable. Let's split those logic out into dedicated functions where the codes looks more easy to be reviewed (e.g have a better error handling etc). > + } > + > + v->shadow_vqs_enabled = enable; > + > + if (!enable) { > + /* Disable all queues or clean up failed start */ > + for (n = 0; n < v->shadow_vqs->len; ++n) { > + unsigned vq_idx = vhost_vdpa_get_vq_index(hdev, n); > + VhostShadowVirtqueue *svq = g_ptr_array_index(v->shadow_vqs, n); > + vhost_svq_stop(hdev, n, svq); > + vhost_virtqueue_start(hdev, hdev->vdev, &hdev->vqs[n], vq_idx); > + } > + > + /* Resources cleanup */ > + g_ptr_array_set_size(v->shadow_vqs, 0); > + } > + > + return n; > +} > > void qmp_x_vhost_enable_shadow_vq(const char *name, bool enable, Error **errp) > { > - error_setg(errp, "Shadow virtqueue still not implemented"); > + struct vhost_vdpa *v; > + const char *err_cause = NULL; > + bool r; > + > + QLIST_FOREACH(v, &vhost_vdpa_devices, entry) { > + if (v->dev->vdev && 0 == strcmp(v->dev->vdev->name, name)) { > + break; > + } > + } I think you can iterate the NetClientStates to ge tthe vhost-vdpa backends. > + > + if (!v) { > + err_cause = "Device not found"; > + goto err; > + } else if (v->notifier[0].addr) { > + err_cause = "Device has host notifiers enabled"; I don't get this. Btw this function should be implemented in an independent patch after svq is fully functional. Thanks > + goto err; > + } > + > + r = vhost_vdpa_enable_svq(v, enable); > + if (unlikely(!r)) { > + err_cause = "Error enabling (see monitor)"; > + goto err; > + } > + > +err: > + if (err_cause) { > + error_setg(errp, "Can't enable shadow vq on %s: %s", name, err_cause); > + } > } > > const VhostOps vdpa_ops = {