From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEED9C46460 for ; Wed, 15 May 2019 02:49:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 880B320862 for ; Wed, 15 May 2019 02:49:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726467AbfEOCs6 (ORCPT ); Tue, 14 May 2019 22:48:58 -0400 Received: from mx1.redhat.com ([209.132.183.28]:40508 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726211AbfEOCs6 (ORCPT ); Tue, 14 May 2019 22:48:58 -0400 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7D8785D5FD; Wed, 15 May 2019 02:48:57 +0000 (UTC) Received: from [10.72.12.103] (ovpn-12-103.pek2.redhat.com [10.72.12.103]) by smtp.corp.redhat.com (Postfix) with ESMTP id 653CF1001F40; Wed, 15 May 2019 02:48:44 +0000 (UTC) Subject: Re: [PATCH v2 1/8] vsock/virtio: limit the memory used per-socket To: Stefano Garzarella Cc: netdev@vger.kernel.org, "David S. Miller" , "Michael S. Tsirkin" , virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Stefan Hajnoczi References: <20190510125843.95587-1-sgarzare@redhat.com> <20190510125843.95587-2-sgarzare@redhat.com> <3b275b52-63d9-d260-1652-8e8bf7dd679f@redhat.com> <20190513172322.vcgenx7xk4v6r2ay@steredhat> <20190514163500.a7moalixvpn5mkcr@steredhat> From: Jason Wang Message-ID: <034a5081-b4fb-011f-b5b7-fbf293c13b23@redhat.com> Date: Wed, 15 May 2019 10:48:44 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <20190514163500.a7moalixvpn5mkcr@steredhat> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Wed, 15 May 2019 02:48:57 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2019/5/15 上午12:35, Stefano Garzarella wrote: > On Tue, May 14, 2019 at 11:25:34AM +0800, Jason Wang wrote: >> On 2019/5/14 上午1:23, Stefano Garzarella wrote: >>> On Mon, May 13, 2019 at 05:58:53PM +0800, Jason Wang wrote: >>>> On 2019/5/10 下午8:58, Stefano Garzarella wrote: >>>>> Since virtio-vsock was introduced, the buffers filled by the host >>>>> and pushed to the guest using the vring, are directly queued in >>>>> a per-socket list avoiding to copy it. >>>>> These buffers are preallocated by the guest with a fixed >>>>> size (4 KB). >>>>> >>>>> The maximum amount of memory used by each socket should be >>>>> controlled by the credit mechanism. >>>>> The default credit available per-socket is 256 KB, but if we use >>>>> only 1 byte per packet, the guest can queue up to 262144 of 4 KB >>>>> buffers, using up to 1 GB of memory per-socket. In addition, the >>>>> guest will continue to fill the vring with new 4 KB free buffers >>>>> to avoid starvation of her sockets. >>>>> >>>>> This patch solves this issue copying the payload in a new buffer. >>>>> Then it is queued in the per-socket list, and the 4KB buffer used >>>>> by the host is freed. >>>>> >>>>> In this way, the memory used by each socket respects the credit >>>>> available, and we still avoid starvation, paying the cost of an >>>>> extra memory copy. When the buffer is completely full we do a >>>>> "zero-copy", moving the buffer directly in the per-socket list. >>>> I wonder in the long run we should use generic socket accouting mechanism >>>> provided by kernel (e.g socket, skb, sndbuf, recvbug, truesize) instead of >>>> vsock specific thing to avoid duplicating efforts. >>> I agree, the idea is to switch to sk_buff but this should require an huge >>> change. If we will use the virtio-net datapath, it will become simpler. >> >> Yes, unix domain socket is one example that uses general skb and socket >> structure. And we probably need some kind of socket pair on host. Using >> socket can also simplify the unification with vhost-net which depends on the >> socket proto_ops to work. I admit it's a huge change probably, we can do it >> gradually. >> > Yes, I also prefer to do this change gradually :) > >>>>> Signed-off-by: Stefano Garzarella >>>>> --- >>>>> drivers/vhost/vsock.c | 2 + >>>>> include/linux/virtio_vsock.h | 8 +++ >>>>> net/vmw_vsock/virtio_transport.c | 1 + >>>>> net/vmw_vsock/virtio_transport_common.c | 95 ++++++++++++++++++------- >>>>> 4 files changed, 81 insertions(+), 25 deletions(-) >>>>> >>>>> diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c >>>>> index bb5fc0e9fbc2..7964e2daee09 100644 >>>>> --- a/drivers/vhost/vsock.c >>>>> +++ b/drivers/vhost/vsock.c >>>>> @@ -320,6 +320,8 @@ vhost_vsock_alloc_pkt(struct vhost_virtqueue *vq, >>>>> return NULL; >>>>> } >>>>> + pkt->buf_len = pkt->len; >>>>> + >>>>> nbytes = copy_from_iter(pkt->buf, pkt->len, &iov_iter); >>>>> if (nbytes != pkt->len) { >>>>> vq_err(vq, "Expected %u byte payload, got %zu bytes\n", >>>>> diff --git a/include/linux/virtio_vsock.h b/include/linux/virtio_vsock.h >>>>> index e223e2632edd..345f04ee9193 100644 >>>>> --- a/include/linux/virtio_vsock.h >>>>> +++ b/include/linux/virtio_vsock.h >>>>> @@ -54,9 +54,17 @@ struct virtio_vsock_pkt { >>>>> void *buf; >>>>> u32 len; >>>>> u32 off; >>>>> + u32 buf_len; >>>>> bool reply; >>>>> }; >>>>> +struct virtio_vsock_buf { >>>>> + struct list_head list; >>>>> + void *addr; >>>>> + u32 len; >>>>> + u32 off; >>>>> +}; >>>>> + >>>>> struct virtio_vsock_pkt_info { >>>>> u32 remote_cid, remote_port; >>>>> struct vsock_sock *vsk; >>>>> diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c >>>>> index 15eb5d3d4750..af1d2ce12f54 100644 >>>>> --- a/net/vmw_vsock/virtio_transport.c >>>>> +++ b/net/vmw_vsock/virtio_transport.c >>>>> @@ -280,6 +280,7 @@ static void virtio_vsock_rx_fill(struct virtio_vsock *vsock) >>>>> break; >>>>> } >>>>> + pkt->buf_len = buf_len; >>>>> pkt->len = buf_len; >>>>> sg_init_one(&hdr, &pkt->hdr, sizeof(pkt->hdr)); >>>>> diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c >>>>> index 602715fc9a75..0248d6808755 100644 >>>>> --- a/net/vmw_vsock/virtio_transport_common.c >>>>> +++ b/net/vmw_vsock/virtio_transport_common.c >>>>> @@ -65,6 +65,9 @@ virtio_transport_alloc_pkt(struct virtio_vsock_pkt_info *info, >>>>> pkt->buf = kmalloc(len, GFP_KERNEL); >>>>> if (!pkt->buf) >>>>> goto out_pkt; >>>>> + >>>>> + pkt->buf_len = len; >>>>> + >>>>> err = memcpy_from_msg(pkt->buf, info->msg, len); >>>>> if (err) >>>>> goto out; >>>>> @@ -86,6 +89,46 @@ virtio_transport_alloc_pkt(struct virtio_vsock_pkt_info *info, >>>>> return NULL; >>>>> } >>>>> +static struct virtio_vsock_buf * >>>>> +virtio_transport_alloc_buf(struct virtio_vsock_pkt *pkt, bool zero_copy) >>>>> +{ >>>>> + struct virtio_vsock_buf *buf; >>>>> + >>>>> + if (pkt->len == 0) >>>>> + return NULL; >>>>> + >>>>> + buf = kzalloc(sizeof(*buf), GFP_KERNEL); >>>>> + if (!buf) >>>>> + return NULL; >>>>> + >>>>> + /* If the buffer in the virtio_vsock_pkt is full, we can move it to >>>>> + * the new virtio_vsock_buf avoiding the copy, because we are sure that >>>>> + * we are not use more memory than that counted by the credit mechanism. >>>>> + */ >>>>> + if (zero_copy && pkt->len == pkt->buf_len) { >>>>> + buf->addr = pkt->buf; >>>>> + pkt->buf = NULL; >>>>> + } else { >>>> Is the copy still needed if we're just few bytes less? We meet similar issue >>>> for virito-net, and virtio-net solve this by always copy first 128bytes for >>>> big packets. >>>> >>>> See receive_big() >>> I'm seeing, It is more sophisticated. >>> IIUC, virtio-net allocates a sk_buff with 128 bytes of buffer, then copies the >>> first 128 bytes, then adds the buffer used to receive the packet as a frag to >>> the skb. >> >> Yes and the point is if the packet is smaller than 128 bytes the pages will >> be recycled. >> >> > So it's avoid the overhead of allocation of a large buffer. I got it. > > Just a curiosity, why the threshold is 128 bytes? From its name (GOOD_COPY_LEN), I think it just a value that won't lose much performance, e.g the size two cachelines. Thanks > >>> Do you suggest to implement something similar, or for now we can use my >>> approach and if we will merge the datapath we can reuse the virtio-net >>> approach? >> >> I think we need a better threshold. If I understand the patch correctly, we >> will do copy unless the packet is 64K when guest is doing receiving. 1 byte >> packet is indeed a problem, but we need to solve it without losing too much >> performance. > It is correct. I'll try to figure out a better threshold and the usage of > order 0 page. > > Thanks again for your advices, > Stefano From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jason Wang Subject: Re: [PATCH v2 1/8] vsock/virtio: limit the memory used per-socket Date: Wed, 15 May 2019 10:48:44 +0800 Message-ID: <034a5081-b4fb-011f-b5b7-fbf293c13b23@redhat.com> References: <20190510125843.95587-1-sgarzare@redhat.com> <20190510125843.95587-2-sgarzare@redhat.com> <3b275b52-63d9-d260-1652-8e8bf7dd679f@redhat.com> <20190513172322.vcgenx7xk4v6r2ay@steredhat> <20190514163500.a7moalixvpn5mkcr@steredhat> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8"; Format="flowed" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <20190514163500.a7moalixvpn5mkcr@steredhat> Content-Language: en-US List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: virtualization-bounces@lists.linux-foundation.org Errors-To: virtualization-bounces@lists.linux-foundation.org To: Stefano Garzarella Cc: kvm@vger.kernel.org, "Michael S. Tsirkin" , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, Stefan Hajnoczi , "David S. Miller" List-Id: virtualization@lists.linuxfoundation.org Ck9uIDIwMTkvNS8xNSDkuIrljYgxMjozNSwgU3RlZmFubyBHYXJ6YXJlbGxhIHdyb3RlOgo+IE9u IFR1ZSwgTWF5IDE0LCAyMDE5IGF0IDExOjI1OjM0QU0gKzA4MDAsIEphc29uIFdhbmcgd3JvdGU6 Cj4+IE9uIDIwMTkvNS8xNCDkuIrljYgxOjIzLCBTdGVmYW5vIEdhcnphcmVsbGEgd3JvdGU6Cj4+ PiBPbiBNb24sIE1heSAxMywgMjAxOSBhdCAwNTo1ODo1M1BNICswODAwLCBKYXNvbiBXYW5nIHdy b3RlOgo+Pj4+IE9uIDIwMTkvNS8xMCDkuIvljYg4OjU4LCBTdGVmYW5vIEdhcnphcmVsbGEgd3Jv dGU6Cj4+Pj4+IFNpbmNlIHZpcnRpby12c29jayB3YXMgaW50cm9kdWNlZCwgdGhlIGJ1ZmZlcnMg ZmlsbGVkIGJ5IHRoZSBob3N0Cj4+Pj4+IGFuZCBwdXNoZWQgdG8gdGhlIGd1ZXN0IHVzaW5nIHRo ZSB2cmluZywgYXJlIGRpcmVjdGx5IHF1ZXVlZCBpbgo+Pj4+PiBhIHBlci1zb2NrZXQgbGlzdCBh dm9pZGluZyB0byBjb3B5IGl0Lgo+Pj4+PiBUaGVzZSBidWZmZXJzIGFyZSBwcmVhbGxvY2F0ZWQg YnkgdGhlIGd1ZXN0IHdpdGggYSBmaXhlZAo+Pj4+PiBzaXplICg0IEtCKS4KPj4+Pj4KPj4+Pj4g VGhlIG1heGltdW0gYW1vdW50IG9mIG1lbW9yeSB1c2VkIGJ5IGVhY2ggc29ja2V0IHNob3VsZCBi ZQo+Pj4+PiBjb250cm9sbGVkIGJ5IHRoZSBjcmVkaXQgbWVjaGFuaXNtLgo+Pj4+PiBUaGUgZGVm YXVsdCBjcmVkaXQgYXZhaWxhYmxlIHBlci1zb2NrZXQgaXMgMjU2IEtCLCBidXQgaWYgd2UgdXNl Cj4+Pj4+IG9ubHkgMSBieXRlIHBlciBwYWNrZXQsIHRoZSBndWVzdCBjYW4gcXVldWUgdXAgdG8g MjYyMTQ0IG9mIDQgS0IKPj4+Pj4gYnVmZmVycywgdXNpbmcgdXAgdG8gMSBHQiBvZiBtZW1vcnkg cGVyLXNvY2tldC4gSW4gYWRkaXRpb24sIHRoZQo+Pj4+PiBndWVzdCB3aWxsIGNvbnRpbnVlIHRv IGZpbGwgdGhlIHZyaW5nIHdpdGggbmV3IDQgS0IgZnJlZSBidWZmZXJzCj4+Pj4+IHRvIGF2b2lk IHN0YXJ2YXRpb24gb2YgaGVyIHNvY2tldHMuCj4+Pj4+Cj4+Pj4+IFRoaXMgcGF0Y2ggc29sdmVz IHRoaXMgaXNzdWUgY29weWluZyB0aGUgcGF5bG9hZCBpbiBhIG5ldyBidWZmZXIuCj4+Pj4+IFRo ZW4gaXQgaXMgcXVldWVkIGluIHRoZSBwZXItc29ja2V0IGxpc3QsIGFuZCB0aGUgNEtCIGJ1ZmZl ciB1c2VkCj4+Pj4+IGJ5IHRoZSBob3N0IGlzIGZyZWVkLgo+Pj4+Pgo+Pj4+PiBJbiB0aGlzIHdh eSwgdGhlIG1lbW9yeSB1c2VkIGJ5IGVhY2ggc29ja2V0IHJlc3BlY3RzIHRoZSBjcmVkaXQKPj4+ Pj4gYXZhaWxhYmxlLCBhbmQgd2Ugc3RpbGwgYXZvaWQgc3RhcnZhdGlvbiwgcGF5aW5nIHRoZSBj b3N0IG9mIGFuCj4+Pj4+IGV4dHJhIG1lbW9yeSBjb3B5LiBXaGVuIHRoZSBidWZmZXIgaXMgY29t cGxldGVseSBmdWxsIHdlIGRvIGEKPj4+Pj4gInplcm8tY29weSIsIG1vdmluZyB0aGUgYnVmZmVy IGRpcmVjdGx5IGluIHRoZSBwZXItc29ja2V0IGxpc3QuCj4+Pj4gSSB3b25kZXIgaW4gdGhlIGxv bmcgcnVuIHdlIHNob3VsZCB1c2UgZ2VuZXJpYyBzb2NrZXQgYWNjb3V0aW5nIG1lY2hhbmlzbQo+ Pj4+IHByb3ZpZGVkIGJ5IGtlcm5lbCAoZS5nIHNvY2tldCwgc2tiLCBzbmRidWYsIHJlY3ZidWcs IHRydWVzaXplKSBpbnN0ZWFkIG9mCj4+Pj4gdnNvY2sgc3BlY2lmaWMgdGhpbmcgdG8gYXZvaWQg ZHVwbGljYXRpbmcgZWZmb3J0cy4KPj4+IEkgYWdyZWUsIHRoZSBpZGVhIGlzIHRvIHN3aXRjaCB0 byBza19idWZmIGJ1dCB0aGlzIHNob3VsZCByZXF1aXJlIGFuIGh1Z2UKPj4+IGNoYW5nZS4gSWYg d2Ugd2lsbCB1c2UgdGhlIHZpcnRpby1uZXQgZGF0YXBhdGgsIGl0IHdpbGwgYmVjb21lIHNpbXBs ZXIuCj4+Cj4+IFllcywgdW5peCBkb21haW4gc29ja2V0IGlzIG9uZSBleGFtcGxlIHRoYXQgdXNl cyBnZW5lcmFsIHNrYiBhbmQgc29ja2V0Cj4+IHN0cnVjdHVyZS4gQW5kIHdlIHByb2JhYmx5IG5l ZWQgc29tZSBraW5kIG9mIHNvY2tldCBwYWlyIG9uIGhvc3QuIFVzaW5nCj4+IHNvY2tldCBjYW4g YWxzbyBzaW1wbGlmeSB0aGUgdW5pZmljYXRpb24gd2l0aCB2aG9zdC1uZXQgd2hpY2ggZGVwZW5k cyBvbiB0aGUKPj4gc29ja2V0IHByb3RvX29wcyB0byB3b3JrLiBJIGFkbWl0IGl0J3MgYSBodWdl IGNoYW5nZSBwcm9iYWJseSwgd2UgY2FuIGRvIGl0Cj4+IGdyYWR1YWxseS4KPj4KPiBZZXMsIEkg YWxzbyBwcmVmZXIgdG8gZG8gdGhpcyBjaGFuZ2UgZ3JhZHVhbGx5IDopCj4KPj4+Pj4gU2lnbmVk LW9mZi1ieTogU3RlZmFubyBHYXJ6YXJlbGxhIDxzZ2FyemFyZUByZWRoYXQuY29tPgo+Pj4+PiAt LS0KPj4+Pj4gICAgIGRyaXZlcnMvdmhvc3QvdnNvY2suYyAgICAgICAgICAgICAgICAgICB8ICAy ICsKPj4+Pj4gICAgIGluY2x1ZGUvbGludXgvdmlydGlvX3Zzb2NrLmggICAgICAgICAgICB8ICA4 ICsrKwo+Pj4+PiAgICAgbmV0L3Ztd192c29jay92aXJ0aW9fdHJhbnNwb3J0LmMgICAgICAgIHwg IDEgKwo+Pj4+PiAgICAgbmV0L3Ztd192c29jay92aXJ0aW9fdHJhbnNwb3J0X2NvbW1vbi5jIHwg OTUgKysrKysrKysrKysrKysrKysrLS0tLS0tLQo+Pj4+PiAgICAgNCBmaWxlcyBjaGFuZ2VkLCA4 MSBpbnNlcnRpb25zKCspLCAyNSBkZWxldGlvbnMoLSkKPj4+Pj4KPj4+Pj4gZGlmZiAtLWdpdCBh L2RyaXZlcnMvdmhvc3QvdnNvY2suYyBiL2RyaXZlcnMvdmhvc3QvdnNvY2suYwo+Pj4+PiBpbmRl eCBiYjVmYzBlOWZiYzIuLjc5NjRlMmRhZWUwOSAxMDA2NDQKPj4+Pj4gLS0tIGEvZHJpdmVycy92 aG9zdC92c29jay5jCj4+Pj4+ICsrKyBiL2RyaXZlcnMvdmhvc3QvdnNvY2suYwo+Pj4+PiBAQCAt MzIwLDYgKzMyMCw4IEBAIHZob3N0X3Zzb2NrX2FsbG9jX3BrdChzdHJ1Y3Qgdmhvc3RfdmlydHF1 ZXVlICp2cSwKPj4+Pj4gICAgIAkJcmV0dXJuIE5VTEw7Cj4+Pj4+ICAgICAJfQo+Pj4+PiArCXBr dC0+YnVmX2xlbiA9IHBrdC0+bGVuOwo+Pj4+PiArCj4+Pj4+ICAgICAJbmJ5dGVzID0gY29weV9m cm9tX2l0ZXIocGt0LT5idWYsIHBrdC0+bGVuLCAmaW92X2l0ZXIpOwo+Pj4+PiAgICAgCWlmIChu Ynl0ZXMgIT0gcGt0LT5sZW4pIHsKPj4+Pj4gICAgIAkJdnFfZXJyKHZxLCAiRXhwZWN0ZWQgJXUg Ynl0ZSBwYXlsb2FkLCBnb3QgJXp1IGJ5dGVzXG4iLAo+Pj4+PiBkaWZmIC0tZ2l0IGEvaW5jbHVk ZS9saW51eC92aXJ0aW9fdnNvY2suaCBiL2luY2x1ZGUvbGludXgvdmlydGlvX3Zzb2NrLmgKPj4+ Pj4gaW5kZXggZTIyM2UyNjMyZWRkLi4zNDVmMDRlZTkxOTMgMTAwNjQ0Cj4+Pj4+IC0tLSBhL2lu Y2x1ZGUvbGludXgvdmlydGlvX3Zzb2NrLmgKPj4+Pj4gKysrIGIvaW5jbHVkZS9saW51eC92aXJ0 aW9fdnNvY2suaAo+Pj4+PiBAQCAtNTQsOSArNTQsMTcgQEAgc3RydWN0IHZpcnRpb192c29ja19w a3Qgewo+Pj4+PiAgICAgCXZvaWQgKmJ1ZjsKPj4+Pj4gICAgIAl1MzIgbGVuOwo+Pj4+PiAgICAg CXUzMiBvZmY7Cj4+Pj4+ICsJdTMyIGJ1Zl9sZW47Cj4+Pj4+ICAgICAJYm9vbCByZXBseTsKPj4+ Pj4gICAgIH07Cj4+Pj4+ICtzdHJ1Y3QgdmlydGlvX3Zzb2NrX2J1ZiB7Cj4+Pj4+ICsJc3RydWN0 IGxpc3RfaGVhZCBsaXN0Owo+Pj4+PiArCXZvaWQgKmFkZHI7Cj4+Pj4+ICsJdTMyIGxlbjsKPj4+ Pj4gKwl1MzIgb2ZmOwo+Pj4+PiArfTsKPj4+Pj4gKwo+Pj4+PiAgICAgc3RydWN0IHZpcnRpb192 c29ja19wa3RfaW5mbyB7Cj4+Pj4+ICAgICAJdTMyIHJlbW90ZV9jaWQsIHJlbW90ZV9wb3J0Owo+ Pj4+PiAgICAgCXN0cnVjdCB2c29ja19zb2NrICp2c2s7Cj4+Pj4+IGRpZmYgLS1naXQgYS9uZXQv dm13X3Zzb2NrL3ZpcnRpb190cmFuc3BvcnQuYyBiL25ldC92bXdfdnNvY2svdmlydGlvX3RyYW5z cG9ydC5jCj4+Pj4+IGluZGV4IDE1ZWI1ZDNkNDc1MC4uYWYxZDJjZTEyZjU0IDEwMDY0NAo+Pj4+ PiAtLS0gYS9uZXQvdm13X3Zzb2NrL3ZpcnRpb190cmFuc3BvcnQuYwo+Pj4+PiArKysgYi9uZXQv dm13X3Zzb2NrL3ZpcnRpb190cmFuc3BvcnQuYwo+Pj4+PiBAQCAtMjgwLDYgKzI4MCw3IEBAIHN0 YXRpYyB2b2lkIHZpcnRpb192c29ja19yeF9maWxsKHN0cnVjdCB2aXJ0aW9fdnNvY2sgKnZzb2Nr KQo+Pj4+PiAgICAgCQkJYnJlYWs7Cj4+Pj4+ICAgICAJCX0KPj4+Pj4gKwkJcGt0LT5idWZfbGVu ID0gYnVmX2xlbjsKPj4+Pj4gICAgIAkJcGt0LT5sZW4gPSBidWZfbGVuOwo+Pj4+PiAgICAgCQlz Z19pbml0X29uZSgmaGRyLCAmcGt0LT5oZHIsIHNpemVvZihwa3QtPmhkcikpOwo+Pj4+PiBkaWZm IC0tZ2l0IGEvbmV0L3Ztd192c29jay92aXJ0aW9fdHJhbnNwb3J0X2NvbW1vbi5jIGIvbmV0L3Zt d192c29jay92aXJ0aW9fdHJhbnNwb3J0X2NvbW1vbi5jCj4+Pj4+IGluZGV4IDYwMjcxNWZjOWE3 NS4uMDI0OGQ2ODA4NzU1IDEwMDY0NAo+Pj4+PiAtLS0gYS9uZXQvdm13X3Zzb2NrL3ZpcnRpb190 cmFuc3BvcnRfY29tbW9uLmMKPj4+Pj4gKysrIGIvbmV0L3Ztd192c29jay92aXJ0aW9fdHJhbnNw b3J0X2NvbW1vbi5jCj4+Pj4+IEBAIC02NSw2ICs2NSw5IEBAIHZpcnRpb190cmFuc3BvcnRfYWxs b2NfcGt0KHN0cnVjdCB2aXJ0aW9fdnNvY2tfcGt0X2luZm8gKmluZm8sCj4+Pj4+ICAgICAJCXBr dC0+YnVmID0ga21hbGxvYyhsZW4sIEdGUF9LRVJORUwpOwo+Pj4+PiAgICAgCQlpZiAoIXBrdC0+ YnVmKQo+Pj4+PiAgICAgCQkJZ290byBvdXRfcGt0Owo+Pj4+PiArCj4+Pj4+ICsJCXBrdC0+YnVm X2xlbiA9IGxlbjsKPj4+Pj4gKwo+Pj4+PiAgICAgCQllcnIgPSBtZW1jcHlfZnJvbV9tc2cocGt0 LT5idWYsIGluZm8tPm1zZywgbGVuKTsKPj4+Pj4gICAgIAkJaWYgKGVycikKPj4+Pj4gICAgIAkJ CWdvdG8gb3V0Owo+Pj4+PiBAQCAtODYsNiArODksNDYgQEAgdmlydGlvX3RyYW5zcG9ydF9hbGxv Y19wa3Qoc3RydWN0IHZpcnRpb192c29ja19wa3RfaW5mbyAqaW5mbywKPj4+Pj4gICAgIAlyZXR1 cm4gTlVMTDsKPj4+Pj4gICAgIH0KPj4+Pj4gK3N0YXRpYyBzdHJ1Y3QgdmlydGlvX3Zzb2NrX2J1 ZiAqCj4+Pj4+ICt2aXJ0aW9fdHJhbnNwb3J0X2FsbG9jX2J1ZihzdHJ1Y3QgdmlydGlvX3Zzb2Nr X3BrdCAqcGt0LCBib29sIHplcm9fY29weSkKPj4+Pj4gK3sKPj4+Pj4gKwlzdHJ1Y3QgdmlydGlv X3Zzb2NrX2J1ZiAqYnVmOwo+Pj4+PiArCj4+Pj4+ICsJaWYgKHBrdC0+bGVuID09IDApCj4+Pj4+ ICsJCXJldHVybiBOVUxMOwo+Pj4+PiArCj4+Pj4+ICsJYnVmID0ga3phbGxvYyhzaXplb2YoKmJ1 ZiksIEdGUF9LRVJORUwpOwo+Pj4+PiArCWlmICghYnVmKQo+Pj4+PiArCQlyZXR1cm4gTlVMTDsK Pj4+Pj4gKwo+Pj4+PiArCS8qIElmIHRoZSBidWZmZXIgaW4gdGhlIHZpcnRpb192c29ja19wa3Qg aXMgZnVsbCwgd2UgY2FuIG1vdmUgaXQgdG8KPj4+Pj4gKwkgKiB0aGUgbmV3IHZpcnRpb192c29j a19idWYgYXZvaWRpbmcgdGhlIGNvcHksIGJlY2F1c2Ugd2UgYXJlIHN1cmUgdGhhdAo+Pj4+PiAr CSAqIHdlIGFyZSBub3QgdXNlIG1vcmUgbWVtb3J5IHRoYW4gdGhhdCBjb3VudGVkIGJ5IHRoZSBj cmVkaXQgbWVjaGFuaXNtLgo+Pj4+PiArCSAqLwo+Pj4+PiArCWlmICh6ZXJvX2NvcHkgJiYgcGt0 LT5sZW4gPT0gcGt0LT5idWZfbGVuKSB7Cj4+Pj4+ICsJCWJ1Zi0+YWRkciA9IHBrdC0+YnVmOwo+ Pj4+PiArCQlwa3QtPmJ1ZiA9IE5VTEw7Cj4+Pj4+ICsJfSBlbHNlIHsKPj4+PiBJcyB0aGUgY29w eSBzdGlsbCBuZWVkZWQgaWYgd2UncmUganVzdCBmZXcgYnl0ZXMgbGVzcz8gV2UgbWVldCBzaW1p bGFyIGlzc3VlCj4+Pj4gZm9yIHZpcml0by1uZXQsIGFuZCB2aXJ0aW8tbmV0IHNvbHZlIHRoaXMg YnkgYWx3YXlzIGNvcHkgZmlyc3QgMTI4Ynl0ZXMgZm9yCj4+Pj4gYmlnIHBhY2tldHMuCj4+Pj4K Pj4+PiBTZWUgcmVjZWl2ZV9iaWcoKQo+Pj4gSSdtIHNlZWluZywgSXQgaXMgbW9yZSBzb3BoaXN0 aWNhdGVkLgo+Pj4gSUlVQywgdmlydGlvLW5ldCBhbGxvY2F0ZXMgYSBza19idWZmIHdpdGggMTI4 IGJ5dGVzIG9mIGJ1ZmZlciwgdGhlbiBjb3BpZXMgdGhlCj4+PiBmaXJzdCAxMjggYnl0ZXMsIHRo ZW4gYWRkcyB0aGUgYnVmZmVyIHVzZWQgdG8gcmVjZWl2ZSB0aGUgcGFja2V0IGFzIGEgZnJhZyB0 bwo+Pj4gdGhlIHNrYi4KPj4KPj4gWWVzIGFuZCB0aGUgcG9pbnQgaXMgaWYgdGhlIHBhY2tldCBp cyBzbWFsbGVyIHRoYW4gMTI4IGJ5dGVzIHRoZSBwYWdlcyB3aWxsCj4+IGJlIHJlY3ljbGVkLgo+ Pgo+Pgo+IFNvIGl0J3MgYXZvaWQgdGhlIG92ZXJoZWFkIG9mIGFsbG9jYXRpb24gb2YgYSBsYXJn ZSBidWZmZXIuIEkgZ290IGl0Lgo+Cj4gSnVzdCBhIGN1cmlvc2l0eSwgd2h5IHRoZSB0aHJlc2hv bGQgaXMgMTI4IGJ5dGVzPwoKCiBGcm9tIGl0cyBuYW1lIChHT09EX0NPUFlfTEVOKSwgSSB0aGlu ayBpdCBqdXN0IGEgdmFsdWUgdGhhdCB3b24ndCBsb3NlIAptdWNoIHBlcmZvcm1hbmNlLCBlLmcg dGhlIHNpemUgdHdvIGNhY2hlbGluZXMuCgpUaGFua3MKCgo+Cj4+PiBEbyB5b3Ugc3VnZ2VzdCB0 byBpbXBsZW1lbnQgc29tZXRoaW5nIHNpbWlsYXIsIG9yIGZvciBub3cgd2UgY2FuIHVzZSBteQo+ Pj4gYXBwcm9hY2ggYW5kIGlmIHdlIHdpbGwgbWVyZ2UgdGhlIGRhdGFwYXRoIHdlIGNhbiByZXVz ZSB0aGUgdmlydGlvLW5ldAo+Pj4gYXBwcm9hY2g/Cj4+Cj4+IEkgdGhpbmsgd2UgbmVlZCBhIGJl dHRlciB0aHJlc2hvbGQuIElmIEkgdW5kZXJzdGFuZCB0aGUgcGF0Y2ggY29ycmVjdGx5LCB3ZQo+ PiB3aWxsIGRvIGNvcHkgdW5sZXNzIHRoZSBwYWNrZXQgaXMgNjRLIHdoZW4gZ3Vlc3QgaXMgZG9p bmcgcmVjZWl2aW5nLiAxIGJ5dGUKPj4gcGFja2V0IGlzIGluZGVlZCBhIHByb2JsZW0sIGJ1dCB3 ZSBuZWVkIHRvIHNvbHZlIGl0IHdpdGhvdXQgbG9zaW5nIHRvbyBtdWNoCj4+IHBlcmZvcm1hbmNl Lgo+IEl0IGlzIGNvcnJlY3QuIEknbGwgdHJ5IHRvIGZpZ3VyZSBvdXQgYSBiZXR0ZXIgdGhyZXNo b2xkIGFuZCB0aGUgdXNhZ2Ugb2YKPiBvcmRlciAwIHBhZ2UuCj4KPiBUaGFua3MgYWdhaW4gZm9y IHlvdXIgYWR2aWNlcywKPiBTdGVmYW5vCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fClZpcnR1YWxpemF0aW9uIG1haWxpbmcgbGlzdApWaXJ0dWFsaXphdGlv bkBsaXN0cy5saW51eC1mb3VuZGF0aW9uLm9yZwpodHRwczovL2xpc3RzLmxpbnV4Zm91bmRhdGlv bi5vcmcvbWFpbG1hbi9saXN0aW5mby92aXJ0dWFsaXphdGlvbg==