From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.3 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4BF28C4361B for ; Thu, 17 Dec 2020 09:04:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 099922388E for ; Thu, 17 Dec 2020 09:04:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727287AbgLQJEl (ORCPT ); Thu, 17 Dec 2020 04:04:41 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:25853 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726416AbgLQJEg (ORCPT ); Thu, 17 Dec 2020 04:04:36 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1608195789; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=N7BMLAt3Nkb4GWZAxfBiiUdsoewOzT4BGc7hVKvcSzc=; b=KTWMHWqkz1LIHEcC3MTPK3RuUhRKfZ7iYPM6eTPOCc0Nkomw58TLFrV6KPUo1Dgu0bFoeb 2uUH7n2z/99Zhxz1RwPTh1XdiuTwcjE2GGV4x0M9NhKpHBoiZ/m8u6LY9eZoiShfNsqkuj CUmCOgAnnbQCJg4Oxkl2V4gf5E7WRcI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-55-6rdIExmXMbSqGRSDQB8yNA-1; Thu, 17 Dec 2020 04:03:05 -0500 X-MC-Unique: 6rdIExmXMbSqGRSDQB8yNA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 81710190A7A4; Thu, 17 Dec 2020 09:03:03 +0000 (UTC) Received: from [10.72.12.223] (ovpn-12-223.pek2.redhat.com [10.72.12.223]) by smtp.corp.redhat.com (Postfix) with ESMTP id A76125D9C0; Thu, 17 Dec 2020 09:02:50 +0000 (UTC) Subject: Re: [PATCH 00/21] Control VQ support in vDPA To: "Michael S. Tsirkin" Cc: eperezma@redhat.com, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, lulu@redhat.com, eli@mellanox.com, lingshan.zhu@intel.com, rob.miller@broadcom.com, stefanha@redhat.com, sgarzare@redhat.com References: <20201216064818.48239-1-jasowang@redhat.com> <20201216044051-mutt-send-email-mst@kernel.org> <20201217025410-mutt-send-email-mst@kernel.org> From: Jason Wang Message-ID: <61b60985-142b-10f2-58b8-1d9f57c0cfca@redhat.com> Date: Thu, 17 Dec 2020 17:02:49 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <20201217025410-mutt-send-email-mst@kernel.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2020/12/17 下午3:58, Michael S. Tsirkin wrote: > On Thu, Dec 17, 2020 at 11:30:18AM +0800, Jason Wang wrote: >> On 2020/12/16 下午5:47, Michael S. Tsirkin wrote: >>> On Wed, Dec 16, 2020 at 02:47:57PM +0800, Jason Wang wrote: >>>> Hi All: >>>> >>>> This series tries to add the support for control virtqueue in vDPA. >>>> >>>> Control virtqueue is used by networking device for accepting various >>>> commands from the driver. It's a must to support multiqueue and other >>>> configurations. >>>> >>>> When used by vhost-vDPA bus driver for VM, the control virtqueue >>>> should be shadowed via userspace VMM (Qemu) instead of being assigned >>>> directly to Guest. This is because Qemu needs to know the device state >>>> in order to start and stop device correctly (e.g for Live Migration). >>>> >>>> This requies to isolate the memory mapping for control virtqueue >>>> presented by vhost-vDPA to prevent guest from accesing it directly. >>>> To achieve this, vDPA introduce two new abstractions: >>>> >>>> - address space: identified through address space id (ASID) and a set >>>> of memory mapping in maintained >>>> - virtqueue group: the minimal set of virtqueues that must share an >>>> address space >>> How will this support the pretty common case where control vq >>> is programmed by the kernel through the PF, and others by the VFs? >> >> In this case, the VF parent need to provide a software control vq and decode >> the command then send them to VF. > > But how does that tie to the address space infrastructure? In this case, address space is not a must. But the idea is to make control vq works for all types of hardware: 1) control virtqueue is implemented via VF/PF communication 2) control virtqueue is implemented by VF but not through DMA 3) control virtqueue is implemented by VF DMA, it could be either a hardware control virtqueue or other type of DMA The address space is a must for 3) to work and can work for both 1) and 2). > > > >>> >>> I actually thought the way to support it is by exposing >>> something like an "inject buffers" API which sends data to a given VQ. >>> Maybe an ioctl, and maybe down the road uio ring can support batching >>> these .... >> >> So the virtuqueue allows the request to be processed asynchronously (e.g >> driver may choose to use interrupt for control vq). This means we need to >> support that in uAPI level. > I don't think we need to make it async, just a regular ioctl will do. > In fact no guest uses the asynchronous property. It was not forbidden by the spec then we need to support that. E.g we can not assume driver doesn't assign interrupt for cvq. > > >> And if we manage to do that, it's just another >> type of virtqueue. >> >> For virtio-vDPA, this also means the extensions for queue processing which >> is a functional duplication. > I don't see why, just send it to the actual control vq :) But in the case you've pointed out, there's no hardware control vq in fact. > >> Using what proposed in this series, we don't >> need any changes for kernel virtio drivers. >> >> What's more important, this series could be used for future features that >> requires DMA isolation between virtqueues: >> >> - report dirty pages via virtqueue >> - sub function level device slicing > > I agree these are nice to have, but I am not sure basic control vq must > be tied to that. If the control virtqueue is implemented via DMA through VF, it looks like a must. Thanks > >> ... >> >> Thanks >> >> >>> >>>> Device needs to advertise the following attributes to vDPA: >>>> >>>> - the number of address spaces supported in the device >>>> - the number of virtqueue groups supported in the device >>>> - the mappings from a specific virtqueue to its virtqueue groups >>>> >>>> The mappings from virtqueue to virtqueue groups is fixed and defined >>>> by vDPA device driver. E.g: >>>> >>>> - For the device that has hardware ASID support, it can simply >>>> advertise a per virtqueue virtqueue group. >>>> - For the device that does not have hardware ASID support, it can >>>> simply advertise a single virtqueue group that contains all >>>> virtqueues. Or if it wants a software emulated control virtqueue, it >>>> can advertise two virtqueue groups, one is for cvq, another is for >>>> the rest virtqueues. >>>> >>>> vDPA also allow to change the association between virtqueue group and >>>> address space. So in the case of control virtqueue, userspace >>>> VMM(Qemu) may use a dedicated address space for the control virtqueue >>>> group to isolate the memory mapping. >>>> >>>> The vhost/vhost-vDPA is also extend for the userspace to: >>>> >>>> - query the number of virtqueue groups and address spaces supported by >>>> the device >>>> - query the virtqueue group for a specific virtqueue >>>> - assocaite a virtqueue group with an address space >>>> - send ASID based IOTLB commands >>>> >>>> This will help userspace VMM(Qemu) to detect whether the control vq >>>> could be supported and isolate memory mappings of control virtqueue >>>> from the others. >>>> >>>> To demonstrate the usage, vDPA simulator is extended to support >>>> setting MAC address via a emulated control virtqueue. >>>> >>>> Please review. >>>> >>>> Changes since RFC: >>>> >>>> - tweak vhost uAPI documentation >>>> - switch to use device specific IOTLB really in patch 4 >>>> - tweak the commit log >>>> - fix that ASID in vhost is claimed to be 32 actually but 16bit >>>> actually >>>> - fix use after free when using ASID with IOTLB batching requests >>>> - switch to use Stefano's patch for having separated iov >>>> - remove unused "used_as" variable >>>> - fix the iotlb/asid checking in vhost_vdpa_unmap() >>>> >>>> Thanks >>>> >>>> Jason Wang (20): >>>> vhost: move the backend feature bits to vhost_types.h >>>> virtio-vdpa: don't set callback if virtio doesn't need it >>>> vhost-vdpa: passing iotlb to IOMMU mapping helpers >>>> vhost-vdpa: switch to use vhost-vdpa specific IOTLB >>>> vdpa: add the missing comment for nvqs in struct vdpa_device >>>> vdpa: introduce virtqueue groups >>>> vdpa: multiple address spaces support >>>> vdpa: introduce config operations for associating ASID to a virtqueue >>>> group >>>> vhost_iotlb: split out IOTLB initialization >>>> vhost: support ASID in IOTLB API >>>> vhost-vdpa: introduce asid based IOTLB >>>> vhost-vdpa: introduce uAPI to get the number of virtqueue groups >>>> vhost-vdpa: introduce uAPI to get the number of address spaces >>>> vhost-vdpa: uAPI to get virtqueue group id >>>> vhost-vdpa: introduce uAPI to set group ASID >>>> vhost-vdpa: support ASID based IOTLB API >>>> vdpa_sim: advertise VIRTIO_NET_F_MTU >>>> vdpa_sim: factor out buffer completion logic >>>> vdpa_sim: filter destination mac address >>>> vdpasim: control virtqueue support >>>> >>>> Stefano Garzarella (1): >>>> vdpa_sim: split vdpasim_virtqueue's iov field in out_iov and in_iov >>>> >>>> drivers/vdpa/ifcvf/ifcvf_main.c | 9 +- >>>> drivers/vdpa/mlx5/net/mlx5_vnet.c | 11 +- >>>> drivers/vdpa/vdpa.c | 8 +- >>>> drivers/vdpa/vdpa_sim/vdpa_sim.c | 292 ++++++++++++++++++++++++------ >>>> drivers/vhost/iotlb.c | 23 ++- >>>> drivers/vhost/vdpa.c | 246 ++++++++++++++++++++----- >>>> drivers/vhost/vhost.c | 23 ++- >>>> drivers/vhost/vhost.h | 4 +- >>>> drivers/virtio/virtio_vdpa.c | 2 +- >>>> include/linux/vdpa.h | 42 ++++- >>>> include/linux/vhost_iotlb.h | 2 + >>>> include/uapi/linux/vhost.h | 25 ++- >>>> include/uapi/linux/vhost_types.h | 10 +- >>>> 13 files changed, 561 insertions(+), 136 deletions(-) >>>> >>>> -- >>>> 2.25.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.1 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F10ABC4361B for ; Thu, 17 Dec 2020 09:03:12 +0000 (UTC) Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7C0372389E for ; Thu, 17 Dec 2020 09:03:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7C0372389E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=virtualization-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id DFACA87877; Thu, 17 Dec 2020 09:03:11 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JS8mnBTY4Bcw; Thu, 17 Dec 2020 09:03:11 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by whitealder.osuosl.org (Postfix) with ESMTP id F2D0A8785B; Thu, 17 Dec 2020 09:03:10 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id DCC4FC1833; Thu, 17 Dec 2020 09:03:10 +0000 (UTC) Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id BA7F8C013B for ; Thu, 17 Dec 2020 09:03:09 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id A4E8387555 for ; Thu, 17 Dec 2020 09:03:09 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3kAwrExz-T5h for ; Thu, 17 Dec 2020 09:03:08 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 91C8C87554 for ; Thu, 17 Dec 2020 09:03:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1608195787; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=N7BMLAt3Nkb4GWZAxfBiiUdsoewOzT4BGc7hVKvcSzc=; b=DQldMh+Hs8uJuUhnTtjwGK5c/K0Wrfm+/Mwbe6V1QOSLBiy7hHEI3/+W9hMXElQZVwi3AN 3S/h1gLW1/Lz8/0LBqvFSeF53tuCQQ7hdRPDIUzNpWAU7bq0vbelbTn3rOzZylI+ebJCQ6 f2YuYfM3i7rbtL4O5hrNUWtQikJcYJM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-55-6rdIExmXMbSqGRSDQB8yNA-1; Thu, 17 Dec 2020 04:03:05 -0500 X-MC-Unique: 6rdIExmXMbSqGRSDQB8yNA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 81710190A7A4; Thu, 17 Dec 2020 09:03:03 +0000 (UTC) Received: from [10.72.12.223] (ovpn-12-223.pek2.redhat.com [10.72.12.223]) by smtp.corp.redhat.com (Postfix) with ESMTP id A76125D9C0; Thu, 17 Dec 2020 09:02:50 +0000 (UTC) Subject: Re: [PATCH 00/21] Control VQ support in vDPA To: "Michael S. Tsirkin" References: <20201216064818.48239-1-jasowang@redhat.com> <20201216044051-mutt-send-email-mst@kernel.org> <20201217025410-mutt-send-email-mst@kernel.org> From: Jason Wang Message-ID: <61b60985-142b-10f2-58b8-1d9f57c0cfca@redhat.com> Date: Thu, 17 Dec 2020 17:02:49 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <20201217025410-mutt-send-email-mst@kernel.org> Content-Language: en-US X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Cc: kvm@vger.kernel.org, lulu@redhat.com, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, eperezma@redhat.com, stefanha@redhat.com, eli@mellanox.com, lingshan.zhu@intel.com, rob.miller@broadcom.com X-BeenThere: virtualization@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Linux virtualization List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Errors-To: virtualization-bounces@lists.linux-foundation.org Sender: "Virtualization" Ck9uIDIwMjAvMTIvMTcg5LiL5Y2IMzo1OCwgTWljaGFlbCBTLiBUc2lya2luIHdyb3RlOgo+IE9u IFRodSwgRGVjIDE3LCAyMDIwIGF0IDExOjMwOjE4QU0gKzA4MDAsIEphc29uIFdhbmcgd3JvdGU6 Cj4+IE9uIDIwMjAvMTIvMTYg5LiL5Y2INTo0NywgTWljaGFlbCBTLiBUc2lya2luIHdyb3RlOgo+ Pj4gT24gV2VkLCBEZWMgMTYsIDIwMjAgYXQgMDI6NDc6NTdQTSArMDgwMCwgSmFzb24gV2FuZyB3 cm90ZToKPj4+PiBIaSBBbGw6Cj4+Pj4KPj4+PiBUaGlzIHNlcmllcyB0cmllcyB0byBhZGQgdGhl IHN1cHBvcnQgZm9yIGNvbnRyb2wgdmlydHF1ZXVlIGluIHZEUEEuCj4+Pj4KPj4+PiBDb250cm9s IHZpcnRxdWV1ZSBpcyB1c2VkIGJ5IG5ldHdvcmtpbmcgZGV2aWNlIGZvciBhY2NlcHRpbmcgdmFy aW91cwo+Pj4+IGNvbW1hbmRzIGZyb20gdGhlIGRyaXZlci4gSXQncyBhIG11c3QgdG8gc3VwcG9y dCBtdWx0aXF1ZXVlIGFuZCBvdGhlcgo+Pj4+IGNvbmZpZ3VyYXRpb25zLgo+Pj4+Cj4+Pj4gV2hl biB1c2VkIGJ5IHZob3N0LXZEUEEgYnVzIGRyaXZlciBmb3IgVk0sIHRoZSBjb250cm9sIHZpcnRx dWV1ZQo+Pj4+IHNob3VsZCBiZSBzaGFkb3dlZCB2aWEgdXNlcnNwYWNlIFZNTSAoUWVtdSkgaW5z dGVhZCBvZiBiZWluZyBhc3NpZ25lZAo+Pj4+IGRpcmVjdGx5IHRvIEd1ZXN0LiBUaGlzIGlzIGJl Y2F1c2UgUWVtdSBuZWVkcyB0byBrbm93IHRoZSBkZXZpY2Ugc3RhdGUKPj4+PiBpbiBvcmRlciB0 byBzdGFydCBhbmQgc3RvcCBkZXZpY2UgY29ycmVjdGx5IChlLmcgZm9yIExpdmUgTWlncmF0aW9u KS4KPj4+Pgo+Pj4+IFRoaXMgcmVxdWllcyB0byBpc29sYXRlIHRoZSBtZW1vcnkgbWFwcGluZyBm b3IgY29udHJvbCB2aXJ0cXVldWUKPj4+PiBwcmVzZW50ZWQgYnkgdmhvc3QtdkRQQSB0byBwcmV2 ZW50IGd1ZXN0IGZyb20gYWNjZXNpbmcgaXQgZGlyZWN0bHkuCj4+Pj4gVG8gYWNoaWV2ZSB0aGlz LCB2RFBBIGludHJvZHVjZSB0d28gbmV3IGFic3RyYWN0aW9uczoKPj4+Pgo+Pj4+IC0gYWRkcmVz cyBzcGFjZTogaWRlbnRpZmllZCB0aHJvdWdoIGFkZHJlc3Mgc3BhY2UgaWQgKEFTSUQpIGFuZCBh IHNldAo+Pj4+ICAgICAgICAgICAgICAgICAgICBvZiBtZW1vcnkgbWFwcGluZyBpbiBtYWludGFp bmVkCj4+Pj4gLSB2aXJ0cXVldWUgZ3JvdXA6IHRoZSBtaW5pbWFsIHNldCBvZiB2aXJ0cXVldWVz IHRoYXQgbXVzdCBzaGFyZSBhbgo+Pj4+ICAgICAgICAgICAgICAgICAgICBhZGRyZXNzIHNwYWNl Cj4+PiBIb3cgd2lsbCB0aGlzIHN1cHBvcnQgdGhlIHByZXR0eSBjb21tb24gY2FzZSB3aGVyZSBj b250cm9sIHZxCj4+PiBpcyBwcm9ncmFtbWVkIGJ5IHRoZSBrZXJuZWwgdGhyb3VnaCB0aGUgUEYs IGFuZCBvdGhlcnMgYnkgdGhlIFZGcz8KPj4KPj4gSW4gdGhpcyBjYXNlLCB0aGUgVkYgcGFyZW50 IG5lZWQgdG8gcHJvdmlkZSBhIHNvZnR3YXJlIGNvbnRyb2wgdnEgYW5kIGRlY29kZQo+PiB0aGUg Y29tbWFuZCB0aGVuIHNlbmQgdGhlbSB0byBWRi4KPgo+IEJ1dCBob3cgZG9lcyB0aGF0IHRpZSB0 byB0aGUgYWRkcmVzcyBzcGFjZSBpbmZyYXN0cnVjdHVyZT8KCgpJbiB0aGlzIGNhc2UsIGFkZHJl c3Mgc3BhY2UgaXMgbm90IGEgbXVzdC4gQnV0IHRoZSBpZGVhIGlzIHRvIG1ha2UgCmNvbnRyb2wg dnEgd29ya3MgZm9yIGFsbCB0eXBlcyBvZiBoYXJkd2FyZToKCjEpIGNvbnRyb2wgdmlydHF1ZXVl IGlzIGltcGxlbWVudGVkIHZpYSBWRi9QRiBjb21tdW5pY2F0aW9uCjIpIGNvbnRyb2wgdmlydHF1 ZXVlIGlzIGltcGxlbWVudGVkIGJ5IFZGIGJ1dCBub3QgdGhyb3VnaCBETUEKMykgY29udHJvbCB2 aXJ0cXVldWUgaXMgaW1wbGVtZW50ZWQgYnkgVkYgRE1BLCBpdCBjb3VsZCBiZSBlaXRoZXIgYSAK aGFyZHdhcmUgY29udHJvbCB2aXJ0cXVldWUgb3Igb3RoZXIgdHlwZSBvZiBETUEKClRoZSBhZGRy ZXNzIHNwYWNlIGlzIGEgbXVzdCBmb3IgMykgdG8gd29yayBhbmQgY2FuIHdvcmsgZm9yIGJvdGgg MSkgYW5kIDIpLgoKCj4KPgo+Cj4+Pgo+Pj4gSSBhY3R1YWxseSB0aG91Z2h0IHRoZSB3YXkgdG8g c3VwcG9ydCBpdCBpcyBieSBleHBvc2luZwo+Pj4gc29tZXRoaW5nIGxpa2UgYW4gImluamVjdCBi dWZmZXJzIiBBUEkgd2hpY2ggc2VuZHMgZGF0YSB0byBhIGdpdmVuIFZRLgo+Pj4gTWF5YmUgYW4g aW9jdGwsIGFuZCBtYXliZSBkb3duIHRoZSByb2FkIHVpbyByaW5nIGNhbiBzdXBwb3J0IGJhdGNo aW5nCj4+PiB0aGVzZSAuLi4uCj4+Cj4+IFNvIHRoZSB2aXJ0dXF1ZXVlIGFsbG93cyB0aGUgcmVx dWVzdCB0byBiZSBwcm9jZXNzZWQgYXN5bmNocm9ub3VzbHkgKGUuZwo+PiBkcml2ZXIgbWF5IGNo b29zZSB0byB1c2UgaW50ZXJydXB0IGZvciBjb250cm9sIHZxKS4gVGhpcyBtZWFucyB3ZSBuZWVk IHRvCj4+IHN1cHBvcnQgdGhhdCBpbiB1QVBJIGxldmVsLgo+IEkgZG9uJ3QgdGhpbmsgd2UgbmVl ZCB0byBtYWtlIGl0IGFzeW5jLCBqdXN0IGEgcmVndWxhciBpb2N0bCB3aWxsIGRvLgo+IEluIGZh Y3Qgbm8gZ3Vlc3QgdXNlcyB0aGUgYXN5bmNocm9ub3VzIHByb3BlcnR5LgoKCkl0IHdhcyBub3Qg Zm9yYmlkZGVuIGJ5IHRoZSBzcGVjIHRoZW4gd2UgbmVlZCB0byBzdXBwb3J0IHRoYXQuIEUuZyB3 ZSAKY2FuIG5vdCBhc3N1bWUgZHJpdmVyIGRvZXNuJ3QgYXNzaWduIGludGVycnVwdCBmb3IgY3Zx LgoKCj4KPgo+PiBBbmQgaWYgd2UgbWFuYWdlIHRvIGRvIHRoYXQsIGl0J3MganVzdCBhbm90aGVy Cj4+IHR5cGUgb2YgdmlydHF1ZXVlLgo+Pgo+PiBGb3IgdmlydGlvLXZEUEEsIHRoaXMgYWxzbyBt ZWFucyB0aGUgZXh0ZW5zaW9ucyBmb3IgcXVldWUgcHJvY2Vzc2luZyB3aGljaAo+PiBpcyBhIGZ1 bmN0aW9uYWwgZHVwbGljYXRpb24uCj4gSSBkb24ndCBzZWUgd2h5LCBqdXN0IHNlbmQgaXQgdG8g dGhlIGFjdHVhbCBjb250cm9sIHZxIDopCgoKQnV0IGluIHRoZSBjYXNlIHlvdSd2ZSBwb2ludGVk IG91dCwgdGhlcmUncyBubyBoYXJkd2FyZSBjb250cm9sIHZxIGluIGZhY3QuCgoKPgo+PiBVc2lu ZyB3aGF0IHByb3Bvc2VkIGluIHRoaXMgc2VyaWVzLCB3ZSBkb24ndAo+PiBuZWVkIGFueSBjaGFu Z2VzIGZvciBrZXJuZWwgdmlydGlvIGRyaXZlcnMuCj4+Cj4+IFdoYXQncyBtb3JlIGltcG9ydGFu dCwgdGhpcyBzZXJpZXMgY291bGQgYmUgdXNlZCBmb3IgZnV0dXJlIGZlYXR1cmVzIHRoYXQKPj4g cmVxdWlyZXMgRE1BIGlzb2xhdGlvbiBiZXR3ZWVuIHZpcnRxdWV1ZXM6Cj4+Cj4+IC0gcmVwb3J0 IGRpcnR5IHBhZ2VzIHZpYSB2aXJ0cXVldWUKPj4gLSBzdWIgZnVuY3Rpb24gbGV2ZWwgZGV2aWNl IHNsaWNpbmcKPgo+IEkgYWdyZWUgdGhlc2UgYXJlIG5pY2UgdG8gaGF2ZSwgYnV0IEkgYW0gbm90 IHN1cmUgYmFzaWMgY29udHJvbCB2cSBtdXN0Cj4gYmUgdGllZCB0byB0aGF0LgoKCklmIHRoZSBj b250cm9sIHZpcnRxdWV1ZSBpcyBpbXBsZW1lbnRlZCB2aWEgRE1BIHRocm91Z2ggVkYsIGl0IGxv b2tzIApsaWtlIGEgbXVzdC4KClRoYW5rcwoKCj4KPj4gLi4uCj4+Cj4+IFRoYW5rcwo+Pgo+Pgo+ Pj4KPj4+PiBEZXZpY2UgbmVlZHMgdG8gYWR2ZXJ0aXNlIHRoZSBmb2xsb3dpbmcgYXR0cmlidXRl cyB0byB2RFBBOgo+Pj4+Cj4+Pj4gLSB0aGUgbnVtYmVyIG9mIGFkZHJlc3Mgc3BhY2VzIHN1cHBv cnRlZCBpbiB0aGUgZGV2aWNlCj4+Pj4gLSB0aGUgbnVtYmVyIG9mIHZpcnRxdWV1ZSBncm91cHMg c3VwcG9ydGVkIGluIHRoZSBkZXZpY2UKPj4+PiAtIHRoZSBtYXBwaW5ncyBmcm9tIGEgc3BlY2lm aWMgdmlydHF1ZXVlIHRvIGl0cyB2aXJ0cXVldWUgZ3JvdXBzCj4+Pj4KPj4+PiBUaGUgbWFwcGlu Z3MgZnJvbSB2aXJ0cXVldWUgdG8gdmlydHF1ZXVlIGdyb3VwcyBpcyBmaXhlZCBhbmQgZGVmaW5l ZAo+Pj4+IGJ5IHZEUEEgZGV2aWNlIGRyaXZlci4gRS5nOgo+Pj4+Cj4+Pj4gLSBGb3IgdGhlIGRl dmljZSB0aGF0IGhhcyBoYXJkd2FyZSBBU0lEIHN1cHBvcnQsIGl0IGNhbiBzaW1wbHkKPj4+PiAg ICAgYWR2ZXJ0aXNlIGEgcGVyIHZpcnRxdWV1ZSB2aXJ0cXVldWUgZ3JvdXAuCj4+Pj4gLSBGb3Ig dGhlIGRldmljZSB0aGF0IGRvZXMgbm90IGhhdmUgaGFyZHdhcmUgQVNJRCBzdXBwb3J0LCBpdCBj YW4KPj4+PiAgICAgc2ltcGx5IGFkdmVydGlzZSBhIHNpbmdsZSB2aXJ0cXVldWUgZ3JvdXAgdGhh dCBjb250YWlucyBhbGwKPj4+PiAgICAgdmlydHF1ZXVlcy4gT3IgaWYgaXQgd2FudHMgYSBzb2Z0 d2FyZSBlbXVsYXRlZCBjb250cm9sIHZpcnRxdWV1ZSwgaXQKPj4+PiAgICAgY2FuIGFkdmVydGlz ZSB0d28gdmlydHF1ZXVlIGdyb3Vwcywgb25lIGlzIGZvciBjdnEsIGFub3RoZXIgaXMgZm9yCj4+ Pj4gICAgIHRoZSByZXN0IHZpcnRxdWV1ZXMuCj4+Pj4KPj4+PiB2RFBBIGFsc28gYWxsb3cgdG8g Y2hhbmdlIHRoZSBhc3NvY2lhdGlvbiBiZXR3ZWVuIHZpcnRxdWV1ZSBncm91cCBhbmQKPj4+PiBh ZGRyZXNzIHNwYWNlLiBTbyBpbiB0aGUgY2FzZSBvZiBjb250cm9sIHZpcnRxdWV1ZSwgdXNlcnNw YWNlCj4+Pj4gVk1NKFFlbXUpIG1heSB1c2UgYSBkZWRpY2F0ZWQgYWRkcmVzcyBzcGFjZSBmb3Ig dGhlIGNvbnRyb2wgdmlydHF1ZXVlCj4+Pj4gZ3JvdXAgdG8gaXNvbGF0ZSB0aGUgbWVtb3J5IG1h cHBpbmcuCj4+Pj4KPj4+PiBUaGUgdmhvc3Qvdmhvc3QtdkRQQSBpcyBhbHNvIGV4dGVuZCBmb3Ig dGhlIHVzZXJzcGFjZSB0bzoKPj4+Pgo+Pj4+IC0gcXVlcnkgdGhlIG51bWJlciBvZiB2aXJ0cXVl dWUgZ3JvdXBzIGFuZCBhZGRyZXNzIHNwYWNlcyBzdXBwb3J0ZWQgYnkKPj4+PiAgICAgdGhlIGRl dmljZQo+Pj4+IC0gcXVlcnkgdGhlIHZpcnRxdWV1ZSBncm91cCBmb3IgYSBzcGVjaWZpYyB2aXJ0 cXVldWUKPj4+PiAtIGFzc29jYWl0ZSBhIHZpcnRxdWV1ZSBncm91cCB3aXRoIGFuIGFkZHJlc3Mg c3BhY2UKPj4+PiAtIHNlbmQgQVNJRCBiYXNlZCBJT1RMQiBjb21tYW5kcwo+Pj4+Cj4+Pj4gVGhp cyB3aWxsIGhlbHAgdXNlcnNwYWNlIFZNTShRZW11KSB0byBkZXRlY3Qgd2hldGhlciB0aGUgY29u dHJvbCB2cQo+Pj4+IGNvdWxkIGJlIHN1cHBvcnRlZCBhbmQgaXNvbGF0ZSBtZW1vcnkgbWFwcGlu Z3Mgb2YgY29udHJvbCB2aXJ0cXVldWUKPj4+PiBmcm9tIHRoZSBvdGhlcnMuCj4+Pj4KPj4+PiBU byBkZW1vbnN0cmF0ZSB0aGUgdXNhZ2UsIHZEUEEgc2ltdWxhdG9yIGlzIGV4dGVuZGVkIHRvIHN1 cHBvcnQKPj4+PiBzZXR0aW5nIE1BQyBhZGRyZXNzIHZpYSBhIGVtdWxhdGVkIGNvbnRyb2wgdmly dHF1ZXVlLgo+Pj4+Cj4+Pj4gUGxlYXNlIHJldmlldy4KPj4+Pgo+Pj4+IENoYW5nZXMgc2luY2Ug UkZDOgo+Pj4+Cj4+Pj4gLSB0d2VhayB2aG9zdCB1QVBJIGRvY3VtZW50YXRpb24KPj4+PiAtIHN3 aXRjaCB0byB1c2UgZGV2aWNlIHNwZWNpZmljIElPVExCIHJlYWxseSBpbiBwYXRjaCA0Cj4+Pj4g LSB0d2VhayB0aGUgY29tbWl0IGxvZwo+Pj4+IC0gZml4IHRoYXQgQVNJRCBpbiB2aG9zdCBpcyBj bGFpbWVkIHRvIGJlIDMyIGFjdHVhbGx5IGJ1dCAxNmJpdAo+Pj4+ICAgICBhY3R1YWxseQo+Pj4+ IC0gZml4IHVzZSBhZnRlciBmcmVlIHdoZW4gdXNpbmcgQVNJRCB3aXRoIElPVExCIGJhdGNoaW5n IHJlcXVlc3RzCj4+Pj4gLSBzd2l0Y2ggdG8gdXNlIFN0ZWZhbm8ncyBwYXRjaCBmb3IgaGF2aW5n IHNlcGFyYXRlZCBpb3YKPj4+PiAtIHJlbW92ZSB1bnVzZWQgInVzZWRfYXMiIHZhcmlhYmxlCj4+ Pj4gLSBmaXggdGhlIGlvdGxiL2FzaWQgY2hlY2tpbmcgaW4gdmhvc3RfdmRwYV91bm1hcCgpCj4+ Pj4KPj4+PiBUaGFua3MKPj4+Pgo+Pj4+IEphc29uIFdhbmcgKDIwKToKPj4+PiAgICAgdmhvc3Q6 IG1vdmUgdGhlIGJhY2tlbmQgZmVhdHVyZSBiaXRzIHRvIHZob3N0X3R5cGVzLmgKPj4+PiAgICAg dmlydGlvLXZkcGE6IGRvbid0IHNldCBjYWxsYmFjayBpZiB2aXJ0aW8gZG9lc24ndCBuZWVkIGl0 Cj4+Pj4gICAgIHZob3N0LXZkcGE6IHBhc3NpbmcgaW90bGIgdG8gSU9NTVUgbWFwcGluZyBoZWxw ZXJzCj4+Pj4gICAgIHZob3N0LXZkcGE6IHN3aXRjaCB0byB1c2Ugdmhvc3QtdmRwYSBzcGVjaWZp YyBJT1RMQgo+Pj4+ICAgICB2ZHBhOiBhZGQgdGhlIG1pc3NpbmcgY29tbWVudCBmb3IgbnZxcyBp biBzdHJ1Y3QgdmRwYV9kZXZpY2UKPj4+PiAgICAgdmRwYTogaW50cm9kdWNlIHZpcnRxdWV1ZSBn cm91cHMKPj4+PiAgICAgdmRwYTogbXVsdGlwbGUgYWRkcmVzcyBzcGFjZXMgc3VwcG9ydAo+Pj4+ ICAgICB2ZHBhOiBpbnRyb2R1Y2UgY29uZmlnIG9wZXJhdGlvbnMgZm9yIGFzc29jaWF0aW5nIEFT SUQgdG8gYSB2aXJ0cXVldWUKPj4+PiAgICAgICBncm91cAo+Pj4+ICAgICB2aG9zdF9pb3RsYjog c3BsaXQgb3V0IElPVExCIGluaXRpYWxpemF0aW9uCj4+Pj4gICAgIHZob3N0OiBzdXBwb3J0IEFT SUQgaW4gSU9UTEIgQVBJCj4+Pj4gICAgIHZob3N0LXZkcGE6IGludHJvZHVjZSBhc2lkIGJhc2Vk IElPVExCCj4+Pj4gICAgIHZob3N0LXZkcGE6IGludHJvZHVjZSB1QVBJIHRvIGdldCB0aGUgbnVt YmVyIG9mIHZpcnRxdWV1ZSBncm91cHMKPj4+PiAgICAgdmhvc3QtdmRwYTogaW50cm9kdWNlIHVB UEkgdG8gZ2V0IHRoZSBudW1iZXIgb2YgYWRkcmVzcyBzcGFjZXMKPj4+PiAgICAgdmhvc3QtdmRw YTogdUFQSSB0byBnZXQgdmlydHF1ZXVlIGdyb3VwIGlkCj4+Pj4gICAgIHZob3N0LXZkcGE6IGlu dHJvZHVjZSB1QVBJIHRvIHNldCBncm91cCBBU0lECj4+Pj4gICAgIHZob3N0LXZkcGE6IHN1cHBv cnQgQVNJRCBiYXNlZCBJT1RMQiBBUEkKPj4+PiAgICAgdmRwYV9zaW06IGFkdmVydGlzZSBWSVJU SU9fTkVUX0ZfTVRVCj4+Pj4gICAgIHZkcGFfc2ltOiBmYWN0b3Igb3V0IGJ1ZmZlciBjb21wbGV0 aW9uIGxvZ2ljCj4+Pj4gICAgIHZkcGFfc2ltOiBmaWx0ZXIgZGVzdGluYXRpb24gbWFjIGFkZHJl c3MKPj4+PiAgICAgdmRwYXNpbTogY29udHJvbCB2aXJ0cXVldWUgc3VwcG9ydAo+Pj4+Cj4+Pj4g U3RlZmFubyBHYXJ6YXJlbGxhICgxKToKPj4+PiAgICAgdmRwYV9zaW06IHNwbGl0IHZkcGFzaW1f dmlydHF1ZXVlJ3MgaW92IGZpZWxkIGluIG91dF9pb3YgYW5kIGluX2lvdgo+Pj4+Cj4+Pj4gICAg ZHJpdmVycy92ZHBhL2lmY3ZmL2lmY3ZmX21haW4uYyAgIHwgICA5ICstCj4+Pj4gICAgZHJpdmVy cy92ZHBhL21seDUvbmV0L21seDVfdm5ldC5jIHwgIDExICstCj4+Pj4gICAgZHJpdmVycy92ZHBh L3ZkcGEuYyAgICAgICAgICAgICAgIHwgICA4ICstCj4+Pj4gICAgZHJpdmVycy92ZHBhL3ZkcGFf c2ltL3ZkcGFfc2ltLmMgIHwgMjkyICsrKysrKysrKysrKysrKysrKysrKysrKy0tLS0tLQo+Pj4+ ICAgIGRyaXZlcnMvdmhvc3QvaW90bGIuYyAgICAgICAgICAgICB8ICAyMyArKy0KPj4+PiAgICBk cml2ZXJzL3Zob3N0L3ZkcGEuYyAgICAgICAgICAgICAgfCAyNDYgKysrKysrKysrKysrKysrKysr KystLS0tLQo+Pj4+ICAgIGRyaXZlcnMvdmhvc3Qvdmhvc3QuYyAgICAgICAgICAgICB8ICAyMyAr Ky0KPj4+PiAgICBkcml2ZXJzL3Zob3N0L3Zob3N0LmggICAgICAgICAgICAgfCAgIDQgKy0KPj4+ PiAgICBkcml2ZXJzL3ZpcnRpby92aXJ0aW9fdmRwYS5jICAgICAgfCAgIDIgKy0KPj4+PiAgICBp bmNsdWRlL2xpbnV4L3ZkcGEuaCAgICAgICAgICAgICAgfCAgNDIgKysrKy0KPj4+PiAgICBpbmNs dWRlL2xpbnV4L3Zob3N0X2lvdGxiLmggICAgICAgfCAgIDIgKwo+Pj4+ICAgIGluY2x1ZGUvdWFw aS9saW51eC92aG9zdC5oICAgICAgICB8ICAyNSArKy0KPj4+PiAgICBpbmNsdWRlL3VhcGkvbGlu dXgvdmhvc3RfdHlwZXMuaCAgfCAgMTAgKy0KPj4+PiAgICAxMyBmaWxlcyBjaGFuZ2VkLCA1NjEg aW5zZXJ0aW9ucygrKSwgMTM2IGRlbGV0aW9ucygtKQo+Pj4+Cj4+Pj4gLS0gCj4+Pj4gMi4yNS4x CgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpWaXJ0dWFs aXphdGlvbiBtYWlsaW5nIGxpc3QKVmlydHVhbGl6YXRpb25AbGlzdHMubGludXgtZm91bmRhdGlv bi5vcmcKaHR0cHM6Ly9saXN0cy5saW51eGZvdW5kYXRpb24ub3JnL21haWxtYW4vbGlzdGluZm8v dmlydHVhbGl6YXRpb24=