From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.4 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 048F6C07E95 for ; Wed, 14 Jul 2021 03:06:06 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 896EE61380 for ; Wed, 14 Jul 2021 03:06:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 896EE61380 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:54900 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m3VDo-0000sY-Mh for qemu-devel@archiver.kernel.org; Tue, 13 Jul 2021 23:06:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49520) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m3VCY-0008SE-Mz for qemu-devel@nongnu.org; Tue, 13 Jul 2021 23:04:47 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:44150) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m3VCU-0000AP-G5 for qemu-devel@nongnu.org; Tue, 13 Jul 2021 23:04:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626231880; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dt39hgR3TCbA4So26eoQKwSZdlXSs+WH9pS39/Tpb4k=; b=JQ4OGn4UQvZSXL96p9uXHs8jiNrGtBYNNTSkfH6n/xZrRL0T+ALdIAXOzpBrf8dzZPFesH TPYPmCNbOT0LWuarOjKZt5ja1H2l51Wu1ZK7S5aGtbDz/s4WaAVSm0ifIyrEHYSkCAB30t PYvnE6bp28NFVWGQaUQkcw8fLTDvw7g= Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-518-RuDS_bTwOiW8bpX_xbFmWw-1; Tue, 13 Jul 2021 23:04:39 -0400 X-MC-Unique: RuDS_bTwOiW8bpX_xbFmWw-1 Received: by mail-pl1-f200.google.com with SMTP id i7-20020a170902c947b029012b41197000so613409pla.16 for ; Tue, 13 Jul 2021 20:04:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=dt39hgR3TCbA4So26eoQKwSZdlXSs+WH9pS39/Tpb4k=; b=XMyVdl6gntkIrpa2AKmlP2ZLr5tsFrh8sShuc/w7+O5OkZugNSIYjIZxO3jWVq2S0f IclCGWA0xXXCEvFA6xGTWpDbyeAETFNjcHjFdRzLRSS+7wYi1pg0guO+Tdmw6mGQ8ydY fSMzXjfAoDyuxNainUPs2rVKy805/v0r6lGoPwXj9JrWJuX8uTDG9nueCTk1K+6dwbri J9QoGRACNJ467ECjKU6+KaRTUXoGSMEfW8QMCxSPclb/sdAPRXFkxlPesCALS7VSl6mi Cdsqco08gVWEBPFuOK8QirkRwDLnulUVyDzSOQ0rBlonG9Z+UCi0+CCO2jRXdxo9p6Q1 +kqA== X-Gm-Message-State: AOAM532o3CGoKkH8afob3wOlKCr4MIN5buuHDBubX+nuaaUovBFx6/tN AyvSLNpf4lBuDv9xFSpGR4r/4H4qpa+Tggd19QZGZBh5Khh8rKx3m3qEMe+3jhCD7Nn7c/DBrXb Ll6hyqiyS9J9IL1o= X-Received: by 2002:a17:902:7246:b029:129:dcf5:b583 with SMTP id c6-20020a1709027246b0290129dcf5b583mr5994618pll.73.1626231878202; Tue, 13 Jul 2021 20:04:38 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwYXIk0cQ8yABYM8CWPAOCyvLBqwBZMqOrjv/MDVqBpHXvDwT2YFUrlgEArcS+rk6/ytP/YxQ== X-Received: by 2002:a17:902:7246:b029:129:dcf5:b583 with SMTP id c6-20020a1709027246b0290129dcf5b583mr5994591pll.73.1626231877870; Tue, 13 Jul 2021 20:04:37 -0700 (PDT) Received: from wangxiaodeMacBook-Air.local ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id a20sm550813pfv.101.2021.07.13.20.04.32 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 13 Jul 2021 20:04:36 -0700 (PDT) Subject: Re: [RFC v3 21/29] vhost: Add VhostIOVATree To: Eugenio Perez Martin References: <20210519162903.1172366-1-eperezma@redhat.com> <20210519162903.1172366-22-eperezma@redhat.com> <755e2d24-946b-775c-e4e9-e81a16e7b6dd@redhat.com> From: Jason Wang Message-ID: <882296f9-adea-9b3b-5a2f-df0c746b4ad1@redhat.com> Date: Wed, 14 Jul 2021 11:04:27 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jasowang@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US Received-SPF: pass client-ip=170.10.133.124; envelope-from=jasowang@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -38 X-Spam_score: -3.9 X-Spam_bar: --- X-Spam_report: (-3.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.7, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, NICE_REPLY_A=-0.368, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Parav Pandit , "Michael S. Tsirkin" , Juan Quintela , Markus Armbruster , qemu-level , Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , virtualization@lists.linux-foundation.org, Eric Blake , Michael Lilja , Stefano Garzarella Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" 在 2021/6/1 下午4:15, Eugenio Perez Martin 写道: > On Mon, May 31, 2021 at 11:40 AM Jason Wang wrote: >> >> 在 2021/5/20 上午12:28, Eugenio Pérez 写道: >>> This tree is able to look for a translated address from a IOVA address. >>> >>> At first glance is similar to util/iova-tree. However, SVQ working on >>> devices with limited IOVA space need more capabilities, like allocating >>> IOVA chunks or perform reverse translations (qemu addresses to iova). >>> >>> Starting a sepparated implementation. Knowing than insertions/deletions >>> will not be as frequent as searches, >> >> This might not be true if vIOMMU is enabled. >> > Right. > >>> it uses an ordered array at >>> implementation. >> >> I wonder how much overhead could g_array be if it needs to grow. >> > I didn't do any tests, actually. But I see this VhostIOVATree as a > replaceable tool, just to get the buffer translations to work. So I'm > both ok with change it now and ok to delay it, since they should not > be hard to do. > >>> A different name could be used, but ordered >>> searchable array is a little bit long though. >> >> Note that we had a very good example for this. That is the kernel iova >> allocator which is implemented via rbtree. >> >> Instead of figuring out g_array vs g_tree stuffs, I would simple go with >> g_tree first (based on util/iova-tree) and borrow the well design kernel >> iova allocator API to have a generic IOVA one instead of coupling it >> with vhost. It could be used by other userspace driver in the future: >> >> init_iova_domain()/put_iova_domain(); >> >> alloc_iova()/free_iova(); >> >> find_iova(); >> > We could go that way, but then the iova-tree should be extended to > support both translations (iova->translated_addr is now implemented in > iova-tree, the reverse is not). If I understood you correctly, > borrowing the kernel iova allocator would give us both, right? No the reverse lookup is done via a specific IOMMU driver if I understand it correctly. And if the mapping is 1:1 we can just use two iova-tree I guess. > > Note that it is not coupled to vhost at all except in the name: all > the implementations only work with hwaddr and void pointers memory. > Just to illustrate the point, I think it could be a drop-in > replacement for iova-tree at this moment (with all the > drawbacks/advantages of an array vs tree). Ok. Thanks > >> Another reference is the iova allocator that is implemented in VFIO. > I will check this too. > > Thanks! > > >> Thanks >> >> >>> Signed-off-by: Eugenio Pérez >>> --- >>> hw/virtio/vhost-iova-tree.h | 50 ++++++++++ >>> hw/virtio/vhost-iova-tree.c | 188 ++++++++++++++++++++++++++++++++++++ >>> hw/virtio/meson.build | 2 +- >>> 3 files changed, 239 insertions(+), 1 deletion(-) >>> create mode 100644 hw/virtio/vhost-iova-tree.h >>> create mode 100644 hw/virtio/vhost-iova-tree.c >>> >>> diff --git a/hw/virtio/vhost-iova-tree.h b/hw/virtio/vhost-iova-tree.h >>> new file mode 100644 >>> index 0000000000..2a44af8b3a >>> --- /dev/null >>> +++ b/hw/virtio/vhost-iova-tree.h >>> @@ -0,0 +1,50 @@ >>> +/* >>> + * vhost software live migration ring >>> + * >>> + * SPDX-FileCopyrightText: Red Hat, Inc. 2021 >>> + * SPDX-FileContributor: Author: Eugenio Pérez >>> + * >>> + * SPDX-License-Identifier: GPL-2.0-or-later >>> + */ >>> + >>> +#ifndef HW_VIRTIO_VHOST_IOVA_TREE_H >>> +#define HW_VIRTIO_VHOST_IOVA_TREE_H >>> + >>> +#include >>> + >>> +#include "exec/memory.h" >>> + >>> +typedef struct VhostDMAMap { >>> + void *translated_addr; >>> + hwaddr iova; >>> + hwaddr size; /* Inclusive */ >>> + IOMMUAccessFlags perm; >>> +} VhostDMAMap; >>> + >>> +typedef enum VhostDMAMapNewRC { >>> + VHOST_DMA_MAP_OVERLAP = -2, >>> + VHOST_DMA_MAP_INVALID = -1, >>> + VHOST_DMA_MAP_OK = 0, >>> +} VhostDMAMapNewRC; >>> + >>> +/** >>> + * VhostIOVATree >>> + * >>> + * Store and search IOVA -> Translated mappings. >>> + * >>> + * Note that it cannot remove nodes. >>> + */ >>> +typedef struct VhostIOVATree { >>> + /* Ordered array of reverse translations, IOVA address to qemu memory. */ >>> + GArray *iova_taddr_map; >>> +} VhostIOVATree; >>> + >>> +void vhost_iova_tree_new(VhostIOVATree *iova_rm); >>> +void vhost_iova_tree_destroy(VhostIOVATree *iova_rm); >>> + >>> +const VhostDMAMap *vhost_iova_tree_find_taddr(const VhostIOVATree *iova_rm, >>> + const VhostDMAMap *map); >>> +VhostDMAMapNewRC vhost_iova_tree_insert(VhostIOVATree *iova_rm, >>> + VhostDMAMap *map); >>> + >>> +#endif >>> diff --git a/hw/virtio/vhost-iova-tree.c b/hw/virtio/vhost-iova-tree.c >>> new file mode 100644 >>> index 0000000000..dfd7e448b5 >>> --- /dev/null >>> +++ b/hw/virtio/vhost-iova-tree.c >>> @@ -0,0 +1,188 @@ >>> +/* >>> + * vhost software live migration ring >>> + * >>> + * SPDX-FileCopyrightText: Red Hat, Inc. 2021 >>> + * SPDX-FileContributor: Author: Eugenio Pérez >>> + * >>> + * SPDX-License-Identifier: GPL-2.0-or-later >>> + */ >>> + >>> +#include "qemu/osdep.h" >>> +#include "vhost-iova-tree.h" >>> + >>> +#define G_ARRAY_NOT_ZERO_TERMINATED false >>> +#define G_ARRAY_NOT_CLEAR_ON_ALLOC false >>> + >>> +/** >>> + * Inserts an element after an existing one in garray. >>> + * >>> + * @array The array >>> + * @prev_elem The previous element of array of NULL if prepending >>> + * @map The DMA map >>> + * >>> + * It provides the aditional advantage of being type safe over >>> + * g_array_insert_val, which accepts a reference pointer instead of a value >>> + * with no complains. >>> + */ >>> +static void vhost_iova_tree_insert_after(GArray *array, >>> + const VhostDMAMap *prev_elem, >>> + const VhostDMAMap *map) >>> +{ >>> + size_t pos; >>> + >>> + if (!prev_elem) { >>> + pos = 0; >>> + } else { >>> + pos = prev_elem - &g_array_index(array, typeof(*prev_elem), 0) + 1; >>> + } >>> + >>> + g_array_insert_val(array, pos, *map); >>> +} >>> + >>> +static gint vhost_iova_tree_cmp_iova(gconstpointer a, gconstpointer b) >>> +{ >>> + const VhostDMAMap *m1 = a, *m2 = b; >>> + >>> + if (m1->iova > m2->iova + m2->size) { >>> + return 1; >>> + } >>> + >>> + if (m1->iova + m1->size < m2->iova) { >>> + return -1; >>> + } >>> + >>> + /* Overlapped */ >>> + return 0; >>> +} >>> + >>> +/** >>> + * Find the previous node to a given iova >>> + * >>> + * @array The ascending ordered-by-translated-addr array of VhostDMAMap >>> + * @map The map to insert >>> + * @prev Returned location of the previous map >>> + * >>> + * Return VHOST_DMA_MAP_OK if everything went well, or VHOST_DMA_MAP_OVERLAP if >>> + * it already exists. It is ok to use this function to check if a given range >>> + * exists, but it will use a linear search. >>> + * >>> + * TODO: We can use bsearch to locate the entry if we save the state in the >>> + * needle, knowing that the needle is always the first argument to >>> + * compare_func. >>> + */ >>> +static VhostDMAMapNewRC vhost_iova_tree_find_prev(const GArray *array, >>> + GCompareFunc compare_func, >>> + const VhostDMAMap *map, >>> + const VhostDMAMap **prev) >>> +{ >>> + size_t i; >>> + int r; >>> + >>> + *prev = NULL; >>> + for (i = 0; i < array->len; ++i) { >>> + r = compare_func(map, &g_array_index(array, typeof(*map), i)); >>> + if (r == 0) { >>> + return VHOST_DMA_MAP_OVERLAP; >>> + } >>> + if (r < 0) { >>> + return VHOST_DMA_MAP_OK; >>> + } >>> + >>> + *prev = &g_array_index(array, typeof(**prev), i); >>> + } >>> + >>> + return VHOST_DMA_MAP_OK; >>> +} >>> + >>> +/** >>> + * Create a new IOVA tree >>> + * >>> + * @tree The IOVA tree >>> + */ >>> +void vhost_iova_tree_new(VhostIOVATree *tree) >>> +{ >>> + assert(tree); >>> + >>> + tree->iova_taddr_map = g_array_new(G_ARRAY_NOT_ZERO_TERMINATED, >>> + G_ARRAY_NOT_CLEAR_ON_ALLOC, >>> + sizeof(VhostDMAMap)); >>> +} >>> + >>> +/** >>> + * Destroy an IOVA tree >>> + * >>> + * @tree The iova tree >>> + */ >>> +void vhost_iova_tree_destroy(VhostIOVATree *tree) >>> +{ >>> + g_array_unref(g_steal_pointer(&tree->iova_taddr_map)); >>> +} >>> + >>> +/** >>> + * Perform a search on a GArray. >>> + * >>> + * @array Glib array >>> + * @map Map to look up >>> + * @compare_func Compare function to use >>> + * >>> + * Return The found element or NULL if not found. >>> + * >>> + * This can be replaced with g_array_binary_search (Since glib 2.62) when that >>> + * is common enough. >>> + */ >>> +static const VhostDMAMap *vhost_iova_tree_bsearch(const GArray *array, >>> + const VhostDMAMap *map, >>> + GCompareFunc compare_func) >>> +{ >>> + return bsearch(map, array->data, array->len, sizeof(*map), compare_func); >>> +} >>> + >>> +/** >>> + * Find the translated address stored from a IOVA address >>> + * >>> + * @tree The iova tree >>> + * @map The map with the memory address >>> + * >>> + * Return the stored mapping, or NULL if not found. >>> + */ >>> +const VhostDMAMap *vhost_iova_tree_find_taddr(const VhostIOVATree *tree, >>> + const VhostDMAMap *map) >>> +{ >>> + return vhost_iova_tree_bsearch(tree->iova_taddr_map, map, >>> + vhost_iova_tree_cmp_iova); >>> +} >>> + >>> +/** >>> + * Insert a new map >>> + * >>> + * @tree The iova tree >>> + * @map The iova map >>> + * >>> + * Returns: >>> + * - VHOST_DMA_MAP_OK if the map fits in the container >>> + * - VHOST_DMA_MAP_INVALID if the map does not make sense (like size overflow) >>> + * - VHOST_DMA_MAP_OVERLAP if the tree already contains that map >>> + * Can query the assignated iova in map. >>> + */ >>> +VhostDMAMapNewRC vhost_iova_tree_insert(VhostIOVATree *tree, >>> + VhostDMAMap *map) >>> +{ >>> + const VhostDMAMap *prev; >>> + int find_prev_rc; >>> + >>> + if (map->translated_addr + map->size < map->translated_addr || >>> + map->iova + map->size < map->iova || map->perm == IOMMU_NONE) { >>> + return VHOST_DMA_MAP_INVALID; >>> + } >>> + >>> + /* Check for duplicates, and save position for insertion */ >>> + find_prev_rc = vhost_iova_tree_find_prev(tree->iova_taddr_map, >>> + vhost_iova_tree_cmp_iova, map, >>> + &prev); >>> + if (find_prev_rc == VHOST_DMA_MAP_OVERLAP) { >>> + return VHOST_DMA_MAP_OVERLAP; >>> + } >>> + >>> + vhost_iova_tree_insert_after(tree->iova_taddr_map, prev, map); >>> + return VHOST_DMA_MAP_OK; >>> +} >>> diff --git a/hw/virtio/meson.build b/hw/virtio/meson.build >>> index 8b5a0225fe..cb306b83c6 100644 >>> --- a/hw/virtio/meson.build >>> +++ b/hw/virtio/meson.build >>> @@ -11,7 +11,7 @@ softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('vhost-stub.c')) >>> >>> virtio_ss = ss.source_set() >>> virtio_ss.add(files('virtio.c')) >>> -virtio_ss.add(when: 'CONFIG_VHOST', if_true: files('vhost.c', 'vhost-backend.c', 'vhost-shadow-virtqueue.c')) >>> +virtio_ss.add(when: 'CONFIG_VHOST', if_true: files('vhost.c', 'vhost-backend.c', 'vhost-shadow-virtqueue.c', 'vhost-iova-tree.c')) >>> virtio_ss.add(when: 'CONFIG_VHOST_USER', if_true: files('vhost-user.c')) >>> virtio_ss.add(when: 'CONFIG_VHOST_VDPA', if_true: files('vhost-vdpa.c')) >>> virtio_ss.add(when: 'CONFIG_VIRTIO_BALLOON', if_true: files('virtio-balloon.c')) From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.4 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2562AC07E95 for ; Wed, 14 Jul 2021 03:04:47 +0000 (UTC) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C2ED561380 for ; Wed, 14 Jul 2021 03:04:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C2ED561380 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=virtualization-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 75E8840556; Wed, 14 Jul 2021 03:04:46 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zfUpUrNCckYF; Wed, 14 Jul 2021 03:04:45 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id AF09640251; Wed, 14 Jul 2021 03:04:44 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 846F2C0010; Wed, 14 Jul 2021 03:04:44 +0000 (UTC) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1FC61C000E for ; Wed, 14 Jul 2021 03:04:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 0572183B73 for ; Wed, 14 Jul 2021 03:04:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qweWETuaH1ZJ for ; Wed, 14 Jul 2021 03:04:42 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id DC22F83AFB for ; Wed, 14 Jul 2021 03:04:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626231880; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dt39hgR3TCbA4So26eoQKwSZdlXSs+WH9pS39/Tpb4k=; b=JQ4OGn4UQvZSXL96p9uXHs8jiNrGtBYNNTSkfH6n/xZrRL0T+ALdIAXOzpBrf8dzZPFesH TPYPmCNbOT0LWuarOjKZt5ja1H2l51Wu1ZK7S5aGtbDz/s4WaAVSm0ifIyrEHYSkCAB30t PYvnE6bp28NFVWGQaUQkcw8fLTDvw7g= Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-518-tlijEzzrOXG_TMqqj8zMBQ-1; Tue, 13 Jul 2021 23:04:39 -0400 X-MC-Unique: tlijEzzrOXG_TMqqj8zMBQ-1 Received: by mail-pg1-f198.google.com with SMTP id k25-20020a635a590000b02902288bbae35bso478481pgm.0 for ; Tue, 13 Jul 2021 20:04:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=dt39hgR3TCbA4So26eoQKwSZdlXSs+WH9pS39/Tpb4k=; b=gQ4mmCEczJf1nuGfadapzYeCZQECOYKKbfkTx/rHPMND82glugrIGH5M9+MlOMWFtg hKEtVHy+bsEGBwWOE3E6fljDl287JxyaLKJR0P73Q4kIrUEkHqMDLAt6QaRZsfTY1yr+ o7VdHyKQpNmA9IHkuyyNJnBHSBYjRlZEoCl+Yh9EgbfyuWtL+fvqodFNm9Et6H8MNWOD 7MLdkNofFnxtRSLLwlnVAPt6j8OYgHuHMLPHl5OQxNmRHrTVP5Q78ztSffSZMZUEleat JiYnsnBCl0Ekf0F7p+K8ZVJrsmUFd8rfvZ6ElBuLhBealuvEU6ajdsql+cHrcef7XLWK Uu9g== X-Gm-Message-State: AOAM530WtDt0nfX3V6fX6rjnX/13T08VGsDF74W+R9uFLzP4lXmsjZGB YgQpTr9C39jlQIZsZTcTs+hsY0BLaTPot0Qw54InF5grXwwi59ZBngPK0vqSQd433Nnrr2NbmJe ENnaFufgONshr4U3N5FgjAvMyXxRV1JCbHzCHhq17ZQjAkiv91AebtTPME1V+knAB0Z1UgT7ZHy T5NAdLxzJHx2JNpJ7umw== X-Received: by 2002:a17:902:7246:b029:129:dcf5:b583 with SMTP id c6-20020a1709027246b0290129dcf5b583mr5994630pll.73.1626231878302; Tue, 13 Jul 2021 20:04:38 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwYXIk0cQ8yABYM8CWPAOCyvLBqwBZMqOrjv/MDVqBpHXvDwT2YFUrlgEArcS+rk6/ytP/YxQ== X-Received: by 2002:a17:902:7246:b029:129:dcf5:b583 with SMTP id c6-20020a1709027246b0290129dcf5b583mr5994591pll.73.1626231877870; Tue, 13 Jul 2021 20:04:37 -0700 (PDT) Received: from wangxiaodeMacBook-Air.local ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id a20sm550813pfv.101.2021.07.13.20.04.32 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 13 Jul 2021 20:04:36 -0700 (PDT) Subject: Re: [RFC v3 21/29] vhost: Add VhostIOVATree To: Eugenio Perez Martin References: <20210519162903.1172366-1-eperezma@redhat.com> <20210519162903.1172366-22-eperezma@redhat.com> <755e2d24-946b-775c-e4e9-e81a16e7b6dd@redhat.com> From: Jason Wang Message-ID: <882296f9-adea-9b3b-5a2f-df0c746b4ad1@redhat.com> Date: Wed, 14 Jul 2021 11:04:27 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jasowang@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Cc: Parav Pandit , "Michael S. Tsirkin" , Markus Armbruster , qemu-level , Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , virtualization@lists.linux-foundation.org, Eric Blake , Michael Lilja X-BeenThere: virtualization@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Linux virtualization List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Errors-To: virtualization-bounces@lists.linux-foundation.org Sender: "Virtualization" CuWcqCAyMDIxLzYvMSDkuIvljYg0OjE1LCBFdWdlbmlvIFBlcmV6IE1hcnRpbiDlhpnpgZM6Cj4g T24gTW9uLCBNYXkgMzEsIDIwMjEgYXQgMTE6NDAgQU0gSmFzb24gV2FuZyA8amFzb3dhbmdAcmVk aGF0LmNvbT4gd3JvdGU6Cj4+Cj4+IOWcqCAyMDIxLzUvMjAg5LiK5Y2IMTI6MjgsIEV1Z2VuaW8g UMOpcmV6IOWGmemBkzoKPj4+IFRoaXMgdHJlZSBpcyBhYmxlIHRvIGxvb2sgZm9yIGEgdHJhbnNs YXRlZCBhZGRyZXNzIGZyb20gYSBJT1ZBIGFkZHJlc3MuCj4+Pgo+Pj4gQXQgZmlyc3QgZ2xhbmNl IGlzIHNpbWlsYXIgdG8gdXRpbC9pb3ZhLXRyZWUuIEhvd2V2ZXIsIFNWUSB3b3JraW5nIG9uCj4+ PiBkZXZpY2VzIHdpdGggbGltaXRlZCBJT1ZBIHNwYWNlIG5lZWQgbW9yZSBjYXBhYmlsaXRpZXMs IGxpa2UgYWxsb2NhdGluZwo+Pj4gSU9WQSBjaHVua3Mgb3IgcGVyZm9ybSByZXZlcnNlIHRyYW5z bGF0aW9ucyAocWVtdSBhZGRyZXNzZXMgdG8gaW92YSkuCj4+Pgo+Pj4gU3RhcnRpbmcgYSBzZXBw YXJhdGVkIGltcGxlbWVudGF0aW9uLiBLbm93aW5nIHRoYW4gaW5zZXJ0aW9ucy9kZWxldGlvbnMK Pj4+IHdpbGwgbm90IGJlIGFzIGZyZXF1ZW50IGFzIHNlYXJjaGVzLAo+Pgo+PiBUaGlzIG1pZ2h0 IG5vdCBiZSB0cnVlIGlmIHZJT01NVSBpcyBlbmFibGVkLgo+Pgo+IFJpZ2h0Lgo+Cj4+PiBpdCB1 c2VzIGFuIG9yZGVyZWQgYXJyYXkgYXQKPj4+IGltcGxlbWVudGF0aW9uLgo+Pgo+PiBJIHdvbmRl ciBob3cgbXVjaCBvdmVyaGVhZCBjb3VsZCBnX2FycmF5IGJlIGlmIGl0IG5lZWRzIHRvIGdyb3cu Cj4+Cj4gSSBkaWRuJ3QgZG8gYW55IHRlc3RzLCBhY3R1YWxseS4gQnV0IEkgc2VlIHRoaXMgVmhv c3RJT1ZBVHJlZSBhcyBhCj4gcmVwbGFjZWFibGUgdG9vbCwganVzdCB0byBnZXQgdGhlIGJ1ZmZl ciB0cmFuc2xhdGlvbnMgdG8gd29yay4gU28gSSdtCj4gYm90aCBvayB3aXRoIGNoYW5nZSBpdCBu b3cgYW5kIG9rIHRvIGRlbGF5IGl0LCBzaW5jZSB0aGV5IHNob3VsZCBub3QKPiBiZSBoYXJkIHRv IGRvLgo+Cj4+PiAgICBBIGRpZmZlcmVudCBuYW1lIGNvdWxkIGJlIHVzZWQsIGJ1dCBvcmRlcmVk Cj4+PiBzZWFyY2hhYmxlIGFycmF5IGlzIGEgbGl0dGxlIGJpdCBsb25nIHRob3VnaC4KPj4KPj4g Tm90ZSB0aGF0IHdlIGhhZCBhIHZlcnkgZ29vZCBleGFtcGxlIGZvciB0aGlzLiBUaGF0IGlzIHRo ZSBrZXJuZWwgaW92YQo+PiBhbGxvY2F0b3Igd2hpY2ggaXMgaW1wbGVtZW50ZWQgdmlhIHJidHJl ZS4KPj4KPj4gSW5zdGVhZCBvZiBmaWd1cmluZyBvdXQgZ19hcnJheSB2cyBnX3RyZWUgc3R1ZmZz LCBJIHdvdWxkIHNpbXBsZSBnbyB3aXRoCj4+IGdfdHJlZSBmaXJzdCAoYmFzZWQgb24gdXRpbC9p b3ZhLXRyZWUpIGFuZCBib3Jyb3cgdGhlIHdlbGwgZGVzaWduIGtlcm5lbAo+PiBpb3ZhIGFsbG9j YXRvciBBUEkgdG8gaGF2ZSBhIGdlbmVyaWMgSU9WQSBvbmUgaW5zdGVhZCBvZiBjb3VwbGluZyBp dAo+PiB3aXRoIHZob3N0LiBJdCBjb3VsZCBiZSB1c2VkIGJ5IG90aGVyIHVzZXJzcGFjZSBkcml2 ZXIgaW4gdGhlIGZ1dHVyZToKPj4KPj4gaW5pdF9pb3ZhX2RvbWFpbigpL3B1dF9pb3ZhX2RvbWFp bigpOwo+Pgo+PiBhbGxvY19pb3ZhKCkvZnJlZV9pb3ZhKCk7Cj4+Cj4+IGZpbmRfaW92YSgpOwo+ Pgo+IFdlIGNvdWxkIGdvIHRoYXQgd2F5LCBidXQgdGhlbiB0aGUgaW92YS10cmVlIHNob3VsZCBi ZSBleHRlbmRlZCB0bwo+IHN1cHBvcnQgYm90aCB0cmFuc2xhdGlvbnMgKGlvdmEtPnRyYW5zbGF0 ZWRfYWRkciBpcyBub3cgaW1wbGVtZW50ZWQgaW4KPiBpb3ZhLXRyZWUsIHRoZSByZXZlcnNlIGlz IG5vdCkuIElmIEkgdW5kZXJzdG9vZCB5b3UgY29ycmVjdGx5LAo+IGJvcnJvd2luZyB0aGUga2Vy bmVsIGlvdmEgYWxsb2NhdG9yIHdvdWxkIGdpdmUgdXMgYm90aCwgcmlnaHQ/CgoKTm8gdGhlIHJl dmVyc2UgbG9va3VwIGlzIGRvbmUgdmlhIGEgc3BlY2lmaWMgSU9NTVUgZHJpdmVyIGlmIEkgCnVu ZGVyc3RhbmQgaXQgY29ycmVjdGx5LgoKQW5kIGlmIHRoZSBtYXBwaW5nIGlzIDE6MSB3ZSBjYW4g anVzdCB1c2UgdHdvIGlvdmEtdHJlZSBJIGd1ZXNzLgoKCj4KPiBOb3RlIHRoYXQgaXQgaXMgbm90 IGNvdXBsZWQgdG8gdmhvc3QgYXQgYWxsIGV4Y2VwdCBpbiB0aGUgbmFtZTogYWxsCj4gdGhlIGlt cGxlbWVudGF0aW9ucyBvbmx5IHdvcmsgd2l0aCBod2FkZHIgYW5kIHZvaWQgcG9pbnRlcnMgbWVt b3J5Lgo+IEp1c3QgdG8gaWxsdXN0cmF0ZSB0aGUgcG9pbnQsIEkgdGhpbmsgaXQgY291bGQgYmUg YSBkcm9wLWluCj4gcmVwbGFjZW1lbnQgZm9yIGlvdmEtdHJlZSBhdCB0aGlzIG1vbWVudCAod2l0 aCBhbGwgdGhlCj4gZHJhd2JhY2tzL2FkdmFudGFnZXMgb2YgYW4gYXJyYXkgdnMgdHJlZSkuCgoK T2suCgpUaGFua3MKCgo+Cj4+IEFub3RoZXIgcmVmZXJlbmNlIGlzIHRoZSBpb3ZhIGFsbG9jYXRv ciB0aGF0IGlzIGltcGxlbWVudGVkIGluIFZGSU8uCj4gSSB3aWxsIGNoZWNrIHRoaXMgdG9vLgo+ Cj4gVGhhbmtzIQo+Cj4KPj4gVGhhbmtzCj4+Cj4+Cj4+PiBTaWduZWQtb2ZmLWJ5OiBFdWdlbmlv IFDDqXJleiA8ZXBlcmV6bWFAcmVkaGF0LmNvbT4KPj4+IC0tLQo+Pj4gICAgaHcvdmlydGlvL3Zo b3N0LWlvdmEtdHJlZS5oIHwgIDUwICsrKysrKysrKysKPj4+ICAgIGh3L3ZpcnRpby92aG9zdC1p b3ZhLXRyZWUuYyB8IDE4OCArKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysKPj4+ ICAgIGh3L3ZpcnRpby9tZXNvbi5idWlsZCAgICAgICB8ICAgMiArLQo+Pj4gICAgMyBmaWxlcyBj aGFuZ2VkLCAyMzkgaW5zZXJ0aW9ucygrKSwgMSBkZWxldGlvbigtKQo+Pj4gICAgY3JlYXRlIG1v ZGUgMTAwNjQ0IGh3L3ZpcnRpby92aG9zdC1pb3ZhLXRyZWUuaAo+Pj4gICAgY3JlYXRlIG1vZGUg MTAwNjQ0IGh3L3ZpcnRpby92aG9zdC1pb3ZhLXRyZWUuYwo+Pj4KPj4+IGRpZmYgLS1naXQgYS9o dy92aXJ0aW8vdmhvc3QtaW92YS10cmVlLmggYi9ody92aXJ0aW8vdmhvc3QtaW92YS10cmVlLmgK Pj4+IG5ldyBmaWxlIG1vZGUgMTAwNjQ0Cj4+PiBpbmRleCAwMDAwMDAwMDAwLi4yYTQ0YWY4YjNh Cj4+PiAtLS0gL2Rldi9udWxsCj4+PiArKysgYi9ody92aXJ0aW8vdmhvc3QtaW92YS10cmVlLmgK Pj4+IEBAIC0wLDAgKzEsNTAgQEAKPj4+ICsvKgo+Pj4gKyAqIHZob3N0IHNvZnR3YXJlIGxpdmUg bWlncmF0aW9uIHJpbmcKPj4+ICsgKgo+Pj4gKyAqIFNQRFgtRmlsZUNvcHlyaWdodFRleHQ6IFJl ZCBIYXQsIEluYy4gMjAyMQo+Pj4gKyAqIFNQRFgtRmlsZUNvbnRyaWJ1dG9yOiBBdXRob3I6IEV1 Z2VuaW8gUMOpcmV6IDxlcGVyZXptYUByZWRoYXQuY29tPgo+Pj4gKyAqCj4+PiArICogU1BEWC1M aWNlbnNlLUlkZW50aWZpZXI6IEdQTC0yLjAtb3ItbGF0ZXIKPj4+ICsgKi8KPj4+ICsKPj4+ICsj aWZuZGVmIEhXX1ZJUlRJT19WSE9TVF9JT1ZBX1RSRUVfSAo+Pj4gKyNkZWZpbmUgSFdfVklSVElP X1ZIT1NUX0lPVkFfVFJFRV9ICj4+PiArCj4+PiArI2luY2x1ZGUgPGdtb2R1bGUuaD4KPj4+ICsK Pj4+ICsjaW5jbHVkZSAiZXhlYy9tZW1vcnkuaCIKPj4+ICsKPj4+ICt0eXBlZGVmIHN0cnVjdCBW aG9zdERNQU1hcCB7Cj4+PiArICAgIHZvaWQgKnRyYW5zbGF0ZWRfYWRkcjsKPj4+ICsgICAgaHdh ZGRyIGlvdmE7Cj4+PiArICAgIGh3YWRkciBzaXplOyAgICAgICAgICAgICAgICAvKiBJbmNsdXNp dmUgKi8KPj4+ICsgICAgSU9NTVVBY2Nlc3NGbGFncyBwZXJtOwo+Pj4gK30gVmhvc3RETUFNYXA7 Cj4+PiArCj4+PiArdHlwZWRlZiBlbnVtIFZob3N0RE1BTWFwTmV3UkMgewo+Pj4gKyAgICBWSE9T VF9ETUFfTUFQX09WRVJMQVAgPSAtMiwKPj4+ICsgICAgVkhPU1RfRE1BX01BUF9JTlZBTElEID0g LTEsCj4+PiArICAgIFZIT1NUX0RNQV9NQVBfT0sgPSAwLAo+Pj4gK30gVmhvc3RETUFNYXBOZXdS QzsKPj4+ICsKPj4+ICsvKioKPj4+ICsgKiBWaG9zdElPVkFUcmVlCj4+PiArICoKPj4+ICsgKiBT dG9yZSBhbmQgc2VhcmNoIElPVkEgLT4gVHJhbnNsYXRlZCBtYXBwaW5ncy4KPj4+ICsgKgo+Pj4g KyAqIE5vdGUgdGhhdCBpdCBjYW5ub3QgcmVtb3ZlIG5vZGVzLgo+Pj4gKyAqLwo+Pj4gK3R5cGVk ZWYgc3RydWN0IFZob3N0SU9WQVRyZWUgewo+Pj4gKyAgICAvKiBPcmRlcmVkIGFycmF5IG9mIHJl dmVyc2UgdHJhbnNsYXRpb25zLCBJT1ZBIGFkZHJlc3MgdG8gcWVtdSBtZW1vcnkuICovCj4+PiAr ICAgIEdBcnJheSAqaW92YV90YWRkcl9tYXA7Cj4+PiArfSBWaG9zdElPVkFUcmVlOwo+Pj4gKwo+ Pj4gK3ZvaWQgdmhvc3RfaW92YV90cmVlX25ldyhWaG9zdElPVkFUcmVlICppb3ZhX3JtKTsKPj4+ ICt2b2lkIHZob3N0X2lvdmFfdHJlZV9kZXN0cm95KFZob3N0SU9WQVRyZWUgKmlvdmFfcm0pOwo+ Pj4gKwo+Pj4gK2NvbnN0IFZob3N0RE1BTWFwICp2aG9zdF9pb3ZhX3RyZWVfZmluZF90YWRkcihj b25zdCBWaG9zdElPVkFUcmVlICppb3ZhX3JtLAo+Pj4gKyAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICBjb25zdCBWaG9zdERNQU1hcCAqbWFwKTsKPj4+ICtWaG9z dERNQU1hcE5ld1JDIHZob3N0X2lvdmFfdHJlZV9pbnNlcnQoVmhvc3RJT1ZBVHJlZSAqaW92YV9y bSwKPj4+ICsgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgVmhvc3RETUFN YXAgKm1hcCk7Cj4+PiArCj4+PiArI2VuZGlmCj4+PiBkaWZmIC0tZ2l0IGEvaHcvdmlydGlvL3Zo b3N0LWlvdmEtdHJlZS5jIGIvaHcvdmlydGlvL3Zob3N0LWlvdmEtdHJlZS5jCj4+PiBuZXcgZmls ZSBtb2RlIDEwMDY0NAo+Pj4gaW5kZXggMDAwMDAwMDAwMC4uZGZkN2U0NDhiNQo+Pj4gLS0tIC9k ZXYvbnVsbAo+Pj4gKysrIGIvaHcvdmlydGlvL3Zob3N0LWlvdmEtdHJlZS5jCj4+PiBAQCAtMCww ICsxLDE4OCBAQAo+Pj4gKy8qCj4+PiArICogdmhvc3Qgc29mdHdhcmUgbGl2ZSBtaWdyYXRpb24g cmluZwo+Pj4gKyAqCj4+PiArICogU1BEWC1GaWxlQ29weXJpZ2h0VGV4dDogUmVkIEhhdCwgSW5j LiAyMDIxCj4+PiArICogU1BEWC1GaWxlQ29udHJpYnV0b3I6IEF1dGhvcjogRXVnZW5pbyBQw6ly ZXogPGVwZXJlem1hQHJlZGhhdC5jb20+Cj4+PiArICoKPj4+ICsgKiBTUERYLUxpY2Vuc2UtSWRl bnRpZmllcjogR1BMLTIuMC1vci1sYXRlcgo+Pj4gKyAqLwo+Pj4gKwo+Pj4gKyNpbmNsdWRlICJx ZW11L29zZGVwLmgiCj4+PiArI2luY2x1ZGUgInZob3N0LWlvdmEtdHJlZS5oIgo+Pj4gKwo+Pj4g KyNkZWZpbmUgR19BUlJBWV9OT1RfWkVST19URVJNSU5BVEVEIGZhbHNlCj4+PiArI2RlZmluZSBH X0FSUkFZX05PVF9DTEVBUl9PTl9BTExPQyBmYWxzZQo+Pj4gKwo+Pj4gKy8qKgo+Pj4gKyAqIElu c2VydHMgYW4gZWxlbWVudCBhZnRlciBhbiBleGlzdGluZyBvbmUgaW4gZ2FycmF5Lgo+Pj4gKyAq Cj4+PiArICogQGFycmF5ICAgICAgVGhlIGFycmF5Cj4+PiArICogQHByZXZfZWxlbSAgVGhlIHBy ZXZpb3VzIGVsZW1lbnQgb2YgYXJyYXkgb2YgTlVMTCBpZiBwcmVwZW5kaW5nCj4+PiArICogQG1h cCAgICAgICAgVGhlIERNQSBtYXAKPj4+ICsgKgo+Pj4gKyAqIEl0IHByb3ZpZGVzIHRoZSBhZGl0 aW9uYWwgYWR2YW50YWdlIG9mIGJlaW5nIHR5cGUgc2FmZSBvdmVyCj4+PiArICogZ19hcnJheV9p bnNlcnRfdmFsLCB3aGljaCBhY2NlcHRzIGEgcmVmZXJlbmNlIHBvaW50ZXIgaW5zdGVhZCBvZiBh IHZhbHVlCj4+PiArICogd2l0aCBubyBjb21wbGFpbnMuCj4+PiArICovCj4+PiArc3RhdGljIHZv aWQgdmhvc3RfaW92YV90cmVlX2luc2VydF9hZnRlcihHQXJyYXkgKmFycmF5LAo+Pj4gKyAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY29uc3QgVmhvc3RETUFNYXAgKnBy ZXZfZWxlbSwKPj4+ICsgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNv bnN0IFZob3N0RE1BTWFwICptYXApCj4+PiArewo+Pj4gKyAgICBzaXplX3QgcG9zOwo+Pj4gKwo+ Pj4gKyAgICBpZiAoIXByZXZfZWxlbSkgewo+Pj4gKyAgICAgICAgcG9zID0gMDsKPj4+ICsgICAg fSBlbHNlIHsKPj4+ICsgICAgICAgIHBvcyA9IHByZXZfZWxlbSAtICZnX2FycmF5X2luZGV4KGFy cmF5LCB0eXBlb2YoKnByZXZfZWxlbSksIDApICsgMTsKPj4+ICsgICAgfQo+Pj4gKwo+Pj4gKyAg ICBnX2FycmF5X2luc2VydF92YWwoYXJyYXksIHBvcywgKm1hcCk7Cj4+PiArfQo+Pj4gKwo+Pj4g K3N0YXRpYyBnaW50IHZob3N0X2lvdmFfdHJlZV9jbXBfaW92YShnY29uc3Rwb2ludGVyIGEsIGdj b25zdHBvaW50ZXIgYikKPj4+ICt7Cj4+PiArICAgIGNvbnN0IFZob3N0RE1BTWFwICptMSA9IGEs ICptMiA9IGI7Cj4+PiArCj4+PiArICAgIGlmIChtMS0+aW92YSA+IG0yLT5pb3ZhICsgbTItPnNp emUpIHsKPj4+ICsgICAgICAgIHJldHVybiAxOwo+Pj4gKyAgICB9Cj4+PiArCj4+PiArICAgIGlm IChtMS0+aW92YSArIG0xLT5zaXplIDwgbTItPmlvdmEpIHsKPj4+ICsgICAgICAgIHJldHVybiAt MTsKPj4+ICsgICAgfQo+Pj4gKwo+Pj4gKyAgICAvKiBPdmVybGFwcGVkICovCj4+PiArICAgIHJl dHVybiAwOwo+Pj4gK30KPj4+ICsKPj4+ICsvKioKPj4+ICsgKiBGaW5kIHRoZSBwcmV2aW91cyBu b2RlIHRvIGEgZ2l2ZW4gaW92YQo+Pj4gKyAqCj4+PiArICogQGFycmF5ICBUaGUgYXNjZW5kaW5n IG9yZGVyZWQtYnktdHJhbnNsYXRlZC1hZGRyIGFycmF5IG9mIFZob3N0RE1BTWFwCj4+PiArICog QG1hcCAgICBUaGUgbWFwIHRvIGluc2VydAo+Pj4gKyAqIEBwcmV2ICAgUmV0dXJuZWQgbG9jYXRp b24gb2YgdGhlIHByZXZpb3VzIG1hcAo+Pj4gKyAqCj4+PiArICogUmV0dXJuIFZIT1NUX0RNQV9N QVBfT0sgaWYgZXZlcnl0aGluZyB3ZW50IHdlbGwsIG9yIFZIT1NUX0RNQV9NQVBfT1ZFUkxBUCBp Zgo+Pj4gKyAqIGl0IGFscmVhZHkgZXhpc3RzLiBJdCBpcyBvayB0byB1c2UgdGhpcyBmdW5jdGlv biB0byBjaGVjayBpZiBhIGdpdmVuIHJhbmdlCj4+PiArICogZXhpc3RzLCBidXQgaXQgd2lsbCB1 c2UgYSBsaW5lYXIgc2VhcmNoLgo+Pj4gKyAqCj4+PiArICogVE9ETzogV2UgY2FuIHVzZSBic2Vh cmNoIHRvIGxvY2F0ZSB0aGUgZW50cnkgaWYgd2Ugc2F2ZSB0aGUgc3RhdGUgaW4gdGhlCj4+PiAr ICogbmVlZGxlLCBrbm93aW5nIHRoYXQgdGhlIG5lZWRsZSBpcyBhbHdheXMgdGhlIGZpcnN0IGFy Z3VtZW50IHRvCj4+PiArICogY29tcGFyZV9mdW5jLgo+Pj4gKyAqLwo+Pj4gK3N0YXRpYyBWaG9z dERNQU1hcE5ld1JDIHZob3N0X2lvdmFfdHJlZV9maW5kX3ByZXYoY29uc3QgR0FycmF5ICphcnJh eSwKPj4+ICsgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IEdDb21wYXJlRnVuYyBjb21wYXJlX2Z1bmMsCj4+PiArICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICBjb25zdCBWaG9zdERNQU1hcCAqbWFwLAo+Pj4gKyAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY29uc3QgVmhv c3RETUFNYXAgKipwcmV2KQo+Pj4gK3sKPj4+ICsgICAgc2l6ZV90IGk7Cj4+PiArICAgIGludCBy Owo+Pj4gKwo+Pj4gKyAgICAqcHJldiA9IE5VTEw7Cj4+PiArICAgIGZvciAoaSA9IDA7IGkgPCBh cnJheS0+bGVuOyArK2kpIHsKPj4+ICsgICAgICAgIHIgPSBjb21wYXJlX2Z1bmMobWFwLCAmZ19h cnJheV9pbmRleChhcnJheSwgdHlwZW9mKCptYXApLCBpKSk7Cj4+PiArICAgICAgICBpZiAociA9 PSAwKSB7Cj4+PiArICAgICAgICAgICAgcmV0dXJuIFZIT1NUX0RNQV9NQVBfT1ZFUkxBUDsKPj4+ ICsgICAgICAgIH0KPj4+ICsgICAgICAgIGlmIChyIDwgMCkgewo+Pj4gKyAgICAgICAgICAgIHJl dHVybiBWSE9TVF9ETUFfTUFQX09LOwo+Pj4gKyAgICAgICAgfQo+Pj4gKwo+Pj4gKyAgICAgICAg KnByZXYgPSAmZ19hcnJheV9pbmRleChhcnJheSwgdHlwZW9mKCoqcHJldiksIGkpOwo+Pj4gKyAg ICB9Cj4+PiArCj4+PiArICAgIHJldHVybiBWSE9TVF9ETUFfTUFQX09LOwo+Pj4gK30KPj4+ICsK Pj4+ICsvKioKPj4+ICsgKiBDcmVhdGUgYSBuZXcgSU9WQSB0cmVlCj4+PiArICoKPj4+ICsgKiBA dHJlZSAgVGhlIElPVkEgdHJlZQo+Pj4gKyAqLwo+Pj4gK3ZvaWQgdmhvc3RfaW92YV90cmVlX25l dyhWaG9zdElPVkFUcmVlICp0cmVlKQo+Pj4gK3sKPj4+ICsgICAgYXNzZXJ0KHRyZWUpOwo+Pj4g Kwo+Pj4gKyAgICB0cmVlLT5pb3ZhX3RhZGRyX21hcCA9IGdfYXJyYXlfbmV3KEdfQVJSQVlfTk9U X1pFUk9fVEVSTUlOQVRFRCwKPj4+ICsgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICBHX0FSUkFZX05PVF9DTEVBUl9PTl9BTExPQywKPj4+ICsgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICBzaXplb2YoVmhvc3RETUFNYXApKTsKPj4+ICt9Cj4+PiArCj4+ PiArLyoqCj4+PiArICogRGVzdHJveSBhbiBJT1ZBIHRyZWUKPj4+ICsgKgo+Pj4gKyAqIEB0cmVl ICBUaGUgaW92YSB0cmVlCj4+PiArICovCj4+PiArdm9pZCB2aG9zdF9pb3ZhX3RyZWVfZGVzdHJv eShWaG9zdElPVkFUcmVlICp0cmVlKQo+Pj4gK3sKPj4+ICsgICAgZ19hcnJheV91bnJlZihnX3N0 ZWFsX3BvaW50ZXIoJnRyZWUtPmlvdmFfdGFkZHJfbWFwKSk7Cj4+PiArfQo+Pj4gKwo+Pj4gKy8q Kgo+Pj4gKyAqIFBlcmZvcm0gYSBzZWFyY2ggb24gYSBHQXJyYXkuCj4+PiArICoKPj4+ICsgKiBA YXJyYXkgR2xpYiBhcnJheQo+Pj4gKyAqIEBtYXAgTWFwIHRvIGxvb2sgdXAKPj4+ICsgKiBAY29t cGFyZV9mdW5jIENvbXBhcmUgZnVuY3Rpb24gdG8gdXNlCj4+PiArICoKPj4+ICsgKiBSZXR1cm4g VGhlIGZvdW5kIGVsZW1lbnQgb3IgTlVMTCBpZiBub3QgZm91bmQuCj4+PiArICoKPj4+ICsgKiBU aGlzIGNhbiBiZSByZXBsYWNlZCB3aXRoIGdfYXJyYXlfYmluYXJ5X3NlYXJjaCAoU2luY2UgZ2xp YiAyLjYyKSB3aGVuIHRoYXQKPj4+ICsgKiBpcyBjb21tb24gZW5vdWdoLgo+Pj4gKyAqLwo+Pj4g K3N0YXRpYyBjb25zdCBWaG9zdERNQU1hcCAqdmhvc3RfaW92YV90cmVlX2JzZWFyY2goY29uc3Qg R0FycmF5ICphcnJheSwKPj4+ICsgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIGNvbnN0IFZob3N0RE1BTWFwICptYXAsCj4+PiArICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBHQ29tcGFyZUZ1bmMgY29tcGFyZV9m dW5jKQo+Pj4gK3sKPj4+ICsgICAgcmV0dXJuIGJzZWFyY2gobWFwLCBhcnJheS0+ZGF0YSwgYXJy YXktPmxlbiwgc2l6ZW9mKCptYXApLCBjb21wYXJlX2Z1bmMpOwo+Pj4gK30KPj4+ICsKPj4+ICsv KioKPj4+ICsgKiBGaW5kIHRoZSB0cmFuc2xhdGVkIGFkZHJlc3Mgc3RvcmVkIGZyb20gYSBJT1ZB IGFkZHJlc3MKPj4+ICsgKgo+Pj4gKyAqIEB0cmVlICBUaGUgaW92YSB0cmVlCj4+PiArICogQG1h cCAgIFRoZSBtYXAgd2l0aCB0aGUgbWVtb3J5IGFkZHJlc3MKPj4+ICsgKgo+Pj4gKyAqIFJldHVy biB0aGUgc3RvcmVkIG1hcHBpbmcsIG9yIE5VTEwgaWYgbm90IGZvdW5kLgo+Pj4gKyAqLwo+Pj4g K2NvbnN0IFZob3N0RE1BTWFwICp2aG9zdF9pb3ZhX3RyZWVfZmluZF90YWRkcihjb25zdCBWaG9z dElPVkFUcmVlICp0cmVlLAo+Pj4gKyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICBjb25zdCBWaG9zdERNQU1hcCAqbWFwKQo+Pj4gK3sKPj4+ICsgICAgcmV0dXJu IHZob3N0X2lvdmFfdHJlZV9ic2VhcmNoKHRyZWUtPmlvdmFfdGFkZHJfbWFwLCBtYXAsCj4+PiAr ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHZob3N0X2lvdmFfdHJlZV9jbXBfaW92 YSk7Cj4+PiArfQo+Pj4gKwo+Pj4gKy8qKgo+Pj4gKyAqIEluc2VydCBhIG5ldyBtYXAKPj4+ICsg Kgo+Pj4gKyAqIEB0cmVlICBUaGUgaW92YSB0cmVlCj4+PiArICogQG1hcCAgIFRoZSBpb3ZhIG1h cAo+Pj4gKyAqCj4+PiArICogUmV0dXJuczoKPj4+ICsgKiAtIFZIT1NUX0RNQV9NQVBfT0sgaWYg dGhlIG1hcCBmaXRzIGluIHRoZSBjb250YWluZXIKPj4+ICsgKiAtIFZIT1NUX0RNQV9NQVBfSU5W QUxJRCBpZiB0aGUgbWFwIGRvZXMgbm90IG1ha2Ugc2Vuc2UgKGxpa2Ugc2l6ZSBvdmVyZmxvdykK Pj4+ICsgKiAtIFZIT1NUX0RNQV9NQVBfT1ZFUkxBUCBpZiB0aGUgdHJlZSBhbHJlYWR5IGNvbnRh aW5zIHRoYXQgbWFwCj4+PiArICogQ2FuIHF1ZXJ5IHRoZSBhc3NpZ25hdGVkIGlvdmEgaW4gbWFw Lgo+Pj4gKyAqLwo+Pj4gK1Zob3N0RE1BTWFwTmV3UkMgdmhvc3RfaW92YV90cmVlX2luc2VydChW aG9zdElPVkFUcmVlICp0cmVlLAo+Pj4gKyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICBWaG9zdERNQU1hcCAqbWFwKQo+Pj4gK3sKPj4+ICsgICAgY29uc3QgVmhvc3RETUFN YXAgKnByZXY7Cj4+PiArICAgIGludCBmaW5kX3ByZXZfcmM7Cj4+PiArCj4+PiArICAgIGlmICht YXAtPnRyYW5zbGF0ZWRfYWRkciArIG1hcC0+c2l6ZSA8IG1hcC0+dHJhbnNsYXRlZF9hZGRyIHx8 Cj4+PiArICAgICAgICBtYXAtPmlvdmEgKyBtYXAtPnNpemUgPCBtYXAtPmlvdmEgfHwgbWFwLT5w ZXJtID09IElPTU1VX05PTkUpIHsKPj4+ICsgICAgICAgIHJldHVybiBWSE9TVF9ETUFfTUFQX0lO VkFMSUQ7Cj4+PiArICAgIH0KPj4+ICsKPj4+ICsgICAgLyogQ2hlY2sgZm9yIGR1cGxpY2F0ZXMs IGFuZCBzYXZlIHBvc2l0aW9uIGZvciBpbnNlcnRpb24gKi8KPj4+ICsgICAgZmluZF9wcmV2X3Jj ID0gdmhvc3RfaW92YV90cmVlX2ZpbmRfcHJldih0cmVlLT5pb3ZhX3RhZGRyX21hcCwKPj4+ICsg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB2aG9zdF9pb3ZhX3Ry ZWVfY21wX2lvdmEsIG1hcCwKPj4+ICsgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAmcHJldik7Cj4+PiArICAgIGlmIChmaW5kX3ByZXZfcmMgPT0gVkhPU1RfRE1B X01BUF9PVkVSTEFQKSB7Cj4+PiArICAgICAgICByZXR1cm4gVkhPU1RfRE1BX01BUF9PVkVSTEFQ Owo+Pj4gKyAgICB9Cj4+PiArCj4+PiArICAgIHZob3N0X2lvdmFfdHJlZV9pbnNlcnRfYWZ0ZXIo dHJlZS0+aW92YV90YWRkcl9tYXAsIHByZXYsIG1hcCk7Cj4+PiArICAgIHJldHVybiBWSE9TVF9E TUFfTUFQX09LOwo+Pj4gK30KPj4+IGRpZmYgLS1naXQgYS9ody92aXJ0aW8vbWVzb24uYnVpbGQg Yi9ody92aXJ0aW8vbWVzb24uYnVpbGQKPj4+IGluZGV4IDhiNWEwMjI1ZmUuLmNiMzA2YjgzYzYg MTAwNjQ0Cj4+PiAtLS0gYS9ody92aXJ0aW8vbWVzb24uYnVpbGQKPj4+ICsrKyBiL2h3L3ZpcnRp by9tZXNvbi5idWlsZAo+Pj4gQEAgLTExLDcgKzExLDcgQEAgc29mdG1tdV9zcy5hZGQod2hlbjog J0NPTkZJR19BTEwnLCBpZl90cnVlOiBmaWxlcygndmhvc3Qtc3R1Yi5jJykpCj4+Pgo+Pj4gICAg dmlydGlvX3NzID0gc3Muc291cmNlX3NldCgpCj4+PiAgICB2aXJ0aW9fc3MuYWRkKGZpbGVzKCd2 aXJ0aW8uYycpKQo+Pj4gLXZpcnRpb19zcy5hZGQod2hlbjogJ0NPTkZJR19WSE9TVCcsIGlmX3Ry dWU6IGZpbGVzKCd2aG9zdC5jJywgJ3Zob3N0LWJhY2tlbmQuYycsICd2aG9zdC1zaGFkb3ctdmly dHF1ZXVlLmMnKSkKPj4+ICt2aXJ0aW9fc3MuYWRkKHdoZW46ICdDT05GSUdfVkhPU1QnLCBpZl90 cnVlOiBmaWxlcygndmhvc3QuYycsICd2aG9zdC1iYWNrZW5kLmMnLCAndmhvc3Qtc2hhZG93LXZp cnRxdWV1ZS5jJywgJ3Zob3N0LWlvdmEtdHJlZS5jJykpCj4+PiAgICB2aXJ0aW9fc3MuYWRkKHdo ZW46ICdDT05GSUdfVkhPU1RfVVNFUicsIGlmX3RydWU6IGZpbGVzKCd2aG9zdC11c2VyLmMnKSkK Pj4+ICAgIHZpcnRpb19zcy5hZGQod2hlbjogJ0NPTkZJR19WSE9TVF9WRFBBJywgaWZfdHJ1ZTog ZmlsZXMoJ3Zob3N0LXZkcGEuYycpKQo+Pj4gICAgdmlydGlvX3NzLmFkZCh3aGVuOiAnQ09ORklH X1ZJUlRJT19CQUxMT09OJywgaWZfdHJ1ZTogZmlsZXMoJ3ZpcnRpby1iYWxsb29uLmMnKSkKCl9f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fClZpcnR1YWxpemF0 aW9uIG1haWxpbmcgbGlzdApWaXJ0dWFsaXphdGlvbkBsaXN0cy5saW51eC1mb3VuZGF0aW9uLm9y ZwpodHRwczovL2xpc3RzLmxpbnV4Zm91bmRhdGlvbi5vcmcvbWFpbG1hbi9saXN0aW5mby92aXJ0 dWFsaXphdGlvbg==