From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4D83C3F6B0 for ; Tue, 23 Aug 2022 03:26:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239832AbiHWD0b (ORCPT ); Mon, 22 Aug 2022 23:26:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54380 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239519AbiHWD03 (ORCPT ); Mon, 22 Aug 2022 23:26:29 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 28AA958535 for ; Mon, 22 Aug 2022 20:26:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1661225186; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MlOaDsdp4+hX3qQ6KtrYrT2iwnuhuhg5bwgVYd5K/i8=; b=PGm+qFp8vRVLaVtuebraZSLsZsXT49EhDihQs8rCqUuj8tiZQZZ+STEV7vf6THFvoQn77O ckSJqghKDLymo4bJi/XGzYHczqOVq5aPpHw3zez7foqyZkXXUBdKopCxiup8yCbxJ2CFrv DS3IhIhmGn0f3kTEL1s9tXnPvkft9QY= Received: from mail-lf1-f72.google.com (mail-lf1-f72.google.com [209.85.167.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-55-1R-dQNCAPiyPayT48w6J6w-1; Mon, 22 Aug 2022 23:26:24 -0400 X-MC-Unique: 1R-dQNCAPiyPayT48w6J6w-1 Received: by mail-lf1-f72.google.com with SMTP id z1-20020a0565120c0100b0048ab2910b13so4002077lfu.23 for ; Mon, 22 Aug 2022 20:26:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc; bh=MlOaDsdp4+hX3qQ6KtrYrT2iwnuhuhg5bwgVYd5K/i8=; b=JhGJBfi01aBnpP0O+cY+QcwWiLKvSjDVx3sfOMlm39LLaVhXtBkHnVXyky/LFHxf43 0L1biQyfqVWwZAnLyQOr0OesezIQg+zxHRx2eFRexCkBcP+AED7CLdXgCD3GpncJeOVJ mhJrZ5a7gemh61UxeIHS9z8qX6DA4ggGq59JJ3Qi20lrb6xN0gzt75A3nWwjmAmgPvi1 EQ8ex3STnejv05oK41WSBWBJuI0Ut/2OfY7XNHt1rl9U5A/x9TXE3VIxc+b0qS6jGZ+n LlmmYaDvQJtJovxbRz5R3aSPZxLRNINTbBo/pY11RxMutpSYv3ko1pZrErhnyQlrRd6b AZug== X-Gm-Message-State: ACgBeo0Gp3baiLxmfmePDQzna0rRpOTfyLCMS7Uuc+1X+/qPAWJHjOot D7eEqQhJ81wOts6MLUdllmktAHT12qXrh3VzwdKnZNoA5l0QOYm3xjoboljFSIwiV3jOPFZQ3JD yAaqjrY+tn3clS2AHEAnNk98iCe/m8JFJ X-Received: by 2002:a05:6512:3da1:b0:48f:cad0:195a with SMTP id k33-20020a0565123da100b0048fcad0195amr8216621lfv.397.1661225182872; Mon, 22 Aug 2022 20:26:22 -0700 (PDT) X-Google-Smtp-Source: AA6agR55MSyAmDLqBjqHOW83UAhKzh0tkGTTtK0wachpRMZqDshvAU8vueQ7o/1jGEPMGTB99+0rwWB+wllsgtNyu24= X-Received: by 2002:a05:6512:3da1:b0:48f:cad0:195a with SMTP id k33-20020a0565123da100b0048fcad0195amr8216608lfv.397.1661225182502; Mon, 22 Aug 2022 20:26:22 -0700 (PDT) MIME-Version: 1.0 References: <20e92551-a639-ec13-3d9c-13bb215422e1@intel.com> <9b6292f3-9bd5-ecd8-5e42-cd5d12f036e7@oracle.com> <22e0236f-b556-c6a8-0043-b39b02928fd6@intel.com> <892b39d6-85f8-bff5-030d-e21288975572@oracle.com> <52a47bc7-bf26-b8f9-257f-7dc5cea66d23@intel.com> <20220817045406-mutt-send-email-mst@kernel.org> <20220817053821-mutt-send-email-mst@kernel.org> <449c2fb2-3920-7bf9-8c5c-a68456dfea76@intel.com> <20220817063450-mutt-send-email-mst@kernel.org> <54aa5a5c-69e2-d372-3e0c-b87f595d213c@redhat.com> <4678fc51-a402-d3ea-e875-6eba175933ba@oracle.com> In-Reply-To: From: Jason Wang Date: Tue, 23 Aug 2022 11:26:11 +0800 Message-ID: Subject: Re: [PATCH 2/2] vDPA: conditionally read fields in virtio-net dev To: "Zhu, Lingshan" Cc: Si-Wei Liu , "Michael S. Tsirkin" , virtualization , netdev , kvm , Parav Pandit , Yongji Xie , "Dawar, Gautam" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On Mon, Aug 22, 2022 at 1:08 PM Zhu, Lingshan wrot= e: > > > > On 8/20/2022 4:55 PM, Si-Wei Liu wrote: > > > > > > On 8/18/2022 5:42 PM, Jason Wang wrote: > >> On Fri, Aug 19, 2022 at 7:20 AM Si-Wei Liu > >> wrote: > >>> > >>> > >>> On 8/17/2022 9:15 PM, Jason Wang wrote: > >>>> =E5=9C=A8 2022/8/17 18:37, Michael S. Tsirkin =E5=86=99=E9=81=93: > >>>>> On Wed, Aug 17, 2022 at 05:43:22PM +0800, Zhu, Lingshan wrote: > >>>>>> On 8/17/2022 5:39 PM, Michael S. Tsirkin wrote: > >>>>>>> On Wed, Aug 17, 2022 at 05:13:59PM +0800, Zhu, Lingshan wrote: > >>>>>>>> On 8/17/2022 4:55 PM, Michael S. Tsirkin wrote: > >>>>>>>>> On Wed, Aug 17, 2022 at 10:14:26AM +0800, Zhu, Lingshan wrote: > >>>>>>>>>> Yes it is a little messy, and we can not check _F_VERSION_1 > >>>>>>>>>> because of > >>>>>>>>>> transitional devices, so maybe this is the best we can do for > >>>>>>>>>> now > >>>>>>>>> I think vhost generally needs an API to declare config space > >>>>>>>>> endian-ness > >>>>>>>>> to kernel. vdpa can reuse that too then. > >>>>>>>> Yes, I remember you have mentioned some IOCTL to set the > >>>>>>>> endian-ness, > >>>>>>>> for vDPA, I think only the vendor driver knows the endian, > >>>>>>>> so we may need a new function vdpa_ops->get_endian(). > >>>>>>>> In the last thread, we say maybe it's better to add a comment fo= r > >>>>>>>> now. > >>>>>>>> But if you think we should add a vdpa_ops->get_endian(), I can > >>>>>>>> work > >>>>>>>> on it for sure! > >>>>>>>> > >>>>>>>> Thanks > >>>>>>>> Zhu Lingshan > >>>>>>> I think QEMU has to set endian-ness. No one else knows. > >>>>>> Yes, for SW based vhost it is true. But for HW vDPA, only > >>>>>> the device & driver knows the endian, I think we can not > >>>>>> "set" a hardware's endian. > >>>>> QEMU knows the guest endian-ness and it knows that > >>>>> device is accessed through the legacy interface. > >>>>> It can accordingly send endian-ness to the kernel and > >>>>> kernel can propagate it to the driver. > >>>> > >>>> I wonder if we can simply force LE and then Qemu can do the endian > >>>> conversion? > >>> convert from LE for config space fields only, or QEMU has to forceful= ly > >>> mediate and covert endianness for all device memory access including > >>> even the datapath (fields in descriptor and avail/used rings)? > >> Former. Actually, I want to force modern devices for vDPA when > >> developing the vDPA framework. But then we see requirements for > >> transitional or even legacy (e.g the Ali ENI parent). So it > >> complicates things a lot. > >> > >> I think several ideas has been proposed: > >> > >> 1) Your proposal of having a vDPA specific way for > >> modern/transitional/legacy awareness. This seems very clean since each > >> transport should have the ability to do that but it still requires > >> some kind of mediation for the case e.g running BE legacy guest on LE > >> host. > > In theory it seems like so, though practically I wonder if we can just > > forbid BE legacy driver from running on modern LE host. For those who > > care about legacy BE guest, they mostly like could and should talk to > > vendor to get native BE support to achieve hardware acceleration, The problem is the hardware still needs a way to know the endian of the gue= st? > > few > > of them would count on QEMU in mediating or emulating the datapath > > (otherwise I don't see the benefit of adopting vDPA?). I still feel > > that not every hardware vendor has to offer backward compatibility > > (transitional device) with legacy interface/behavior (BE being just > > one), Probably, I agree it is a corner case, and dealing with transitional device for the following setups is very challenge for hardware: - driver without IOMMU_PLATFORM support, (requiring device to send translated request which have security implications) - BE legacy guest on LE host, (requiring device to have a way to know the endian) - device specific requirement (e.g modern virtio-net mandate minimal header length to contain mrg_rxbuf even if the device doesn't offer it) It is not obvious for the hardware vendor, so we may end up defecting in the implementation. Dealing with compatibility for the transitional devices is kind of a nightmare which there's no way for the spec to rule the behavior of legacy devices. > > this is unlike the situation on software virtio device, which > > has legacy support since day one. I think we ever discussed it before: > > for those vDPA vendors who don't offer legacy guest support, maybe we > > should mandate some feature for e.g. VERSION_1, as these devices > > really don't offer functionality of the opposite side (!VERSION_1) > > during negotiation. I've tried something similar here (a global mandatory instead of per device= ). https://lkml.org/lkml/2021/6/4/26 But for some reason, it is not applied by Michael. It would be a great relief if we support modern devices only. Maybe it's time to revisit this idea then we can introduce new backend features and then we can mandate VERSION_1 > > > > Having it said, perhaps we should also allow vendor device to > > implement only partial support for legacy. We can define "reversed" > > backend feature to denote some part of the legacy > > interface/functionality not getting implemented by device. For > > instance, VHOST_BACKEND_F_NO_BE_VRING, VHOST_BACKEND_F_NO_BE_CONFIG, > > VHOST_BACKEND_F_NO_ALIGNED_VRING, > > VHOST_BACKEND_NET_F_NO_WRITEABLE_MAC, and et al. Not all of these > > missing features for legacy would be easy for QEMU to make up for, so > > QEMU can selectively emulate those at its best when necessary and > > applicable. In other word, this design shouldn't prevent QEMU from > > making up for vendor device's partial legacy support. This looks too heavyweight since it tries to provide compatibility for legacy drivers. Considering we've introduced modern devices for 5+ years, I'd rather: - Qemu to mediate the config space stuffs - Shadow virtqueue to mediate the datapath (AF_XDP told us shadow ring can perform very well if we do zero-copy). > > > >> > >> 2) Michael suggests using VHOST_SET_VRING_ENDIAN where it means we > >> need a new config ops for vDPA bus, but it doesn't solve the issue for > >> config space (at least from its name). We probably need a new ioctl > >> for both vring and config space. > > Yep adding a new ioctl makes things better, but I think the key is not > > the new ioctl. It's whether or not we should enforce every vDPA vendor > > driver to implement all transitional interfaces to be spec compliant. I think the answer is no since the spec allows transitional device. And we know things will be greatly simplified if vDPA support non transitional device only. So we can change the question to: 1) do we need (or is it too late) to enforce non transitional device? 2) if yes, can transitional device be mediate in an efficient way? For 1), it's probably too late but we can invent new vDPA features as you suggest to be non transitional. Then we can: 1.1) extend the netlink API to provision non-transitonal device 1.2) work on the non-transtional device in the future 1.3) presenting transitional device via mediation The previous transitional vDPA work as is, it's probably too late to fix all the issue we suffer. For 2), the key part is the datapath mediation, we can use shadow virtqueue= . > > If we allow them to reject the VHOST_SET_VRING_ENDIAN or > > VHOST_SET_CONFIG_ENDIAN call, what could we do? We would still end up > > with same situation of either fail the guest, or trying to > > mediate/emulate, right? > > > > Not to mention VHOST_SET_VRING_ENDIAN is rarely supported by vhost > > today - few distro kernel has CONFIG_VHOST_CROSS_ENDIAN_LEGACY enabled > > and QEMU just ignores the result. vhost doesn't necessarily depend on > > it to determine endianness it looks. > I would like to suggest to add two new config ops get/set_vq_endian() > and get/set_config_endian() for vDPA. This is used to: > a) support VHOST_GET/SET_VRING_ENDIAN as MST suggested, and add > VHOST_SET/GET_CONFIG_ENDIAN for vhost_vdpa. > If the device has not implemented interface to set its endianess, then > no matter success or failure of SET_ENDIAN, QEMU knows the endian-ness > anyway. How can Qemu know the endian in this way? And if it can, there's no need for the new API? > In this case, if the device endianess does not match the guest, > there needs a mediation layer or fail. > b) ops->get_config_endian() can always tell the endian-ness of the > device config space after the vendor driver probing the device. So we > can use this ops->get_config_endian() for > MTU, MAC and other fields handling in vdpa_dev_net_config_fill() and we > don't need to set_features in vdpa_get_config_unlocked(), so no race > conditions. > Every time ops->get_config() returned, we can tell the endian by > ops-config_>get_endian(), we don't need set_features(xxx, 0) if features > negotiation not done. > > The question is: Do we need two pairs of ioctls for both vq and config > space? Can config space endian-ness differ from the vqs? > c) do we need a new netlink attr telling the endian-ness to user space? Generally, I'm not sure this is a good design consider it provides neither: Compatibility with the virtio spec nor Compatibility with the existing vhost API (VHOST_SET_VRING_ENDIAN) Thanks > > Thanks, > Zhu Lingshan > > > >> > >> or > >> > >> 3) revisit the idea of forcing modern only device which may simplify > >> things a lot > > I am not actually against forcing modern only config space, given that > > it's not hard for either QEMU or individual driver to mediate or > > emulate, and for the most part it's not conflict with the goal of > > offload or acceleration with vDPA. But forcing LE ring layout IMO > > would just kill off the potential of a very good use case. Currently > > for our use case the priority for supporting 0.9.5 guest with vDPA is > > slightly lower compared to live migration, but it is still in our TODO > > list. > > > > Thanks, > > -Siwei > > > >> > >> which way should we go? > >> > >>> I hope > >>> it's not the latter, otherwise it loses the point to use vDPA for > >>> datapath acceleration. > >>> > >>> Even if its the former, it's a little weird for vendor device to > >>> implement a LE config space with BE ring layout, although still > >>> possible... > >> Right. > >> > >> Thanks > >> > >>> -Siwei > >>>> Thanks > >>>> > >>>> > >>>>>> So if you think we should add a vdpa_ops->get_endian(), > >>>>>> I will drop these comments in the next version of > >>>>>> series, and work on a new patch for get_endian(). > >>>>>> > >>>>>> Thanks, > >>>>>> Zhu Lingshan > >>>>> Guests don't get endian-ness from devices so this seems pointless. > >>>>> > > > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0E6B2C28D13 for ; Tue, 23 Aug 2022 03:26:32 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 8203A60AD3; Tue, 23 Aug 2022 03:26:32 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 8203A60AD3 Authentication-Results: smtp3.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=QHWjoDm3 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9AZLBTMr3yyb; Tue, 23 Aug 2022 03:26:31 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id 95A27605A0; Tue, 23 Aug 2022 03:26:30 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 95A27605A0 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 66F97C0032; Tue, 23 Aug 2022 03:26:30 +0000 (UTC) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id C9AEAC002D for ; Tue, 23 Aug 2022 03:26:28 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 969D560AD3 for ; Tue, 23 Aug 2022 03:26:28 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 969D560AD3 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0DTrQKIusbF2 for ; Tue, 23 Aug 2022 03:26:27 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org E3B13605A0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id E3B13605A0 for ; Tue, 23 Aug 2022 03:26:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1661225185; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MlOaDsdp4+hX3qQ6KtrYrT2iwnuhuhg5bwgVYd5K/i8=; b=QHWjoDm3iliPtoaT+/IQjJdDtJr/IRlxnA+phc54pG+RUexH6iSSXMMOzDacDD0Pf6u23z 7DuwPGTpsWHrlCCbIrZ/OyZwrPJdj4VLBaGxONHvfrhOKnL09D0f/stbcUUi+/8PXkJHRJ 1FgUUwXnLzwQ7crFz/nI12V3knRUfrE= Received: from mail-lf1-f69.google.com (mail-lf1-f69.google.com [209.85.167.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-55-h-NKE3v7N520wGcWDdLdUw-1; Mon, 22 Aug 2022 23:26:24 -0400 X-MC-Unique: h-NKE3v7N520wGcWDdLdUw-1 Received: by mail-lf1-f69.google.com with SMTP id o13-20020ac24bcd000000b00492df6d545eso1872777lfq.20 for ; Mon, 22 Aug 2022 20:26:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc; bh=MlOaDsdp4+hX3qQ6KtrYrT2iwnuhuhg5bwgVYd5K/i8=; b=LF9+XvG4UZicihWV1wsdLShLyMIhQDYTQfosN6L86M9zQoxDEIqiGXaVa8hGxEWZ7j LUjJl3c5S1JvXuxJYSSDKPVQzibBK7gjBBoNf+fG8WulaDAP1OnX/9+r2MEdXBIsmLF5 h6cbwpLZO8JIOWLn8e3sPNq/le8QTJnQlVkCuZzsHwAMfDAYVpQabEKzkvbOKDd7JJ/l 63P/Vk+LbsIz8gbrqOFskBb0pnFBhdlj4aCBpMEffNywD4KizvmbjG35k6xAB0jpsUYB 6QPEP58tUyIN551D1DLpsljbQusU56wfxSUkJIv3Nf8mOvrupJA8dPNoyQ2UVGSkG7yJ t5jA== X-Gm-Message-State: ACgBeo0Rvr8JgLMv+kaO1TUhhp+jIUygxlsbeVU2ZACp86aH2jsNec4X +MiPK9++YeDJ7OrCRoOId136pDvxN2WqIy6kt/JC1vK/AlY/pvzGB42bX1wCdmhB3Pupe8gzP/9 zusPrfGeou4mVnVHccec75fg3XsDNzAYvJ28AX33Mx6MC+In2JaM2xrwr8g== X-Received: by 2002:a05:6512:3da1:b0:48f:cad0:195a with SMTP id k33-20020a0565123da100b0048fcad0195amr8216619lfv.397.1661225182795; Mon, 22 Aug 2022 20:26:22 -0700 (PDT) X-Google-Smtp-Source: AA6agR55MSyAmDLqBjqHOW83UAhKzh0tkGTTtK0wachpRMZqDshvAU8vueQ7o/1jGEPMGTB99+0rwWB+wllsgtNyu24= X-Received: by 2002:a05:6512:3da1:b0:48f:cad0:195a with SMTP id k33-20020a0565123da100b0048fcad0195amr8216608lfv.397.1661225182502; Mon, 22 Aug 2022 20:26:22 -0700 (PDT) MIME-Version: 1.0 References: <20e92551-a639-ec13-3d9c-13bb215422e1@intel.com> <9b6292f3-9bd5-ecd8-5e42-cd5d12f036e7@oracle.com> <22e0236f-b556-c6a8-0043-b39b02928fd6@intel.com> <892b39d6-85f8-bff5-030d-e21288975572@oracle.com> <52a47bc7-bf26-b8f9-257f-7dc5cea66d23@intel.com> <20220817045406-mutt-send-email-mst@kernel.org> <20220817053821-mutt-send-email-mst@kernel.org> <449c2fb2-3920-7bf9-8c5c-a68456dfea76@intel.com> <20220817063450-mutt-send-email-mst@kernel.org> <54aa5a5c-69e2-d372-3e0c-b87f595d213c@redhat.com> <4678fc51-a402-d3ea-e875-6eba175933ba@oracle.com> In-Reply-To: From: Jason Wang Date: Tue, 23 Aug 2022 11:26:11 +0800 Message-ID: Subject: Re: [PATCH 2/2] vDPA: conditionally read fields in virtio-net dev To: "Zhu, Lingshan" X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: kvm , "Michael S. Tsirkin" , netdev , virtualization , Yongji Xie , "Dawar, Gautam" X-BeenThere: virtualization@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Linux virtualization List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: virtualization-bounces@lists.linux-foundation.org Sender: "Virtualization" T24gTW9uLCBBdWcgMjIsIDIwMjIgYXQgMTowOCBQTSBaaHUsIExpbmdzaGFuIDxsaW5nc2hhbi56 aHVAaW50ZWwuY29tPiB3cm90ZToKPgo+Cj4KPiBPbiA4LzIwLzIwMjIgNDo1NSBQTSwgU2ktV2Vp IExpdSB3cm90ZToKPiA+Cj4gPgo+ID4gT24gOC8xOC8yMDIyIDU6NDIgUE0sIEphc29uIFdhbmcg d3JvdGU6Cj4gPj4gT24gRnJpLCBBdWcgMTksIDIwMjIgYXQgNzoyMCBBTSBTaS1XZWkgTGl1IDxz aS13ZWkubGl1QG9yYWNsZS5jb20+Cj4gPj4gd3JvdGU6Cj4gPj4+Cj4gPj4+Cj4gPj4+IE9uIDgv MTcvMjAyMiA5OjE1IFBNLCBKYXNvbiBXYW5nIHdyb3RlOgo+ID4+Pj4g5ZyoIDIwMjIvOC8xNyAx ODozNywgTWljaGFlbCBTLiBUc2lya2luIOWGmemBkzoKPiA+Pj4+PiBPbiBXZWQsIEF1ZyAxNywg MjAyMiBhdCAwNTo0MzoyMlBNICswODAwLCBaaHUsIExpbmdzaGFuIHdyb3RlOgo+ID4+Pj4+PiBP biA4LzE3LzIwMjIgNTozOSBQTSwgTWljaGFlbCBTLiBUc2lya2luIHdyb3RlOgo+ID4+Pj4+Pj4g T24gV2VkLCBBdWcgMTcsIDIwMjIgYXQgMDU6MTM6NTlQTSArMDgwMCwgWmh1LCBMaW5nc2hhbiB3 cm90ZToKPiA+Pj4+Pj4+PiBPbiA4LzE3LzIwMjIgNDo1NSBQTSwgTWljaGFlbCBTLiBUc2lya2lu IHdyb3RlOgo+ID4+Pj4+Pj4+PiBPbiBXZWQsIEF1ZyAxNywgMjAyMiBhdCAxMDoxNDoyNkFNICsw ODAwLCBaaHUsIExpbmdzaGFuIHdyb3RlOgo+ID4+Pj4+Pj4+Pj4gWWVzIGl0IGlzIGEgbGl0dGxl IG1lc3N5LCBhbmQgd2UgY2FuIG5vdCBjaGVjayBfRl9WRVJTSU9OXzEKPiA+Pj4+Pj4+Pj4+IGJl Y2F1c2Ugb2YKPiA+Pj4+Pj4+Pj4+IHRyYW5zaXRpb25hbCBkZXZpY2VzLCBzbyBtYXliZSB0aGlz IGlzIHRoZSBiZXN0IHdlIGNhbiBkbyBmb3IKPiA+Pj4+Pj4+Pj4+IG5vdwo+ID4+Pj4+Pj4+PiBJ IHRoaW5rIHZob3N0IGdlbmVyYWxseSBuZWVkcyBhbiBBUEkgdG8gZGVjbGFyZSBjb25maWcgc3Bh Y2UKPiA+Pj4+Pj4+Pj4gZW5kaWFuLW5lc3MKPiA+Pj4+Pj4+Pj4gdG8ga2VybmVsLiB2ZHBhIGNh biByZXVzZSB0aGF0IHRvbyB0aGVuLgo+ID4+Pj4+Pj4+IFllcywgSSByZW1lbWJlciB5b3UgaGF2 ZSBtZW50aW9uZWQgc29tZSBJT0NUTCB0byBzZXQgdGhlCj4gPj4+Pj4+Pj4gZW5kaWFuLW5lc3Ms Cj4gPj4+Pj4+Pj4gZm9yIHZEUEEsIEkgdGhpbmsgb25seSB0aGUgdmVuZG9yIGRyaXZlciBrbm93 cyB0aGUgZW5kaWFuLAo+ID4+Pj4+Pj4+IHNvIHdlIG1heSBuZWVkIGEgbmV3IGZ1bmN0aW9uIHZk cGFfb3BzLT5nZXRfZW5kaWFuKCkuCj4gPj4+Pj4+Pj4gSW4gdGhlIGxhc3QgdGhyZWFkLCB3ZSBz YXkgbWF5YmUgaXQncyBiZXR0ZXIgdG8gYWRkIGEgY29tbWVudCBmb3IKPiA+Pj4+Pj4+PiBub3cu Cj4gPj4+Pj4+Pj4gQnV0IGlmIHlvdSB0aGluayB3ZSBzaG91bGQgYWRkIGEgdmRwYV9vcHMtPmdl dF9lbmRpYW4oKSwgSSBjYW4KPiA+Pj4+Pj4+PiB3b3JrCj4gPj4+Pj4+Pj4gb24gaXQgZm9yIHN1 cmUhCj4gPj4+Pj4+Pj4KPiA+Pj4+Pj4+PiBUaGFua3MKPiA+Pj4+Pj4+PiBaaHUgTGluZ3NoYW4K PiA+Pj4+Pj4+IEkgdGhpbmsgUUVNVSBoYXMgdG8gc2V0IGVuZGlhbi1uZXNzLiBObyBvbmUgZWxz ZSBrbm93cy4KPiA+Pj4+Pj4gWWVzLCBmb3IgU1cgYmFzZWQgdmhvc3QgaXQgaXMgdHJ1ZS4gQnV0 IGZvciBIVyB2RFBBLCBvbmx5Cj4gPj4+Pj4+IHRoZSBkZXZpY2UgJiBkcml2ZXIga25vd3MgdGhl IGVuZGlhbiwgSSB0aGluayB3ZSBjYW4gbm90Cj4gPj4+Pj4+ICJzZXQiIGEgaGFyZHdhcmUncyBl bmRpYW4uCj4gPj4+Pj4gUUVNVSBrbm93cyB0aGUgZ3Vlc3QgZW5kaWFuLW5lc3MgYW5kIGl0IGtu b3dzIHRoYXQKPiA+Pj4+PiBkZXZpY2UgaXMgYWNjZXNzZWQgdGhyb3VnaCB0aGUgbGVnYWN5IGlu dGVyZmFjZS4KPiA+Pj4+PiBJdCBjYW4gYWNjb3JkaW5nbHkgc2VuZCBlbmRpYW4tbmVzcyB0byB0 aGUga2VybmVsIGFuZAo+ID4+Pj4+IGtlcm5lbCBjYW4gcHJvcGFnYXRlIGl0IHRvIHRoZSBkcml2 ZXIuCj4gPj4+Pgo+ID4+Pj4gSSB3b25kZXIgaWYgd2UgY2FuIHNpbXBseSBmb3JjZSBMRSBhbmQg dGhlbiBRZW11IGNhbiBkbyB0aGUgZW5kaWFuCj4gPj4+PiBjb252ZXJzaW9uPwo+ID4+PiBjb252 ZXJ0IGZyb20gTEUgZm9yIGNvbmZpZyBzcGFjZSBmaWVsZHMgb25seSwgb3IgUUVNVSBoYXMgdG8g Zm9yY2VmdWxseQo+ID4+PiBtZWRpYXRlIGFuZCBjb3ZlcnQgZW5kaWFubmVzcyBmb3IgYWxsIGRl dmljZSBtZW1vcnkgYWNjZXNzIGluY2x1ZGluZwo+ID4+PiBldmVuIHRoZSBkYXRhcGF0aCAoZmll bGRzIGluIGRlc2NyaXB0b3IgYW5kIGF2YWlsL3VzZWQgcmluZ3MpPwo+ID4+IEZvcm1lci4gQWN0 dWFsbHksIEkgd2FudCB0byBmb3JjZSBtb2Rlcm4gZGV2aWNlcyBmb3IgdkRQQSB3aGVuCj4gPj4g ZGV2ZWxvcGluZyB0aGUgdkRQQSBmcmFtZXdvcmsuIEJ1dCB0aGVuIHdlIHNlZSByZXF1aXJlbWVu dHMgZm9yCj4gPj4gdHJhbnNpdGlvbmFsIG9yIGV2ZW4gbGVnYWN5IChlLmcgdGhlIEFsaSBFTkkg cGFyZW50KS4gU28gaXQKPiA+PiBjb21wbGljYXRlcyB0aGluZ3MgYSBsb3QuCj4gPj4KPiA+PiBJ IHRoaW5rIHNldmVyYWwgaWRlYXMgaGFzIGJlZW4gcHJvcG9zZWQ6Cj4gPj4KPiA+PiAxKSBZb3Vy IHByb3Bvc2FsIG9mIGhhdmluZyBhIHZEUEEgc3BlY2lmaWMgd2F5IGZvcgo+ID4+IG1vZGVybi90 cmFuc2l0aW9uYWwvbGVnYWN5IGF3YXJlbmVzcy4gVGhpcyBzZWVtcyB2ZXJ5IGNsZWFuIHNpbmNl IGVhY2gKPiA+PiB0cmFuc3BvcnQgc2hvdWxkIGhhdmUgdGhlIGFiaWxpdHkgdG8gZG8gdGhhdCBi dXQgaXQgc3RpbGwgcmVxdWlyZXMKPiA+PiBzb21lIGtpbmQgb2YgbWVkaWF0aW9uIGZvciB0aGUg Y2FzZSBlLmcgcnVubmluZyBCRSBsZWdhY3kgZ3Vlc3Qgb24gTEUKPiA+PiBob3N0Lgo+ID4gSW4g dGhlb3J5IGl0IHNlZW1zIGxpa2Ugc28sIHRob3VnaCBwcmFjdGljYWxseSBJIHdvbmRlciBpZiB3 ZSBjYW4ganVzdAo+ID4gZm9yYmlkIEJFIGxlZ2FjeSBkcml2ZXIgZnJvbSBydW5uaW5nIG9uIG1v ZGVybiBMRSBob3N0LiBGb3IgdGhvc2Ugd2hvCj4gPiBjYXJlIGFib3V0IGxlZ2FjeSBCRSBndWVz dCwgdGhleSBtb3N0bHkgbGlrZSBjb3VsZCBhbmQgc2hvdWxkIHRhbGsgdG8KPiA+IHZlbmRvciB0 byBnZXQgbmF0aXZlIEJFIHN1cHBvcnQgdG8gYWNoaWV2ZSBoYXJkd2FyZSBhY2NlbGVyYXRpb24s CgpUaGUgcHJvYmxlbSBpcyB0aGUgaGFyZHdhcmUgc3RpbGwgbmVlZHMgYSB3YXkgdG8ga25vdyB0 aGUgZW5kaWFuIG9mIHRoZSBndWVzdD8KCj4gPiBmZXcKPiA+IG9mIHRoZW0gd291bGQgY291bnQg b24gUUVNVSBpbiBtZWRpYXRpbmcgb3IgZW11bGF0aW5nIHRoZSBkYXRhcGF0aAo+ID4gKG90aGVy d2lzZSBJIGRvbid0IHNlZSB0aGUgYmVuZWZpdCBvZiBhZG9wdGluZyB2RFBBPykuIEkgc3RpbGwg ZmVlbAo+ID4gdGhhdCBub3QgZXZlcnkgaGFyZHdhcmUgdmVuZG9yIGhhcyB0byBvZmZlciBiYWNr d2FyZCBjb21wYXRpYmlsaXR5Cj4gPiAodHJhbnNpdGlvbmFsIGRldmljZSkgd2l0aCBsZWdhY3kg aW50ZXJmYWNlL2JlaGF2aW9yIChCRSBiZWluZyBqdXN0Cj4gPiBvbmUpLAoKUHJvYmFibHksIEkg YWdyZWUgaXQgaXMgYSBjb3JuZXIgY2FzZSwgYW5kIGRlYWxpbmcgd2l0aCB0cmFuc2l0aW9uYWwK ZGV2aWNlIGZvciB0aGUgZm9sbG93aW5nIHNldHVwcyBpcyB2ZXJ5IGNoYWxsZW5nZSBmb3IgaGFy ZHdhcmU6CgotIGRyaXZlciB3aXRob3V0IElPTU1VX1BMQVRGT1JNIHN1cHBvcnQsIChyZXF1aXJp bmcgZGV2aWNlIHRvIHNlbmQKdHJhbnNsYXRlZCByZXF1ZXN0IHdoaWNoIGhhdmUgc2VjdXJpdHkg aW1wbGljYXRpb25zKQotIEJFIGxlZ2FjeSBndWVzdCBvbiBMRSBob3N0LCAocmVxdWlyaW5nIGRl dmljZSB0byBoYXZlIGEgd2F5IHRvIGtub3cKdGhlIGVuZGlhbikKLSBkZXZpY2Ugc3BlY2lmaWMg cmVxdWlyZW1lbnQgKGUuZyBtb2Rlcm4gdmlydGlvLW5ldCBtYW5kYXRlIG1pbmltYWwKaGVhZGVy IGxlbmd0aCB0byBjb250YWluIG1yZ19yeGJ1ZiBldmVuIGlmIHRoZSBkZXZpY2UgZG9lc24ndCBv ZmZlcgppdCkKCkl0IGlzIG5vdCBvYnZpb3VzIGZvciB0aGUgaGFyZHdhcmUgdmVuZG9yLCBzbyB3 ZSBtYXkgZW5kIHVwIGRlZmVjdGluZwppbiB0aGUgaW1wbGVtZW50YXRpb24uIERlYWxpbmcgd2l0 aCBjb21wYXRpYmlsaXR5IGZvciB0aGUgdHJhbnNpdGlvbmFsCmRldmljZXMgaXMga2luZCBvZiBh IG5pZ2h0bWFyZSB3aGljaCB0aGVyZSdzIG5vIHdheSBmb3IgdGhlIHNwZWMgdG8KcnVsZSB0aGUg YmVoYXZpb3Igb2YgbGVnYWN5IGRldmljZXMuCgo+ID4gIHRoaXMgaXMgdW5saWtlIHRoZSBzaXR1 YXRpb24gb24gc29mdHdhcmUgdmlydGlvIGRldmljZSwgd2hpY2gKPiA+IGhhcyBsZWdhY3kgc3Vw cG9ydCBzaW5jZSBkYXkgb25lLiBJIHRoaW5rIHdlIGV2ZXIgZGlzY3Vzc2VkIGl0IGJlZm9yZToK PiA+IGZvciB0aG9zZSB2RFBBIHZlbmRvcnMgd2hvIGRvbid0IG9mZmVyIGxlZ2FjeSBndWVzdCBz dXBwb3J0LCBtYXliZSB3ZQo+ID4gc2hvdWxkIG1hbmRhdGUgc29tZSBmZWF0dXJlIGZvciBlLmcu IFZFUlNJT05fMSwgYXMgdGhlc2UgZGV2aWNlcwo+ID4gcmVhbGx5IGRvbid0IG9mZmVyIGZ1bmN0 aW9uYWxpdHkgb2YgdGhlIG9wcG9zaXRlIHNpZGUgKCFWRVJTSU9OXzEpCj4gPiBkdXJpbmcgbmVn b3RpYXRpb24uCgpJJ3ZlIHRyaWVkIHNvbWV0aGluZyBzaW1pbGFyIGhlcmUgKGEgZ2xvYmFsIG1h bmRhdG9yeSBpbnN0ZWFkIG9mIHBlciBkZXZpY2UpLgoKaHR0cHM6Ly9sa21sLm9yZy9sa21sLzIw MjEvNi80LzI2CgpCdXQgZm9yIHNvbWUgcmVhc29uLCBpdCBpcyBub3QgYXBwbGllZCBieSBNaWNo YWVsLiBJdCB3b3VsZCBiZSBhIGdyZWF0CnJlbGllZiBpZiB3ZSBzdXBwb3J0IG1vZGVybiBkZXZp Y2VzIG9ubHkuIE1heWJlIGl0J3MgdGltZSB0byByZXZpc2l0CnRoaXMgaWRlYSB0aGVuIHdlIGNh biBpbnRyb2R1Y2UgbmV3IGJhY2tlbmQgZmVhdHVyZXMgYW5kIHRoZW4gd2UgY2FuCm1hbmRhdGUg VkVSU0lPTl8xCgo+ID4KPiA+IEhhdmluZyBpdCBzYWlkLCBwZXJoYXBzIHdlIHNob3VsZCBhbHNv IGFsbG93IHZlbmRvciBkZXZpY2UgdG8KPiA+IGltcGxlbWVudCBvbmx5IHBhcnRpYWwgc3VwcG9y dCBmb3IgbGVnYWN5LiBXZSBjYW4gZGVmaW5lICJyZXZlcnNlZCIKPiA+IGJhY2tlbmQgZmVhdHVy ZSB0byBkZW5vdGUgc29tZSBwYXJ0IG9mIHRoZSBsZWdhY3kKPiA+IGludGVyZmFjZS9mdW5jdGlv bmFsaXR5IG5vdCBnZXR0aW5nIGltcGxlbWVudGVkIGJ5IGRldmljZS4gRm9yCj4gPiBpbnN0YW5j ZSwgVkhPU1RfQkFDS0VORF9GX05PX0JFX1ZSSU5HLCBWSE9TVF9CQUNLRU5EX0ZfTk9fQkVfQ09O RklHLAo+ID4gVkhPU1RfQkFDS0VORF9GX05PX0FMSUdORURfVlJJTkcsCj4gPiBWSE9TVF9CQUNL RU5EX05FVF9GX05PX1dSSVRFQUJMRV9NQUMsIGFuZCBldCBhbC4gTm90IGFsbCBvZiB0aGVzZQo+ ID4gbWlzc2luZyBmZWF0dXJlcyBmb3IgbGVnYWN5IHdvdWxkIGJlIGVhc3kgZm9yIFFFTVUgdG8g bWFrZSB1cCBmb3IsIHNvCj4gPiBRRU1VIGNhbiBzZWxlY3RpdmVseSBlbXVsYXRlIHRob3NlIGF0 IGl0cyBiZXN0IHdoZW4gbmVjZXNzYXJ5IGFuZAo+ID4gYXBwbGljYWJsZS4gSW4gb3RoZXIgd29y ZCwgdGhpcyBkZXNpZ24gc2hvdWxkbid0IHByZXZlbnQgUUVNVSBmcm9tCj4gPiBtYWtpbmcgdXAg Zm9yIHZlbmRvciBkZXZpY2UncyBwYXJ0aWFsIGxlZ2FjeSBzdXBwb3J0LgoKVGhpcyBsb29rcyB0 b28gaGVhdnl3ZWlnaHQgc2luY2UgaXQgdHJpZXMgdG8gcHJvdmlkZSBjb21wYXRpYmlsaXR5IGZv cgpsZWdhY3kgZHJpdmVycy4gQ29uc2lkZXJpbmcgd2UndmUgaW50cm9kdWNlZCBtb2Rlcm4gZGV2 aWNlcyBmb3IgNSsKeWVhcnMsIEknZCByYXRoZXI6CgotIFFlbXUgdG8gbWVkaWF0ZSB0aGUgY29u ZmlnIHNwYWNlIHN0dWZmcwotIFNoYWRvdyB2aXJ0cXVldWUgdG8gbWVkaWF0ZSB0aGUgZGF0YXBh dGggKEFGX1hEUCB0b2xkIHVzIHNoYWRvdyByaW5nCmNhbiBwZXJmb3JtIHZlcnkgd2VsbCBpZiB3 ZSBkbyB6ZXJvLWNvcHkpLgoKPiA+Cj4gPj4KPiA+PiAyKSBNaWNoYWVsIHN1Z2dlc3RzIHVzaW5n IFZIT1NUX1NFVF9WUklOR19FTkRJQU4gd2hlcmUgaXQgbWVhbnMgd2UKPiA+PiBuZWVkIGEgbmV3 IGNvbmZpZyBvcHMgZm9yIHZEUEEgYnVzLCBidXQgaXQgZG9lc24ndCBzb2x2ZSB0aGUgaXNzdWUg Zm9yCj4gPj4gY29uZmlnIHNwYWNlIChhdCBsZWFzdCBmcm9tIGl0cyBuYW1lKS4gV2UgcHJvYmFi bHkgbmVlZCBhIG5ldyBpb2N0bAo+ID4+IGZvciBib3RoIHZyaW5nIGFuZCBjb25maWcgc3BhY2Uu Cj4gPiBZZXAgYWRkaW5nIGEgbmV3IGlvY3RsIG1ha2VzIHRoaW5ncyBiZXR0ZXIsIGJ1dCBJIHRo aW5rIHRoZSBrZXkgaXMgbm90Cj4gPiB0aGUgbmV3IGlvY3RsLiBJdCdzIHdoZXRoZXIgb3Igbm90 IHdlIHNob3VsZCBlbmZvcmNlIGV2ZXJ5IHZEUEEgdmVuZG9yCj4gPiBkcml2ZXIgdG8gaW1wbGVt ZW50IGFsbCB0cmFuc2l0aW9uYWwgaW50ZXJmYWNlcyB0byBiZSBzcGVjIGNvbXBsaWFudC4KCkkg dGhpbmsgdGhlIGFuc3dlciBpcyBubyBzaW5jZSB0aGUgc3BlYyBhbGxvd3MgdHJhbnNpdGlvbmFs IGRldmljZS4KQW5kIHdlIGtub3cgdGhpbmdzIHdpbGwgYmUgZ3JlYXRseSBzaW1wbGlmaWVkIGlm IHZEUEEgc3VwcG9ydCBub24KdHJhbnNpdGlvbmFsIGRldmljZSBvbmx5LgoKU28gd2UgY2FuIGNo YW5nZSB0aGUgcXVlc3Rpb24gdG86CgoxKSBkbyB3ZSBuZWVkIChvciBpcyBpdCB0b28gbGF0ZSkg dG8gZW5mb3JjZSBub24gdHJhbnNpdGlvbmFsIGRldmljZT8KMikgaWYgeWVzLCBjYW4gdHJhbnNp dGlvbmFsIGRldmljZSBiZSBtZWRpYXRlIGluIGFuIGVmZmljaWVudCB3YXk/CgpGb3IgMSksIGl0 J3MgcHJvYmFibHkgdG9vIGxhdGUgYnV0IHdlIGNhbiBpbnZlbnQgbmV3IHZEUEEgZmVhdHVyZXMg YXMKeW91IHN1Z2dlc3QgdG8gYmUgbm9uIHRyYW5zaXRpb25hbC4gVGhlbiB3ZSBjYW46CgoxLjEp IGV4dGVuZCB0aGUgbmV0bGluayBBUEkgdG8gcHJvdmlzaW9uIG5vbi10cmFuc2l0b25hbCBkZXZp Y2UKMS4yKSB3b3JrIG9uIHRoZSBub24tdHJhbnN0aW9uYWwgZGV2aWNlIGluIHRoZSBmdXR1cmUK MS4zKSBwcmVzZW50aW5nIHRyYW5zaXRpb25hbCBkZXZpY2UgdmlhIG1lZGlhdGlvbgoKVGhlIHBy ZXZpb3VzIHRyYW5zaXRpb25hbCB2RFBBIHdvcmsgYXMgaXMsIGl0J3MgcHJvYmFibHkgdG9vIGxh dGUgdG8KZml4IGFsbCB0aGUgaXNzdWUgd2Ugc3VmZmVyLgoKRm9yIDIpLCB0aGUga2V5IHBhcnQg aXMgdGhlIGRhdGFwYXRoIG1lZGlhdGlvbiwgd2UgY2FuIHVzZSBzaGFkb3cgdmlydHF1ZXVlLgoK PiA+IElmIHdlIGFsbG93IHRoZW0gdG8gcmVqZWN0IHRoZSBWSE9TVF9TRVRfVlJJTkdfRU5ESUFO ICBvcgo+ID4gVkhPU1RfU0VUX0NPTkZJR19FTkRJQU4gY2FsbCwgd2hhdCBjb3VsZCB3ZSBkbz8g V2Ugd291bGQgc3RpbGwgZW5kIHVwCj4gPiB3aXRoIHNhbWUgc2l0dWF0aW9uIG9mIGVpdGhlciBm YWlsIHRoZSBndWVzdCwgb3IgdHJ5aW5nIHRvCj4gPiBtZWRpYXRlL2VtdWxhdGUsIHJpZ2h0Pwo+ ID4KPiA+IE5vdCB0byBtZW50aW9uIFZIT1NUX1NFVF9WUklOR19FTkRJQU4gaXMgcmFyZWx5IHN1 cHBvcnRlZCBieSB2aG9zdAo+ID4gdG9kYXkgLSBmZXcgZGlzdHJvIGtlcm5lbCBoYXMgQ09ORklH X1ZIT1NUX0NST1NTX0VORElBTl9MRUdBQ1kgZW5hYmxlZAo+ID4gYW5kIFFFTVUganVzdCBpZ25v cmVzIHRoZSByZXN1bHQuIHZob3N0IGRvZXNuJ3QgbmVjZXNzYXJpbHkgZGVwZW5kIG9uCj4gPiBp dCB0byBkZXRlcm1pbmUgZW5kaWFubmVzcyBpdCBsb29rcy4KPiBJIHdvdWxkIGxpa2UgdG8gc3Vn Z2VzdCB0byBhZGQgdHdvIG5ldyBjb25maWcgb3BzIGdldC9zZXRfdnFfZW5kaWFuKCkKPiBhbmQg Z2V0L3NldF9jb25maWdfZW5kaWFuKCkgZm9yIHZEUEEuIFRoaXMgaXMgdXNlZCB0bzoKPiBhKSBz dXBwb3J0IFZIT1NUX0dFVC9TRVRfVlJJTkdfRU5ESUFOIGFzIE1TVCBzdWdnZXN0ZWQsIGFuZCBh ZGQKPiBWSE9TVF9TRVQvR0VUX0NPTkZJR19FTkRJQU4gZm9yIHZob3N0X3ZkcGEuCj4gSWYgdGhl IGRldmljZSBoYXMgbm90IGltcGxlbWVudGVkIGludGVyZmFjZSB0byBzZXQgaXRzIGVuZGlhbmVz cywgdGhlbgo+IG5vIG1hdHRlciBzdWNjZXNzIG9yIGZhaWx1cmUgb2YgU0VUX0VORElBTiwgUUVN VSBrbm93cyB0aGUgZW5kaWFuLW5lc3MKPiBhbnl3YXkuCgpIb3cgY2FuIFFlbXUga25vdyB0aGUg ZW5kaWFuIGluIHRoaXMgd2F5PyBBbmQgaWYgaXQgY2FuLCB0aGVyZSdzIG5vCm5lZWQgZm9yIHRo ZSBuZXcgQVBJPwoKPiBJbiB0aGlzIGNhc2UsIGlmIHRoZSBkZXZpY2UgZW5kaWFuZXNzIGRvZXMg bm90IG1hdGNoIHRoZSBndWVzdCwKPiB0aGVyZSBuZWVkcyBhIG1lZGlhdGlvbiBsYXllciBvciBm YWlsLgo+IGIpIG9wcy0+Z2V0X2NvbmZpZ19lbmRpYW4oKSBjYW4gYWx3YXlzIHRlbGwgdGhlIGVu ZGlhbi1uZXNzIG9mIHRoZQo+IGRldmljZSBjb25maWcgc3BhY2UgYWZ0ZXIgdGhlIHZlbmRvciBk cml2ZXIgcHJvYmluZyB0aGUgZGV2aWNlLiBTbyB3ZQo+IGNhbiB1c2UgdGhpcyBvcHMtPmdldF9j b25maWdfZW5kaWFuKCkgZm9yCj4gTVRVLCBNQUMgYW5kIG90aGVyIGZpZWxkcyBoYW5kbGluZyBp biB2ZHBhX2Rldl9uZXRfY29uZmlnX2ZpbGwoKSBhbmQgd2UKPiBkb24ndCBuZWVkIHRvIHNldF9m ZWF0dXJlcyBpbiB2ZHBhX2dldF9jb25maWdfdW5sb2NrZWQoKSwgc28gbm8gcmFjZQo+IGNvbmRp dGlvbnMuCj4gRXZlcnkgdGltZSBvcHMtPmdldF9jb25maWcoKSByZXR1cm5lZCwgd2UgY2FuIHRl bGwgdGhlIGVuZGlhbiBieQo+IG9wcy1jb25maWdfPmdldF9lbmRpYW4oKSwgd2UgZG9uJ3QgbmVl ZCBzZXRfZmVhdHVyZXMoeHh4LCAwKSBpZiBmZWF0dXJlcwo+IG5lZ290aWF0aW9uIG5vdCBkb25l Lgo+Cj4gVGhlIHF1ZXN0aW9uIGlzOiBEbyB3ZSBuZWVkIHR3byBwYWlycyBvZiBpb2N0bHMgZm9y IGJvdGggdnEgYW5kIGNvbmZpZwo+IHNwYWNlPyBDYW4gY29uZmlnIHNwYWNlIGVuZGlhbi1uZXNz IGRpZmZlciBmcm9tIHRoZSB2cXM/Cj4gYykgZG8gd2UgbmVlZCBhIG5ldyBuZXRsaW5rIGF0dHIg dGVsbGluZyB0aGUgZW5kaWFuLW5lc3MgdG8gdXNlciBzcGFjZT8KCkdlbmVyYWxseSwgSSdtIG5v dCBzdXJlIHRoaXMgaXMgYSBnb29kIGRlc2lnbiBjb25zaWRlciBpdCBwcm92aWRlcyBuZWl0aGVy OgoKQ29tcGF0aWJpbGl0eSB3aXRoIHRoZSB2aXJ0aW8gc3BlYwoKbm9yCgpDb21wYXRpYmlsaXR5 IHdpdGggdGhlIGV4aXN0aW5nIHZob3N0IEFQSSAoVkhPU1RfU0VUX1ZSSU5HX0VORElBTikKClRo YW5rcwoKPgo+IFRoYW5rcywKPiBaaHUgTGluZ3NoYW4KPiA+Cj4gPj4KPiA+PiBvcgo+ID4+Cj4g Pj4gMykgcmV2aXNpdCB0aGUgaWRlYSBvZiBmb3JjaW5nIG1vZGVybiBvbmx5IGRldmljZSB3aGlj aCBtYXkgc2ltcGxpZnkKPiA+PiB0aGluZ3MgYSBsb3QKPiA+IEkgYW0gbm90IGFjdHVhbGx5IGFn YWluc3QgZm9yY2luZyBtb2Rlcm4gb25seSBjb25maWcgc3BhY2UsIGdpdmVuIHRoYXQKPiA+IGl0 J3Mgbm90IGhhcmQgZm9yIGVpdGhlciBRRU1VIG9yIGluZGl2aWR1YWwgZHJpdmVyIHRvIG1lZGlh dGUgb3IKPiA+IGVtdWxhdGUsIGFuZCBmb3IgdGhlIG1vc3QgcGFydCBpdCdzIG5vdCBjb25mbGlj dCB3aXRoIHRoZSBnb2FsIG9mCj4gPiBvZmZsb2FkIG9yIGFjY2VsZXJhdGlvbiB3aXRoIHZEUEEu IEJ1dCBmb3JjaW5nIExFIHJpbmcgbGF5b3V0IElNTwo+ID4gd291bGQganVzdCBraWxsIG9mZiB0 aGUgcG90ZW50aWFsIG9mIGEgdmVyeSBnb29kIHVzZSBjYXNlLiBDdXJyZW50bHkKPiA+IGZvciBv dXIgdXNlIGNhc2UgdGhlIHByaW9yaXR5IGZvciBzdXBwb3J0aW5nIDAuOS41IGd1ZXN0IHdpdGgg dkRQQSBpcwo+ID4gc2xpZ2h0bHkgbG93ZXIgY29tcGFyZWQgdG8gbGl2ZSBtaWdyYXRpb24sIGJ1 dCBpdCBpcyBzdGlsbCBpbiBvdXIgVE9ETwo+ID4gbGlzdC4KPiA+Cj4gPiBUaGFua3MsCj4gPiAt U2l3ZWkKPiA+Cj4gPj4KPiA+PiB3aGljaCB3YXkgc2hvdWxkIHdlIGdvPwo+ID4+Cj4gPj4+IEkg aG9wZQo+ID4+PiBpdCdzIG5vdCB0aGUgbGF0dGVyLCBvdGhlcndpc2UgaXQgbG9zZXMgdGhlIHBv aW50IHRvIHVzZSB2RFBBIGZvcgo+ID4+PiBkYXRhcGF0aCBhY2NlbGVyYXRpb24uCj4gPj4+Cj4g Pj4+IEV2ZW4gaWYgaXRzIHRoZSBmb3JtZXIsIGl0J3MgYSBsaXR0bGUgd2VpcmQgZm9yIHZlbmRv ciBkZXZpY2UgdG8KPiA+Pj4gaW1wbGVtZW50IGEgTEUgY29uZmlnIHNwYWNlIHdpdGggQkUgcmlu ZyBsYXlvdXQsIGFsdGhvdWdoIHN0aWxsCj4gPj4+IHBvc3NpYmxlLi4uCj4gPj4gUmlnaHQuCj4g Pj4KPiA+PiBUaGFua3MKPiA+Pgo+ID4+PiAtU2l3ZWkKPiA+Pj4+IFRoYW5rcwo+ID4+Pj4KPiA+ Pj4+Cj4gPj4+Pj4+IFNvIGlmIHlvdSB0aGluayB3ZSBzaG91bGQgYWRkIGEgdmRwYV9vcHMtPmdl dF9lbmRpYW4oKSwKPiA+Pj4+Pj4gSSB3aWxsIGRyb3AgdGhlc2UgY29tbWVudHMgaW4gdGhlIG5l eHQgdmVyc2lvbiBvZgo+ID4+Pj4+PiBzZXJpZXMsIGFuZCB3b3JrIG9uIGEgbmV3IHBhdGNoIGZv ciBnZXRfZW5kaWFuKCkuCj4gPj4+Pj4+Cj4gPj4+Pj4+IFRoYW5rcywKPiA+Pj4+Pj4gWmh1IExp bmdzaGFuCj4gPj4+Pj4gR3Vlc3RzIGRvbid0IGdldCBlbmRpYW4tbmVzcyBmcm9tIGRldmljZXMg c28gdGhpcyBzZWVtcyBwb2ludGxlc3MuCj4gPj4+Pj4KPiA+Cj4KCl9fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fClZpcnR1YWxpemF0aW9uIG1haWxpbmcgbGlz dApWaXJ0dWFsaXphdGlvbkBsaXN0cy5saW51eC1mb3VuZGF0aW9uLm9yZwpodHRwczovL2xpc3Rz LmxpbnV4Zm91bmRhdGlvbi5vcmcvbWFpbG1hbi9saXN0aW5mby92aXJ0dWFsaXphdGlvbg==