From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17454C433F5 for ; Fri, 17 Dec 2021 01:08:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231609AbhLQBIn (ORCPT ); Thu, 16 Dec 2021 20:08:43 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:58048 "EHLO mx0b-00069f02.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231566AbhLQBIl (ORCPT ); Thu, 16 Dec 2021 20:08:41 -0500 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 1BGLE2p8031682; Fri, 17 Dec 2021 01:08:38 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=message-id : date : subject : to : cc : references : from : in-reply-to : content-type : content-transfer-encoding : mime-version; s=corp-2021-07-09; bh=cr6qc2aB6HFqpdpYp6v+0aRhh2Hg+Bg4KN2ituLWZEc=; b=XomEdt259oynkkHxmC3in/erqStSiBbE2Y95MmcT3annRycrVGIN8C+pMTMXkqge3hAF S05ywyC4YPVSYzmJh58Vz/i9WfnLxdztKgZU+/LHFG6hCvXlCOG5MpACg4s9pCtka1kH ifMr1xh4JcC7D4Kdfj7Zyqdr0hEGwBSaw6USjA6YA/O/R5hjGyWNLuNooNmJaMP+qUkG psYoYCxv64vsBoOCHjru5RzOp3vzj3NbEXY6DSkT5SeGL1SuC8j4AZY7h2bZxfXG7nO5 NOFctyqWXAMEto32dKKpb5pEtU1UYyhQVuOu/6K5nupihIrcKGzm34wxvfVnt5awK3uG Xw== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 3cykmbmgqa-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 17 Dec 2021 01:08:37 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.1.2/8.16.1.2) with SMTP id 1BH15tuQ114925; Fri, 17 Dec 2021 01:08:36 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2041.outbound.protection.outlook.com [104.47.66.41]) by userp3020.oracle.com with ESMTP id 3cvneupx5h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 17 Dec 2021 01:08:36 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IhBwq3S466pc1oqfdJuBUnfFyEu+HBd/oBUnOXTO68r6Y0PERX0n55lUnlkElc1OEoyU6Qn/VaU4QrDM5oxXQlwzDpcBkGWmYuQcbl4EqTmAIVZPBZr0OeXXAVodKFhgpyyMEjjHwI3V51FrESJVW38HNRoILWTgpi/LrfE0KfdVKB6UZpU79HtHEuHzMOroFjbOnRmo0nHqQWU6rd8Dd5HbtMLoFLCeCaAH7RMFAtERRtA35mJgp9AKD6birZfGYq+n/E0aDVHjhJcdhvJMGSE+aStrptej5+ylgEyEKjAAlSMbUJ6HwI/+27K7wdLfiQLZWhNdKe9b50DSRDougA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=cr6qc2aB6HFqpdpYp6v+0aRhh2Hg+Bg4KN2ituLWZEc=; b=nDm5yt3T3t6DY2aKhtjjUEMmtLG+IS+OrWrff7mVWe9TNs0fVNN9m94xVY/1YA81bIEnSuZSILNs6hdX047JhAdkghbJiW/s99AbOeAg921R8EbWBxQR/NhKK9bKR3nc5qI7OHI4uwph/jvbYLNpuFbrvb31fkyej/JNQbFUz4dMBn+2mdbQwAsuSc23JW+WD1LHQhIBrZICQ11ZRn5qaeiUaM3AArDQ8B5liFfmu/R8RfRZ1ZqsrOa9BGHWaeyFKVThr9BaZ/03wMh43VKKJiSSPFavlneDRf/oHeJjIseONQnnicwGO9QaOyfSdeZSJ7jyiDi20K8xPeQ7Qb13pw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cr6qc2aB6HFqpdpYp6v+0aRhh2Hg+Bg4KN2ituLWZEc=; b=ecsVxtbILDUnqKIoNQKvfnyO2bfGyV/lg6K/XzeqpwG6Bx3Lk29Fth2uPGtwqFsJ0NM7SPaKaWYN5hhmvHTO5bACJdR+vm/nCGtT8+N3tDMLx5QDlWYnp5SRqAe6dcMrohq4T2wOC8wcv5lWtqxNYJ7JTJYn8CkESw7zSFE4kO8= Received: from BYAPR10MB3287.namprd10.prod.outlook.com (2603:10b6:a03:15c::11) by SJ0PR10MB4734.namprd10.prod.outlook.com (2603:10b6:a03:2d2::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4778.17; Fri, 17 Dec 2021 01:08:33 +0000 Received: from BYAPR10MB3287.namprd10.prod.outlook.com ([fe80::7c7e:4a5e:24e0:309d]) by BYAPR10MB3287.namprd10.prod.outlook.com ([fe80::7c7e:4a5e:24e0:309d%3]) with mapi id 15.20.4778.018; Fri, 17 Dec 2021 01:08:33 +0000 Message-ID: Date: Thu, 16 Dec 2021 17:08:28 -0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.3.2 Subject: Re: vdpa legacy guest support (was Re: [PATCH] vdpa/mlx5: set_features should allow reset to zero) Content-Language: en-US To: Jason Wang Cc: "Michael S. Tsirkin" , Eli Cohen , linux-kernel , virtualization , netdev References: <3ff5fd23-1db0-2f95-4cf9-711ef403fb62@oracle.com> <20210224000057-mutt-send-email-mst@kernel.org> <52836a63-4e00-ff58-50fb-9f450ce968d7@oracle.com> <20210228163031-mutt-send-email-mst@kernel.org> <2cb51a6d-afa0-7cd1-d6f2-6b153186eaca@redhat.com> <20210302043419-mutt-send-email-mst@kernel.org> <178f8ea7-cebd-0e81-3dc7-10a058d22c07@redhat.com> <20211212042311-mutt-send-email-mst@kernel.org> <20211214000245-mutt-send-email-mst@kernel.org> <4fc43d0f-da9e-ce16-1f26-9f0225239b75@oracle.com> <6eaf672c-cc86-b5bf-5b74-c837affeb6e1@oracle.com> From: Si-Wei Liu Organization: Oracle Corporation In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: SN2PR01CA0069.prod.exchangelabs.com (2603:10b6:800::37) To BYAPR10MB3287.namprd10.prod.outlook.com (2603:10b6:a03:15c::11) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c60938e5-8246-4c06-9c0e-08d9c0f9bf15 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4734:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: NmRCKmxk/qJT6z1qOJHZloaO2KxKTNO2Z+3TRgFsZg/y8V7zk8WigpjOvomAwW+RLlJbY9xpitQ69DK9UKwLgqMBLzSh1i0LNTctc+SuuREiss5Gi7+uiKMAzQUXoRAaZ0/+MIb7xPhbQLwE0ScnK4QAtoNpH1iWi52X/JyS5NmQ61BN/5Pl8o4P7QTn5AkKnZKZDi4FbS4rcelhk7j4FwUV0h7SmvzRMtNSMzTyQp3lKGrcKD9NSbXfEGUbmfd2Q0S9PND1/ezkC7KxNka2egS05QBVHTDbF6sry+ZF4hTqvEnOFoGhU6mzRy2FO9ky6K7Yi7kJhNbvttySNVkgTx+dNKDBRuISb5EEmOJN35v2gvotpiHEaC24JoXFpYSVwL+7HQHweK4MC1wyfSU6zmOk7omGss0VUCB57fG2zmbHiBA0tB6rSaAOiqFfQhkGhpfgbTnURzLgPJ7mP+DRZlmGdl8B/2ZY3Zr/zN898vdE2fiPPWVQYyR4eB0qAq91KHtKHmO9OwuqXS2LtCUUuS0S+CyDPdsfq+kV1wlEkC+u92WWyUEkTisiW20xA+5QZqSbsj9A9F4nte5UqQ4BBmCwhxy3/PdLWiwmHwKWslO1wYTLMVdzVh3ouQtb/iB9F2LlJ5VgDsuD2TTNdCouSpX9e2ojm/6H0sqyJW/tXvAlYmJ1z21X8Piwbm/xi1AyRhb+QBvK0/trX2BidtPPxEVmcOkjjhsdMA+LPf33ErUlIE5Rg5R611RUh21wq1pDaJP3rJD2CaGvGhPdL6FuVSoH1fiqC5sJNe4pOeI6uJvDAbekF+siorIIo31MXbO1 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3287.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(83380400001)(31686004)(8936002)(6916009)(66946007)(6506007)(8676002)(31696002)(66556008)(66476007)(36756003)(38100700002)(186003)(508600001)(36916002)(4326008)(6666004)(53546011)(26005)(966005)(2616005)(2906002)(54906003)(6512007)(86362001)(6486002)(5660300002)(316002)(30864003)(45980500001)(43740500002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cCtaT1Fzelg0ZktCdk1pRGtqZ2owY0lUUGhyeUpoMjNTczR6QW9GQ0RMMVNk?= =?utf-8?B?T3B4SzlHS05DMHduNkE5ZDdkTU54cElrR3pqYW43K24yZTVURk1zMzRtaFFz?= =?utf-8?B?WXltTHVqRVRiY1dVdUx5RmJ1M2ZXN0Z5Rmt0MDY4bVhqaTVvclhGVGdZcDlM?= =?utf-8?B?RDdpNUszbEhuSjJPQUQ3VG1aM3J3ZEdUK3F2MjFhS3dlMFhkakhyK2pkbGVF?= =?utf-8?B?ekdIc1J3MXRLRlRFUlRYMmMzU2RMS2ZkZk5ObTRMRjE1QVRjNmUrVVp5OHBz?= =?utf-8?B?cmkvRkZSbWoxdDZsSi9kMkNFL2RRRW5FUnhUbjhoR3FRZzQ0NGlMaHI2dGIv?= =?utf-8?B?eFhTMURxbHhDVlowRDFZYmVTcFQrOTlQb0d0blAwMWhQUldsSFQwUG1qUHdH?= =?utf-8?B?RGNXYTJneGJBZHQ5NlVTaDJrL0toNmhCVE5DUi9MejJPMUUyWFNBSDFsdFc0?= =?utf-8?B?Z2ErWEtHQlNUNUtwQ0UxQW5MVkpDTFpoazJNcitaRnhmaFcvRTNPSEpNZ2NK?= =?utf-8?B?cVYzYldKazlmRDJKaTNla2tNdmlEK0oxUzBUNHB3bmJDRjduR0NqZlI2V01u?= =?utf-8?B?TmVDbWV2czJGaVc3Lyt0UW9NNGwyRDgySG5aNGVQVW9ISWVvWWsxRHVNeWpF?= =?utf-8?B?b0FVWDhoNUd4S2hSb0NLYlJCNzU1b2VaaXhFOWlDV0V3RkJtSnpidElMazNE?= =?utf-8?B?YlhJSUQ4bXlzSkh2U1MyeTFEUXRmb1lBcitYMkcrMTFhbnpMR3FWb04rU0Rx?= =?utf-8?B?M2ljVEpvWk5SRG5IVEFMWnBjM1hZTmpGQW5aRXV4UXhPZWJVek1TYlZHcnNa?= =?utf-8?B?b3dQNm1pT3JwWFRrazV6OTA1bnl2aDFVR3lweHVJY3doeHhRbEpkbGphY0Jl?= =?utf-8?B?c0VRZXQvVWNFaHp5ZjNyRE5DYmp5ci9GbDRWM0FhbGZaNUdHc0N2SVhOeEox?= =?utf-8?B?Y2xWOVlZUG5XdWh2OVZPa0RQMXNtQkw3RVFMdXhNaFUvblM1QktqZTc3eUVH?= =?utf-8?B?RndvdUNIZ3JjekMyVUZEVFhzeEN5ejM5NnZodGpydVVZLzRqTG54ZjZ4UXJm?= =?utf-8?B?Y2xwZnVWSzRPcEZ5TUVKckxGOGZHdEVuMzFqb2RXMW1LQUZkRHdJN3hzV05K?= =?utf-8?B?aWNOUFlMbGdLVUNSaU9pUEt3aCtQNnNQUWdLcVJaSkhVT1o0TmEvbmlCUzly?= =?utf-8?B?MGRRT0EyU2l2cW9BaVJaQkZxSGp0VzQyMnJpRE1qRk1NU1dPUSt2MnJQb3JM?= =?utf-8?B?RWJiaWw3OWFJU3hITVYzdWNCWlJDTDRadis4cjcyRnoyd1RLSWl5NTZXTlpP?= =?utf-8?B?ZHFWWStUVWkzZ0JXTkhxYnBINXM4L1czWGNGTmtWZ1JxUnB3SWF0SjJhZXRC?= =?utf-8?B?MlpnS0NzUGtrMm9BUytLQURCUTR1Y1ZpUnc2L3hYeVYxVXpLTzMwU0Q2TXJG?= =?utf-8?B?Q04xcS9MeHhyMXNwZHlIMWEwb25ZMVZNUVJlTUVyQmJRczVFekZ1ekZyaVBK?= =?utf-8?B?UmIyZkNsajVIWHV5NUcvK1R5U0ZlMEx0MGdvb3VYaTJKSzJabnlvWHJMTnBy?= =?utf-8?B?dW40aFFmSC8yY01UQWQrRVNVVGlTaVg1U1RKU0xqV2xOWnFpQ2pvSm9kMmVu?= =?utf-8?B?VytmczNKSHE5YkFMQW9hUkdVb1Jvb2d3cldFL2FaeFVieDM0aWpRSWw2RDg5?= =?utf-8?B?R2hkVWR2Q1BKek5rbHZXODB1TDc1cWQxY2tUQ3kxZlhFNldMS0V5LzhZRUJO?= =?utf-8?B?byszVTlKME1JQVUwb1ZzVDM5U1BjVENUM1ExU0cwK3hHOUtSSnJXZjVjSTdk?= =?utf-8?B?V0lEU0ZoZFIydlRRWTd1aDR1RlNkSGdBcldteTBQUU9TeG1XTThuekYyRUFO?= =?utf-8?B?MjZVb2F0OXVpdmtTb3kxS1dzeUpKQ0J4Q2lMNjZDYmpCNlVFRDI1SStuTWtx?= =?utf-8?B?M2JtbDBYZWorL3pkUDJxZUNkOXBPcFUxTlQ3M0U5Wm5SWkJoYlRmUm9ubmpW?= =?utf-8?B?NFN6RU9jRDMxcWJYUnB4WWtoTjQrUHd4dWFBam5ucEY1L1pQak1zWnZFeHd1?= =?utf-8?B?YkNQTDJYNGhlZUFGZXpnZ3BYQU84ZlIxSzd4MHdrSzQ0ZUZWOFM3MjFOa2Zk?= =?utf-8?B?eGI1MmRxeVdVcll2cXZ2RnFaSnRXZEU0dDVxRm1WTnZlY2s5SG5IVkVBR0ti?= =?utf-8?Q?ML4mSK8nDUwG6/CyShPnwPk=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: c60938e5-8246-4c06-9c0e-08d9c0f9bf15 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3287.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Dec 2021 01:08:33.6921 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 1NGo6y/VZmVq9ZycmhPjFguRBFF6ALFTl0B3Pa+9dZmxWGZuqI6aeaj7Du4l7VBZQjuEaM6Vd1RrvTc69q9jyw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4734 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10200 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0 malwarescore=0 bulkscore=0 spamscore=0 mlxlogscore=999 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000 definitions=main-2112170005 X-Proofpoint-ORIG-GUID: MiqO2UHQpLkUG4vUAhcUTzqkMAWsis5C X-Proofpoint-GUID: MiqO2UHQpLkUG4vUAhcUTzqkMAWsis5C Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 12/15/2021 7:43 PM, Jason Wang wrote: > On Thu, Dec 16, 2021 at 4:52 AM Si-Wei Liu wrote: >> >> >> On 12/14/2021 6:06 PM, Jason Wang wrote: >>> On Wed, Dec 15, 2021 at 9:05 AM Si-Wei Liu wrote: >>>> >>>> On 12/13/2021 9:06 PM, Michael S. Tsirkin wrote: >>>>> On Mon, Dec 13, 2021 at 05:59:45PM -0800, Si-Wei Liu wrote: >>>>>> On 12/12/2021 1:26 AM, Michael S. Tsirkin wrote: >>>>>>> On Fri, Dec 10, 2021 at 05:44:15PM -0800, Si-Wei Liu wrote: >>>>>>>> Sorry for reviving this ancient thread. I was kinda lost for the conclusion >>>>>>>> it ended up with. I have the following questions, >>>>>>>> >>>>>>>> 1. legacy guest support: from the past conversations it doesn't seem the >>>>>>>> support will be completely dropped from the table, is my understanding >>>>>>>> correct? Actually we're interested in supporting virtio v0.95 guest for x86, >>>>>>>> which is backed by the spec at >>>>>>>> https://urldefense.com/v3/__https://ozlabs.org/*rusty/virtio-spec/virtio-0.9.5.pdf__;fg!!ACWV5N9M2RV99hQ!dTKmzJwwRsFM7BtSuTDu1cNly5n4XCotH0WYmidzGqHSXt40i7ZU43UcNg7GYxZg$ . Though I'm not sure >>>>>>>> if there's request/need to support wilder legacy virtio versions earlier >>>>>>>> beyond. >>>>>>> I personally feel it's less work to add in kernel than try to >>>>>>> work around it in userspace. Jason feels differently. >>>>>>> Maybe post the patches and this will prove to Jason it's not >>>>>>> too terrible? >>>>>> I suppose if the vdpa vendor does support 0.95 in the datapath and ring >>>>>> layout level and is limited to x86 only, there should be easy way out. >>>>> Note a subtle difference: what matters is that guest, not host is x86. >>>>> Matters for emulators which might reorder memory accesses. >>>>> I guess this enforcement belongs in QEMU then? >>>> Right, I mean to get started, the initial guest driver support and the >>>> corresponding QEMU support for transitional vdpa backend can be limited >>>> to x86 guest/host only. Since the config space is emulated in QEMU, I >>>> suppose it's not hard to enforce in QEMU. >>> It's more than just config space, most devices have headers before the buffer. >> The ordering in datapath (data VQs) would have to rely on vendor's >> support. Since ORDER_PLATFORM is pretty new (v1.1), I guess vdpa h/w >> vendor nowadays can/should well support the case when ORDER_PLATFORM is >> not acked by the driver (actually this feature is filtered out by the >> QEMU vhost-vdpa driver today), even with v1.0 spec conforming and modern >> only vDPA device. > That's a bug that needs to be fixed. > >> The control VQ is implemented in software in the >> kernel, which can be easily accommodated/fixed when needed. >> >>>> QEMU can drive GET_LEGACY, >>>> GET_ENDIAN et al ioctls in advance to get the capability from the >>>> individual vendor driver. For that, we need another negotiation protocol >>>> similar to vhost_user's protocol_features between the vdpa kernel and >>>> QEMU, way before the guest driver is ever probed and its feature >>>> negotiation kicks in. Not sure we need a GET_MEMORY_ORDER ioctl call >>>> from the device, but we can assume weak ordering for legacy at this >>>> point (x86 only)? >>> I'm lost here, we have get_features() so: >> I assume here you refer to get_device_features() that Eli just changed >> the name. >>> 1) VERSION_1 means the device uses LE if provided, otherwise natvie >>> 2) ORDER_PLATFORM means device requires platform ordering >>> >>> Any reason for having a new API for this? >> Are you going to enforce all vDPA hardware vendors to support the >> transitional model for legacy guest? meaning guest not acknowledging >> VERSION_1 would use the legacy interfaces captured in the spec section >> 7.4 (regarding ring layout, native endianness, message framing, vq >> alignment of 4096, 32bit feature, no features_ok bit in status, IO port >> interface i.e. all the things) instead? Noted we don't yet have a >> set_device_features() that allows the vdpa device to tell whether it is >> operating in transitional or modern-only mode. For software virtio, all >> support for the legacy part in a transitional model has been built up >> there already, however, it's not easy for vDPA vendors to implement all >> the requirements for an all-or-nothing legacy guest support (big endian >> guest for example). To these vendors, the legacy support within a >> transitional model is more of feature to them and it's best to leave >> some flexibility for them to implement partial support for legacy. That >> in turn calls out the need for a vhost-user protocol feature like >> negotiation API that can prohibit those unsupported guest setups to as >> early as backend_init before launching the VM. >> >> >>>>>> I >>>>>> checked with Eli and other Mellanox/NVDIA folks for hardware/firmware level >>>>>> 0.95 support, it seems all the ingredient had been there already dated back >>>>>> to the DPDK days. The only major thing limiting is in the vDPA software that >>>>>> the current vdpa core has the assumption around VIRTIO_F_ACCESS_PLATFORM for >>>>>> a few DMA setup ops, which is virtio 1.0 only. >>>>>> >>>>>>>> 2. suppose some form of legacy guest support needs to be there, how do we >>>>>>>> deal with the bogus assumption below in vdpa_get_config() in the short term? >>>>>>>> It looks one of the intuitive fix is to move the vdpa_set_features call out >>>>>>>> of vdpa_get_config() to vdpa_set_config(). >>>>>>>> >>>>>>>> /* >>>>>>>> * Config accesses aren't supposed to trigger before features are >>>>>>>> set. >>>>>>>> * If it does happen we assume a legacy guest. >>>>>>>> */ >>>>>>>> if (!vdev->features_valid) >>>>>>>> vdpa_set_features(vdev, 0); >>>>>>>> ops->get_config(vdev, offset, buf, len); >>>>>>>> >>>>>>>> I can post a patch to fix 2) if there's consensus already reached. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> -Siwei >>>>>>> I'm not sure how important it is to change that. >>>>>>> In any case it only affects transitional devices, right? >>>>>>> Legacy only should not care ... >>>>>> Yes I'd like to distinguish legacy driver (suppose it is 0.95) against the >>>>>> modern one in a transitional device model rather than being legacy only. >>>>>> That way a v0.95 and v1.0 supporting vdpa parent can support both types of >>>>>> guests without having to reconfigure. Or are you suggesting limit to legacy >>>>>> only at the time of vdpa creation would simplify the implementation a lot? >>>>>> >>>>>> Thanks, >>>>>> -Siwei >>>>> I don't know for sure. Take a look at the work Halil was doing >>>>> to try and support transitional devices with BE guests. >>>> Hmmm, we can have those endianness ioctls defined but the initial QEMU >>>> implementation can be started to support x86 guest/host with little >>>> endian and weak memory ordering first. The real trick is to detect >>>> legacy guest - I am not sure if it's feasible to shift all the legacy >>>> detection work to QEMU, or the kernel has to be part of the detection >>>> (e.g. the kick before DRIVER_OK thing we have to duplicate the tracking >>>> effort in QEMU) as well. Let me take a further look and get back. >>> Michael may think differently but I think doing this in Qemu is much easier. >> I think the key is whether we position emulating legacy interfaces in >> QEMU doing translation on top of a v1.0 modern-only device in the >> kernel, or we allow vdpa core (or you can say vhost-vdpa) and vendor >> driver to support a transitional model in the kernel that is able to >> work for both v0.95 and v1.0 drivers, with some slight aid from QEMU for >> detecting/emulation/shadowing (for e.g CVQ, I/O port relay). I guess for >> the former we still rely on vendor for a performant data vqs >> implementation, leaving the question to what it may end up eventually in >> the kernel is effectively the latter). > I think we can do the legacy interface emulation on top of the shadow > VQ. And we know it works for sure. But I agree, it would be much > easier if we depend on the vendor to implement a transitional device. First I am not sure if there's a convincing case for users to deploy vDPA with shadow (data) VQ against the pure software based backend. Please enlighten me if there is. For us, the point to deploy vDPA for legacy guest is the acceleration (what "A" stands for in "vDPA") part of it so that we can leverage the hardware potential if at all possible. Not sure how the shadow VQ implementation can easily deal with datapath acceleration without losing too much performance? > So assuming we depend on the vendor, I don't see anything that is > strictly needed in the kernel, the kick or config access before > DRIVER_OK can all be handled easily in Qemu unless I miss something. Right, that's what I think too it's not quite a lot of work in the kernel if vendor device offers the aid/support for transitional. The kernel only provides the abstraction of device model (transitional or modern-only), while vendor driver may implement early platform feature discovery and apply legacy specific quirks (unsupported endianness, mismatched page size, unsupported host memory ordering model) that the device can't adapt to. I don't say we have to depend on the vendor, but the point is that we must assume fully spec compliant transitional support (the datapath in particular) from the vendor to get started, as I guess it's probably the main motivation for users to deploy it - acceleration of legacy guest workload without exhausting host computing resource. Even if we get started with shadow VQ to mediate and translate the datapath, eventually it may evolve towards leveraging datapath offload to hardware if acceleration is the only convincing use case for legacy support. Thanks, -Siwei > The only value to do that in the kernel is that it can work for > virtio-vdpa, but modern only virito-vpda is sufficient; we don't need > any legacy stuff for that. > > Thanks > >> Thanks, >> -Siwei >> >>> Thanks >>> >>> >>> >>>> Meanwhile, I'll check internally to see if a legacy only model would >>>> work. Thanks. >>>> >>>> Thanks, >>>> -Siwei >>>> >>>> >>>>>>>> On 3/2/2021 2:53 AM, Jason Wang wrote: >>>>>>>>> On 2021/3/2 5:47 下午, Michael S. Tsirkin wrote: >>>>>>>>>> On Mon, Mar 01, 2021 at 11:56:50AM +0800, Jason Wang wrote: >>>>>>>>>>> On 2021/3/1 5:34 上午, Michael S. Tsirkin wrote: >>>>>>>>>>>> On Wed, Feb 24, 2021 at 10:24:41AM -0800, Si-Wei Liu wrote: >>>>>>>>>>>>>> Detecting it isn't enough though, we will need a new ioctl to notify >>>>>>>>>>>>>> the kernel that it's a legacy guest. Ugh :( >>>>>>>>>>>>> Well, although I think adding an ioctl is doable, may I >>>>>>>>>>>>> know what the use >>>>>>>>>>>>> case there will be for kernel to leverage such info >>>>>>>>>>>>> directly? Is there a >>>>>>>>>>>>> case QEMU can't do with dedicate ioctls later if there's indeed >>>>>>>>>>>>> differentiation (legacy v.s. modern) needed? >>>>>>>>>>>> BTW a good API could be >>>>>>>>>>>> >>>>>>>>>>>> #define VHOST_SET_ENDIAN _IOW(VHOST_VIRTIO, ?, int) >>>>>>>>>>>> #define VHOST_GET_ENDIAN _IOW(VHOST_VIRTIO, ?, int) >>>>>>>>>>>> >>>>>>>>>>>> we did it per vring but maybe that was a mistake ... >>>>>>>>>>> Actually, I wonder whether it's good time to just not support >>>>>>>>>>> legacy driver >>>>>>>>>>> for vDPA. Consider: >>>>>>>>>>> >>>>>>>>>>> 1) It's definition is no-normative >>>>>>>>>>> 2) A lot of budren of codes >>>>>>>>>>> >>>>>>>>>>> So qemu can still present the legacy device since the config >>>>>>>>>>> space or other >>>>>>>>>>> stuffs that is presented by vhost-vDPA is not expected to be >>>>>>>>>>> accessed by >>>>>>>>>>> guest directly. Qemu can do the endian conversion when necessary >>>>>>>>>>> in this >>>>>>>>>>> case? >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> >>>>>>>>>> Overall I would be fine with this approach but we need to avoid breaking >>>>>>>>>> working userspace, qemu releases with vdpa support are out there and >>>>>>>>>> seem to work for people. Any changes need to take that into account >>>>>>>>>> and document compatibility concerns. >>>>>>>>> Agree, let me check. >>>>>>>>> >>>>>>>>> >>>>>>>>>> I note that any hardware >>>>>>>>>> implementation is already broken for legacy except on platforms with >>>>>>>>>> strong ordering which might be helpful in reducing the scope. >>>>>>>>> Yes. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B3AEDC433FE for ; Fri, 17 Dec 2021 01:08:44 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 2BB2D61073; Fri, 17 Dec 2021 01:08:44 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8YCIvpfIcbLL; Fri, 17 Dec 2021 01:08:42 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 5268061063; Fri, 17 Dec 2021 01:08:42 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 34565C001E; Fri, 17 Dec 2021 01:08:42 +0000 (UTC) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1B633C0012 for ; Fri, 17 Dec 2021 01:08:41 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 04B236106A for ; Fri, 17 Dec 2021 01:08:41 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NLVj0-o5iiHH for ; Fri, 17 Dec 2021 01:08:39 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by smtp3.osuosl.org (Postfix) with ESMTPS id 3DADB61063 for ; Fri, 17 Dec 2021 01:08:39 +0000 (UTC) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 1BGLE2p8031682; Fri, 17 Dec 2021 01:08:38 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=message-id : date : subject : to : cc : references : from : in-reply-to : content-type : content-transfer-encoding : mime-version; s=corp-2021-07-09; bh=cr6qc2aB6HFqpdpYp6v+0aRhh2Hg+Bg4KN2ituLWZEc=; b=XomEdt259oynkkHxmC3in/erqStSiBbE2Y95MmcT3annRycrVGIN8C+pMTMXkqge3hAF S05ywyC4YPVSYzmJh58Vz/i9WfnLxdztKgZU+/LHFG6hCvXlCOG5MpACg4s9pCtka1kH ifMr1xh4JcC7D4Kdfj7Zyqdr0hEGwBSaw6USjA6YA/O/R5hjGyWNLuNooNmJaMP+qUkG psYoYCxv64vsBoOCHjru5RzOp3vzj3NbEXY6DSkT5SeGL1SuC8j4AZY7h2bZxfXG7nO5 NOFctyqWXAMEto32dKKpb5pEtU1UYyhQVuOu/6K5nupihIrcKGzm34wxvfVnt5awK3uG Xw== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 3cykmbmgqa-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 17 Dec 2021 01:08:37 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.1.2/8.16.1.2) with SMTP id 1BH15tuQ114925; Fri, 17 Dec 2021 01:08:36 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2041.outbound.protection.outlook.com [104.47.66.41]) by userp3020.oracle.com with ESMTP id 3cvneupx5h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 17 Dec 2021 01:08:36 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IhBwq3S466pc1oqfdJuBUnfFyEu+HBd/oBUnOXTO68r6Y0PERX0n55lUnlkElc1OEoyU6Qn/VaU4QrDM5oxXQlwzDpcBkGWmYuQcbl4EqTmAIVZPBZr0OeXXAVodKFhgpyyMEjjHwI3V51FrESJVW38HNRoILWTgpi/LrfE0KfdVKB6UZpU79HtHEuHzMOroFjbOnRmo0nHqQWU6rd8Dd5HbtMLoFLCeCaAH7RMFAtERRtA35mJgp9AKD6birZfGYq+n/E0aDVHjhJcdhvJMGSE+aStrptej5+ylgEyEKjAAlSMbUJ6HwI/+27K7wdLfiQLZWhNdKe9b50DSRDougA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=cr6qc2aB6HFqpdpYp6v+0aRhh2Hg+Bg4KN2ituLWZEc=; b=nDm5yt3T3t6DY2aKhtjjUEMmtLG+IS+OrWrff7mVWe9TNs0fVNN9m94xVY/1YA81bIEnSuZSILNs6hdX047JhAdkghbJiW/s99AbOeAg921R8EbWBxQR/NhKK9bKR3nc5qI7OHI4uwph/jvbYLNpuFbrvb31fkyej/JNQbFUz4dMBn+2mdbQwAsuSc23JW+WD1LHQhIBrZICQ11ZRn5qaeiUaM3AArDQ8B5liFfmu/R8RfRZ1ZqsrOa9BGHWaeyFKVThr9BaZ/03wMh43VKKJiSSPFavlneDRf/oHeJjIseONQnnicwGO9QaOyfSdeZSJ7jyiDi20K8xPeQ7Qb13pw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cr6qc2aB6HFqpdpYp6v+0aRhh2Hg+Bg4KN2ituLWZEc=; b=ecsVxtbILDUnqKIoNQKvfnyO2bfGyV/lg6K/XzeqpwG6Bx3Lk29Fth2uPGtwqFsJ0NM7SPaKaWYN5hhmvHTO5bACJdR+vm/nCGtT8+N3tDMLx5QDlWYnp5SRqAe6dcMrohq4T2wOC8wcv5lWtqxNYJ7JTJYn8CkESw7zSFE4kO8= Received: from BYAPR10MB3287.namprd10.prod.outlook.com (2603:10b6:a03:15c::11) by SJ0PR10MB4734.namprd10.prod.outlook.com (2603:10b6:a03:2d2::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4778.17; Fri, 17 Dec 2021 01:08:33 +0000 Received: from BYAPR10MB3287.namprd10.prod.outlook.com ([fe80::7c7e:4a5e:24e0:309d]) by BYAPR10MB3287.namprd10.prod.outlook.com ([fe80::7c7e:4a5e:24e0:309d%3]) with mapi id 15.20.4778.018; Fri, 17 Dec 2021 01:08:33 +0000 Message-ID: Date: Thu, 16 Dec 2021 17:08:28 -0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.3.2 Subject: Re: vdpa legacy guest support (was Re: [PATCH] vdpa/mlx5: set_features should allow reset to zero) Content-Language: en-US To: Jason Wang References: <3ff5fd23-1db0-2f95-4cf9-711ef403fb62@oracle.com> <20210224000057-mutt-send-email-mst@kernel.org> <52836a63-4e00-ff58-50fb-9f450ce968d7@oracle.com> <20210228163031-mutt-send-email-mst@kernel.org> <2cb51a6d-afa0-7cd1-d6f2-6b153186eaca@redhat.com> <20210302043419-mutt-send-email-mst@kernel.org> <178f8ea7-cebd-0e81-3dc7-10a058d22c07@redhat.com> <20211212042311-mutt-send-email-mst@kernel.org> <20211214000245-mutt-send-email-mst@kernel.org> <4fc43d0f-da9e-ce16-1f26-9f0225239b75@oracle.com> <6eaf672c-cc86-b5bf-5b74-c837affeb6e1@oracle.com> From: Si-Wei Liu Organization: Oracle Corporation In-Reply-To: X-ClientProxiedBy: SN2PR01CA0069.prod.exchangelabs.com (2603:10b6:800::37) To BYAPR10MB3287.namprd10.prod.outlook.com (2603:10b6:a03:15c::11) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c60938e5-8246-4c06-9c0e-08d9c0f9bf15 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4734:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: NmRCKmxk/qJT6z1qOJHZloaO2KxKTNO2Z+3TRgFsZg/y8V7zk8WigpjOvomAwW+RLlJbY9xpitQ69DK9UKwLgqMBLzSh1i0LNTctc+SuuREiss5Gi7+uiKMAzQUXoRAaZ0/+MIb7xPhbQLwE0ScnK4QAtoNpH1iWi52X/JyS5NmQ61BN/5Pl8o4P7QTn5AkKnZKZDi4FbS4rcelhk7j4FwUV0h7SmvzRMtNSMzTyQp3lKGrcKD9NSbXfEGUbmfd2Q0S9PND1/ezkC7KxNka2egS05QBVHTDbF6sry+ZF4hTqvEnOFoGhU6mzRy2FO9ky6K7Yi7kJhNbvttySNVkgTx+dNKDBRuISb5EEmOJN35v2gvotpiHEaC24JoXFpYSVwL+7HQHweK4MC1wyfSU6zmOk7omGss0VUCB57fG2zmbHiBA0tB6rSaAOiqFfQhkGhpfgbTnURzLgPJ7mP+DRZlmGdl8B/2ZY3Zr/zN898vdE2fiPPWVQYyR4eB0qAq91KHtKHmO9OwuqXS2LtCUUuS0S+CyDPdsfq+kV1wlEkC+u92WWyUEkTisiW20xA+5QZqSbsj9A9F4nte5UqQ4BBmCwhxy3/PdLWiwmHwKWslO1wYTLMVdzVh3ouQtb/iB9F2LlJ5VgDsuD2TTNdCouSpX9e2ojm/6H0sqyJW/tXvAlYmJ1z21X8Piwbm/xi1AyRhb+QBvK0/trX2BidtPPxEVmcOkjjhsdMA+LPf33ErUlIE5Rg5R611RUh21wq1pDaJP3rJD2CaGvGhPdL6FuVSoH1fiqC5sJNe4pOeI6uJvDAbekF+siorIIo31MXbO1 X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3287.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(83380400001)(31686004)(8936002)(6916009)(66946007)(6506007)(8676002)(31696002)(66556008)(66476007)(36756003)(38100700002)(186003)(508600001)(36916002)(4326008)(6666004)(53546011)(26005)(966005)(2616005)(2906002)(54906003)(6512007)(86362001)(6486002)(5660300002)(316002)(30864003)(45980500001)(43740500002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cCtaT1Fzelg0ZktCdk1pRGtqZ2owY0lUUGhyeUpoMjNTczR6QW9GQ0RMMVNk?= =?utf-8?B?T3B4SzlHS05DMHduNkE5ZDdkTU54cElrR3pqYW43K24yZTVURk1zMzRtaFFz?= =?utf-8?B?WXltTHVqRVRiY1dVdUx5RmJ1M2ZXN0Z5Rmt0MDY4bVhqaTVvclhGVGdZcDlM?= =?utf-8?B?RDdpNUszbEhuSjJPQUQ3VG1aM3J3ZEdUK3F2MjFhS3dlMFhkakhyK2pkbGVF?= =?utf-8?B?ekdIc1J3MXRLRlRFUlRYMmMzU2RMS2ZkZk5ObTRMRjE1QVRjNmUrVVp5OHBz?= =?utf-8?B?cmkvRkZSbWoxdDZsSi9kMkNFL2RRRW5FUnhUbjhoR3FRZzQ0NGlMaHI2dGIv?= =?utf-8?B?eFhTMURxbHhDVlowRDFZYmVTcFQrOTlQb0d0blAwMWhQUldsSFQwUG1qUHdH?= =?utf-8?B?RGNXYTJneGJBZHQ5NlVTaDJrL0toNmhCVE5DUi9MejJPMUUyWFNBSDFsdFc0?= =?utf-8?B?Z2ErWEtHQlNUNUtwQ0UxQW5MVkpDTFpoazJNcitaRnhmaFcvRTNPSEpNZ2NK?= =?utf-8?B?cVYzYldKazlmRDJKaTNla2tNdmlEK0oxUzBUNHB3bmJDRjduR0NqZlI2V01u?= =?utf-8?B?TmVDbWV2czJGaVc3Lyt0UW9NNGwyRDgySG5aNGVQVW9ISWVvWWsxRHVNeWpF?= =?utf-8?B?b0FVWDhoNUd4S2hSb0NLYlJCNzU1b2VaaXhFOWlDV0V3RkJtSnpidElMazNE?= =?utf-8?B?YlhJSUQ4bXlzSkh2U1MyeTFEUXRmb1lBcitYMkcrMTFhbnpMR3FWb04rU0Rx?= =?utf-8?B?M2ljVEpvWk5SRG5IVEFMWnBjM1hZTmpGQW5aRXV4UXhPZWJVek1TYlZHcnNa?= =?utf-8?B?b3dQNm1pT3JwWFRrazV6OTA1bnl2aDFVR3lweHVJY3doeHhRbEpkbGphY0Jl?= =?utf-8?B?c0VRZXQvVWNFaHp5ZjNyRE5DYmp5ci9GbDRWM0FhbGZaNUdHc0N2SVhOeEox?= =?utf-8?B?Y2xWOVlZUG5XdWh2OVZPa0RQMXNtQkw3RVFMdXhNaFUvblM1QktqZTc3eUVH?= =?utf-8?B?RndvdUNIZ3JjekMyVUZEVFhzeEN5ejM5NnZodGpydVVZLzRqTG54ZjZ4UXJm?= =?utf-8?B?Y2xwZnVWSzRPcEZ5TUVKckxGOGZHdEVuMzFqb2RXMW1LQUZkRHdJN3hzV05K?= =?utf-8?B?aWNOUFlMbGdLVUNSaU9pUEt3aCtQNnNQUWdLcVJaSkhVT1o0TmEvbmlCUzly?= =?utf-8?B?MGRRT0EyU2l2cW9BaVJaQkZxSGp0VzQyMnJpRE1qRk1NU1dPUSt2MnJQb3JM?= =?utf-8?B?RWJiaWw3OWFJU3hITVYzdWNCWlJDTDRadis4cjcyRnoyd1RLSWl5NTZXTlpP?= =?utf-8?B?ZHFWWStUVWkzZ0JXTkhxYnBINXM4L1czWGNGTmtWZ1JxUnB3SWF0SjJhZXRC?= =?utf-8?B?MlpnS0NzUGtrMm9BUytLQURCUTR1Y1ZpUnc2L3hYeVYxVXpLTzMwU0Q2TXJG?= =?utf-8?B?Q04xcS9MeHhyMXNwZHlIMWEwb25ZMVZNUVJlTUVyQmJRczVFekZ1ekZyaVBK?= =?utf-8?B?UmIyZkNsajVIWHV5NUcvK1R5U0ZlMEx0MGdvb3VYaTJKSzJabnlvWHJMTnBy?= =?utf-8?B?dW40aFFmSC8yY01UQWQrRVNVVGlTaVg1U1RKU0xqV2xOWnFpQ2pvSm9kMmVu?= =?utf-8?B?VytmczNKSHE5YkFMQW9hUkdVb1Jvb2d3cldFL2FaeFVieDM0aWpRSWw2RDg5?= =?utf-8?B?R2hkVWR2Q1BKek5rbHZXODB1TDc1cWQxY2tUQ3kxZlhFNldMS0V5LzhZRUJO?= =?utf-8?B?byszVTlKME1JQVUwb1ZzVDM5U1BjVENUM1ExU0cwK3hHOUtSSnJXZjVjSTdk?= =?utf-8?B?V0lEU0ZoZFIydlRRWTd1aDR1RlNkSGdBcldteTBQUU9TeG1XTThuekYyRUFO?= =?utf-8?B?MjZVb2F0OXVpdmtTb3kxS1dzeUpKQ0J4Q2lMNjZDYmpCNlVFRDI1SStuTWtx?= =?utf-8?B?M2JtbDBYZWorL3pkUDJxZUNkOXBPcFUxTlQ3M0U5Wm5SWkJoYlRmUm9ubmpW?= =?utf-8?B?NFN6RU9jRDMxcWJYUnB4WWtoTjQrUHd4dWFBam5ucEY1L1pQak1zWnZFeHd1?= =?utf-8?B?YkNQTDJYNGhlZUFGZXpnZ3BYQU84ZlIxSzd4MHdrSzQ0ZUZWOFM3MjFOa2Zk?= =?utf-8?B?eGI1MmRxeVdVcll2cXZ2RnFaSnRXZEU0dDVxRm1WTnZlY2s5SG5IVkVBR0ti?= =?utf-8?Q?ML4mSK8nDUwG6/CyShPnwPk=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: c60938e5-8246-4c06-9c0e-08d9c0f9bf15 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3287.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Dec 2021 01:08:33.6921 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 1NGo6y/VZmVq9ZycmhPjFguRBFF6ALFTl0B3Pa+9dZmxWGZuqI6aeaj7Du4l7VBZQjuEaM6Vd1RrvTc69q9jyw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4734 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10200 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0 malwarescore=0 bulkscore=0 spamscore=0 mlxlogscore=999 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000 definitions=main-2112170005 X-Proofpoint-ORIG-GUID: MiqO2UHQpLkUG4vUAhcUTzqkMAWsis5C X-Proofpoint-GUID: MiqO2UHQpLkUG4vUAhcUTzqkMAWsis5C Cc: netdev , Eli Cohen , virtualization , linux-kernel , "Michael S. Tsirkin" X-BeenThere: virtualization@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Linux virtualization List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Errors-To: virtualization-bounces@lists.linux-foundation.org Sender: "Virtualization" CgpPbiAxMi8xNS8yMDIxIDc6NDMgUE0sIEphc29uIFdhbmcgd3JvdGU6Cj4gT24gVGh1LCBEZWMg MTYsIDIwMjEgYXQgNDo1MiBBTSBTaS1XZWkgTGl1IDxzaS13ZWkubGl1QG9yYWNsZS5jb20+IHdy b3RlOgo+Pgo+Pgo+PiBPbiAxMi8xNC8yMDIxIDY6MDYgUE0sIEphc29uIFdhbmcgd3JvdGU6Cj4+ PiBPbiBXZWQsIERlYyAxNSwgMjAyMSBhdCA5OjA1IEFNIFNpLVdlaSBMaXUgPHNpLXdlaS5saXVA b3JhY2xlLmNvbT4gd3JvdGU6Cj4+Pj4KPj4+PiBPbiAxMi8xMy8yMDIxIDk6MDYgUE0sIE1pY2hh ZWwgUy4gVHNpcmtpbiB3cm90ZToKPj4+Pj4gT24gTW9uLCBEZWMgMTMsIDIwMjEgYXQgMDU6NTk6 NDVQTSAtMDgwMCwgU2ktV2VpIExpdSB3cm90ZToKPj4+Pj4+IE9uIDEyLzEyLzIwMjEgMToyNiBB TSwgTWljaGFlbCBTLiBUc2lya2luIHdyb3RlOgo+Pj4+Pj4+IE9uIEZyaSwgRGVjIDEwLCAyMDIx IGF0IDA1OjQ0OjE1UE0gLTA4MDAsIFNpLVdlaSBMaXUgd3JvdGU6Cj4+Pj4+Pj4+IFNvcnJ5IGZv ciByZXZpdmluZyB0aGlzIGFuY2llbnQgdGhyZWFkLiBJIHdhcyBraW5kYSBsb3N0IGZvciB0aGUg Y29uY2x1c2lvbgo+Pj4+Pj4+PiBpdCBlbmRlZCB1cCB3aXRoLiBJIGhhdmUgdGhlIGZvbGxvd2lu ZyBxdWVzdGlvbnMsCj4+Pj4+Pj4+Cj4+Pj4+Pj4+IDEuIGxlZ2FjeSBndWVzdCBzdXBwb3J0OiBm cm9tIHRoZSBwYXN0IGNvbnZlcnNhdGlvbnMgaXQgZG9lc24ndCBzZWVtIHRoZQo+Pj4+Pj4+PiBz dXBwb3J0IHdpbGwgYmUgY29tcGxldGVseSBkcm9wcGVkIGZyb20gdGhlIHRhYmxlLCBpcyBteSB1 bmRlcnN0YW5kaW5nCj4+Pj4+Pj4+IGNvcnJlY3Q/IEFjdHVhbGx5IHdlJ3JlIGludGVyZXN0ZWQg aW4gc3VwcG9ydGluZyB2aXJ0aW8gdjAuOTUgZ3Vlc3QgZm9yIHg4NiwKPj4+Pj4+Pj4gd2hpY2gg aXMgYmFja2VkIGJ5IHRoZSBzcGVjIGF0Cj4+Pj4+Pj4+IGh0dHBzOi8vdXJsZGVmZW5zZS5jb20v djMvX19odHRwczovL296bGFicy5vcmcvKnJ1c3R5L3ZpcnRpby1zcGVjL3ZpcnRpby0wLjkuNS5w ZGZfXztmZyEhQUNXVjVOOU0yUlY5OWhRIWRUS216Snd3UnNGTTdCdFN1VER1MWNObHk1bjRYQ290 SDBXWW1pZHpHcUhTWHQ0MGk3WlU0M1VjTmc3R1l4WmckIC4gVGhvdWdoIEknbSBub3Qgc3VyZQo+ Pj4+Pj4+PiBpZiB0aGVyZSdzIHJlcXVlc3QvbmVlZCB0byBzdXBwb3J0IHdpbGRlciBsZWdhY3kg dmlydGlvIHZlcnNpb25zIGVhcmxpZXIKPj4+Pj4+Pj4gYmV5b25kLgo+Pj4+Pj4+IEkgcGVyc29u YWxseSBmZWVsIGl0J3MgbGVzcyB3b3JrIHRvIGFkZCBpbiBrZXJuZWwgdGhhbiB0cnkgdG8KPj4+ Pj4+PiB3b3JrIGFyb3VuZCBpdCBpbiB1c2Vyc3BhY2UuIEphc29uIGZlZWxzIGRpZmZlcmVudGx5 Lgo+Pj4+Pj4+IE1heWJlIHBvc3QgdGhlIHBhdGNoZXMgYW5kIHRoaXMgd2lsbCBwcm92ZSB0byBK YXNvbiBpdCdzIG5vdAo+Pj4+Pj4+IHRvbyB0ZXJyaWJsZT8KPj4+Pj4+IEkgc3VwcG9zZSBpZiB0 aGUgdmRwYSB2ZW5kb3IgZG9lcyBzdXBwb3J0IDAuOTUgaW4gdGhlIGRhdGFwYXRoIGFuZCByaW5n Cj4+Pj4+PiBsYXlvdXQgbGV2ZWwgYW5kIGlzIGxpbWl0ZWQgdG8geDg2IG9ubHksIHRoZXJlIHNo b3VsZCBiZSBlYXN5IHdheSBvdXQuCj4+Pj4+IE5vdGUgYSBzdWJ0bGUgZGlmZmVyZW5jZTogd2hh dCBtYXR0ZXJzIGlzIHRoYXQgZ3Vlc3QsIG5vdCBob3N0IGlzIHg4Ni4KPj4+Pj4gTWF0dGVycyBm b3IgZW11bGF0b3JzIHdoaWNoIG1pZ2h0IHJlb3JkZXIgbWVtb3J5IGFjY2Vzc2VzLgo+Pj4+PiBJ IGd1ZXNzIHRoaXMgZW5mb3JjZW1lbnQgYmVsb25ncyBpbiBRRU1VIHRoZW4/Cj4+Pj4gUmlnaHQs IEkgbWVhbiB0byBnZXQgc3RhcnRlZCwgdGhlIGluaXRpYWwgZ3Vlc3QgZHJpdmVyIHN1cHBvcnQg YW5kIHRoZQo+Pj4+IGNvcnJlc3BvbmRpbmcgUUVNVSBzdXBwb3J0IGZvciB0cmFuc2l0aW9uYWwg dmRwYSBiYWNrZW5kIGNhbiBiZSBsaW1pdGVkCj4+Pj4gdG8geDg2IGd1ZXN0L2hvc3Qgb25seS4g U2luY2UgdGhlIGNvbmZpZyBzcGFjZSBpcyBlbXVsYXRlZCBpbiBRRU1VLCBJCj4+Pj4gc3VwcG9z ZSBpdCdzIG5vdCBoYXJkIHRvIGVuZm9yY2UgaW4gUUVNVS4KPj4+IEl0J3MgbW9yZSB0aGFuIGp1 c3QgY29uZmlnIHNwYWNlLCBtb3N0IGRldmljZXMgaGF2ZSBoZWFkZXJzIGJlZm9yZSB0aGUgYnVm ZmVyLgo+PiBUaGUgb3JkZXJpbmcgaW4gZGF0YXBhdGggKGRhdGEgVlFzKSB3b3VsZCBoYXZlIHRv IHJlbHkgb24gdmVuZG9yJ3MKPj4gc3VwcG9ydC4gU2luY2UgT1JERVJfUExBVEZPUk0gaXMgcHJl dHR5IG5ldyAodjEuMSksIEkgZ3Vlc3MgdmRwYSBoL3cKPj4gdmVuZG9yIG5vd2FkYXlzIGNhbi9z aG91bGQgd2VsbCBzdXBwb3J0IHRoZSBjYXNlIHdoZW4gT1JERVJfUExBVEZPUk0gaXMKPj4gbm90 IGFja2VkIGJ5IHRoZSBkcml2ZXIgKGFjdHVhbGx5IHRoaXMgZmVhdHVyZSBpcyBmaWx0ZXJlZCBv dXQgYnkgdGhlCj4+IFFFTVUgdmhvc3QtdmRwYSBkcml2ZXIgdG9kYXkpLCBldmVuIHdpdGggdjEu MCBzcGVjIGNvbmZvcm1pbmcgYW5kIG1vZGVybgo+PiBvbmx5IHZEUEEgZGV2aWNlLgo+IFRoYXQn cyBhIGJ1ZyB0aGF0IG5lZWRzIHRvIGJlIGZpeGVkLgo+Cj4+IFRoZSBjb250cm9sIFZRIGlzIGlt cGxlbWVudGVkIGluIHNvZnR3YXJlIGluIHRoZQo+PiBrZXJuZWwsIHdoaWNoIGNhbiBiZSBlYXNp bHkgYWNjb21tb2RhdGVkL2ZpeGVkIHdoZW4gbmVlZGVkLgo+Pgo+Pj4+IFFFTVUgY2FuIGRyaXZl IEdFVF9MRUdBQ1ksCj4+Pj4gR0VUX0VORElBTiBldCBhbCBpb2N0bHMgaW4gYWR2YW5jZSB0byBn ZXQgdGhlIGNhcGFiaWxpdHkgZnJvbSB0aGUKPj4+PiBpbmRpdmlkdWFsIHZlbmRvciBkcml2ZXIu IEZvciB0aGF0LCB3ZSBuZWVkIGFub3RoZXIgbmVnb3RpYXRpb24gcHJvdG9jb2wKPj4+PiBzaW1p bGFyIHRvIHZob3N0X3VzZXIncyBwcm90b2NvbF9mZWF0dXJlcyBiZXR3ZWVuIHRoZSB2ZHBhIGtl cm5lbCBhbmQKPj4+PiBRRU1VLCB3YXkgYmVmb3JlIHRoZSBndWVzdCBkcml2ZXIgaXMgZXZlciBw cm9iZWQgYW5kIGl0cyBmZWF0dXJlCj4+Pj4gbmVnb3RpYXRpb24ga2lja3MgaW4uIE5vdCBzdXJl IHdlIG5lZWQgYSBHRVRfTUVNT1JZX09SREVSIGlvY3RsIGNhbGwKPj4+PiBmcm9tIHRoZSBkZXZp Y2UsIGJ1dCB3ZSBjYW4gYXNzdW1lIHdlYWsgb3JkZXJpbmcgZm9yIGxlZ2FjeSBhdCB0aGlzCj4+ Pj4gcG9pbnQgKHg4NiBvbmx5KT8KPj4+IEknbSBsb3N0IGhlcmUsIHdlIGhhdmUgZ2V0X2ZlYXR1 cmVzKCkgc286Cj4+IEkgYXNzdW1lIGhlcmUgeW91IHJlZmVyIHRvIGdldF9kZXZpY2VfZmVhdHVy ZXMoKSB0aGF0IEVsaSBqdXN0IGNoYW5nZWQKPj4gdGhlIG5hbWUuCj4+PiAxKSBWRVJTSU9OXzEg bWVhbnMgdGhlIGRldmljZSB1c2VzIExFIGlmIHByb3ZpZGVkLCBvdGhlcndpc2UgbmF0dmllCj4+ PiAyKSBPUkRFUl9QTEFURk9STSBtZWFucyBkZXZpY2UgcmVxdWlyZXMgcGxhdGZvcm0gb3JkZXJp bmcKPj4+Cj4+PiBBbnkgcmVhc29uIGZvciBoYXZpbmcgYSBuZXcgQVBJIGZvciB0aGlzPwo+PiBB cmUgeW91IGdvaW5nIHRvIGVuZm9yY2UgYWxsIHZEUEEgaGFyZHdhcmUgdmVuZG9ycyB0byBzdXBw b3J0IHRoZQo+PiB0cmFuc2l0aW9uYWwgbW9kZWwgZm9yIGxlZ2FjeSBndWVzdD8gbWVhbmluZyBn dWVzdCBub3QgYWNrbm93bGVkZ2luZwo+PiBWRVJTSU9OXzEgd291bGQgdXNlIHRoZSBsZWdhY3kg aW50ZXJmYWNlcyBjYXB0dXJlZCBpbiB0aGUgc3BlYyBzZWN0aW9uCj4+IDcuNCAocmVnYXJkaW5n IHJpbmcgbGF5b3V0LCBuYXRpdmUgZW5kaWFubmVzcywgbWVzc2FnZSBmcmFtaW5nLCB2cQo+PiBh bGlnbm1lbnQgb2YgNDA5NiwgMzJiaXQgZmVhdHVyZSwgbm8gZmVhdHVyZXNfb2sgYml0IGluIHN0 YXR1cywgSU8gcG9ydAo+PiBpbnRlcmZhY2UgaS5lLiBhbGwgdGhlIHRoaW5ncykgaW5zdGVhZD8g Tm90ZWQgd2UgZG9uJ3QgeWV0IGhhdmUgYQo+PiBzZXRfZGV2aWNlX2ZlYXR1cmVzKCkgdGhhdCBh bGxvd3MgdGhlIHZkcGEgZGV2aWNlIHRvIHRlbGwgd2hldGhlciBpdCBpcwo+PiBvcGVyYXRpbmcg aW4gdHJhbnNpdGlvbmFsIG9yIG1vZGVybi1vbmx5IG1vZGUuIEZvciBzb2Z0d2FyZSB2aXJ0aW8s IGFsbAo+PiBzdXBwb3J0IGZvciB0aGUgbGVnYWN5IHBhcnQgaW4gYSB0cmFuc2l0aW9uYWwgbW9k ZWwgaGFzIGJlZW4gYnVpbHQgdXAKPj4gdGhlcmUgYWxyZWFkeSwgaG93ZXZlciwgaXQncyBub3Qg ZWFzeSBmb3IgdkRQQSB2ZW5kb3JzIHRvIGltcGxlbWVudCBhbGwKPj4gdGhlIHJlcXVpcmVtZW50 cyBmb3IgYW4gYWxsLW9yLW5vdGhpbmcgbGVnYWN5IGd1ZXN0IHN1cHBvcnQgKGJpZyBlbmRpYW4K Pj4gZ3Vlc3QgZm9yIGV4YW1wbGUpLiBUbyB0aGVzZSB2ZW5kb3JzLCB0aGUgbGVnYWN5IHN1cHBv cnQgd2l0aGluIGEKPj4gdHJhbnNpdGlvbmFsIG1vZGVsIGlzIG1vcmUgb2YgZmVhdHVyZSB0byB0 aGVtIGFuZCBpdCdzIGJlc3QgdG8gbGVhdmUKPj4gc29tZSBmbGV4aWJpbGl0eSBmb3IgdGhlbSB0 byBpbXBsZW1lbnQgcGFydGlhbCBzdXBwb3J0IGZvciBsZWdhY3kuIFRoYXQKPj4gaW4gdHVybiBj YWxscyBvdXQgdGhlIG5lZWQgZm9yIGEgdmhvc3QtdXNlciBwcm90b2NvbCBmZWF0dXJlIGxpa2UK Pj4gbmVnb3RpYXRpb24gQVBJIHRoYXQgY2FuIHByb2hpYml0IHRob3NlIHVuc3VwcG9ydGVkIGd1 ZXN0IHNldHVwcyB0byBhcwo+PiBlYXJseSBhcyBiYWNrZW5kX2luaXQgYmVmb3JlIGxhdW5jaGlu ZyB0aGUgVk0uCj4+Cj4+Cj4+Pj4+PiBJCj4+Pj4+PiBjaGVja2VkIHdpdGggRWxpIGFuZCBvdGhl ciBNZWxsYW5veC9OVkRJQSBmb2xrcyBmb3IgaGFyZHdhcmUvZmlybXdhcmUgbGV2ZWwKPj4+Pj4+ IDAuOTUgc3VwcG9ydCwgaXQgc2VlbXMgYWxsIHRoZSBpbmdyZWRpZW50IGhhZCBiZWVuIHRoZXJl IGFscmVhZHkgZGF0ZWQgYmFjawo+Pj4+Pj4gdG8gdGhlIERQREsgZGF5cy4gVGhlIG9ubHkgbWFq b3IgdGhpbmcgbGltaXRpbmcgaXMgaW4gdGhlIHZEUEEgc29mdHdhcmUgdGhhdAo+Pj4+Pj4gdGhl IGN1cnJlbnQgdmRwYSBjb3JlIGhhcyB0aGUgYXNzdW1wdGlvbiBhcm91bmQgVklSVElPX0ZfQUND RVNTX1BMQVRGT1JNIGZvcgo+Pj4+Pj4gYSBmZXcgRE1BIHNldHVwIG9wcywgd2hpY2ggaXMgdmly dGlvIDEuMCBvbmx5Lgo+Pj4+Pj4KPj4+Pj4+Pj4gMi4gc3VwcG9zZSBzb21lIGZvcm0gb2YgbGVn YWN5IGd1ZXN0IHN1cHBvcnQgbmVlZHMgdG8gYmUgdGhlcmUsIGhvdyBkbyB3ZQo+Pj4+Pj4+PiBk ZWFsIHdpdGggdGhlIGJvZ3VzIGFzc3VtcHRpb24gYmVsb3cgaW4gdmRwYV9nZXRfY29uZmlnKCkg aW4gdGhlIHNob3J0IHRlcm0/Cj4+Pj4+Pj4+IEl0IGxvb2tzIG9uZSBvZiB0aGUgaW50dWl0aXZl IGZpeCBpcyB0byBtb3ZlIHRoZSB2ZHBhX3NldF9mZWF0dXJlcyBjYWxsIG91dAo+Pj4+Pj4+PiBv ZiB2ZHBhX2dldF9jb25maWcoKSB0byB2ZHBhX3NldF9jb25maWcoKS4KPj4+Pj4+Pj4KPj4+Pj4+ Pj4gICAgICAgICAgICAgLyoKPj4+Pj4+Pj4gICAgICAgICAgICAgICogQ29uZmlnIGFjY2Vzc2Vz IGFyZW4ndCBzdXBwb3NlZCB0byB0cmlnZ2VyIGJlZm9yZSBmZWF0dXJlcyBhcmUKPj4+Pj4+Pj4g c2V0Lgo+Pj4+Pj4+PiAgICAgICAgICAgICAgKiBJZiBpdCBkb2VzIGhhcHBlbiB3ZSBhc3N1bWUg YSBsZWdhY3kgZ3Vlc3QuCj4+Pj4+Pj4+ICAgICAgICAgICAgICAqLwo+Pj4+Pj4+PiAgICAgICAg ICAgICBpZiAoIXZkZXYtPmZlYXR1cmVzX3ZhbGlkKQo+Pj4+Pj4+PiAgICAgICAgICAgICAgICAg ICAgIHZkcGFfc2V0X2ZlYXR1cmVzKHZkZXYsIDApOwo+Pj4+Pj4+PiAgICAgICAgICAgICBvcHMt PmdldF9jb25maWcodmRldiwgb2Zmc2V0LCBidWYsIGxlbik7Cj4+Pj4+Pj4+Cj4+Pj4+Pj4+IEkg Y2FuIHBvc3QgYSBwYXRjaCB0byBmaXggMikgaWYgdGhlcmUncyBjb25zZW5zdXMgYWxyZWFkeSBy ZWFjaGVkLgo+Pj4+Pj4+Pgo+Pj4+Pj4+PiBUaGFua3MsCj4+Pj4+Pj4+IC1TaXdlaQo+Pj4+Pj4+ IEknbSBub3Qgc3VyZSBob3cgaW1wb3J0YW50IGl0IGlzIHRvIGNoYW5nZSB0aGF0Lgo+Pj4+Pj4+ IEluIGFueSBjYXNlIGl0IG9ubHkgYWZmZWN0cyB0cmFuc2l0aW9uYWwgZGV2aWNlcywgcmlnaHQ/ Cj4+Pj4+Pj4gTGVnYWN5IG9ubHkgc2hvdWxkIG5vdCBjYXJlIC4uLgo+Pj4+Pj4gWWVzIEknZCBs aWtlIHRvIGRpc3Rpbmd1aXNoIGxlZ2FjeSBkcml2ZXIgKHN1cHBvc2UgaXQgaXMgMC45NSkgYWdh aW5zdCB0aGUKPj4+Pj4+IG1vZGVybiBvbmUgaW4gYSB0cmFuc2l0aW9uYWwgZGV2aWNlIG1vZGVs IHJhdGhlciB0aGFuIGJlaW5nIGxlZ2FjeSBvbmx5Lgo+Pj4+Pj4gVGhhdCB3YXkgYSB2MC45NSBh bmQgdjEuMCBzdXBwb3J0aW5nIHZkcGEgcGFyZW50IGNhbiBzdXBwb3J0IGJvdGggdHlwZXMgb2YK Pj4+Pj4+IGd1ZXN0cyB3aXRob3V0IGhhdmluZyB0byByZWNvbmZpZ3VyZS4gT3IgYXJlIHlvdSBz dWdnZXN0aW5nIGxpbWl0IHRvIGxlZ2FjeQo+Pj4+Pj4gb25seSBhdCB0aGUgdGltZSBvZiB2ZHBh IGNyZWF0aW9uIHdvdWxkIHNpbXBsaWZ5IHRoZSBpbXBsZW1lbnRhdGlvbiBhIGxvdD8KPj4+Pj4+ Cj4+Pj4+PiBUaGFua3MsCj4+Pj4+PiAtU2l3ZWkKPj4+Pj4gSSBkb24ndCBrbm93IGZvciBzdXJl LiBUYWtlIGEgbG9vayBhdCB0aGUgd29yayBIYWxpbCB3YXMgZG9pbmcKPj4+Pj4gdG8gdHJ5IGFu ZCBzdXBwb3J0IHRyYW5zaXRpb25hbCBkZXZpY2VzIHdpdGggQkUgZ3Vlc3RzLgo+Pj4+IEhtbW0s IHdlIGNhbiBoYXZlIHRob3NlIGVuZGlhbm5lc3MgaW9jdGxzIGRlZmluZWQgYnV0IHRoZSBpbml0 aWFsIFFFTVUKPj4+PiBpbXBsZW1lbnRhdGlvbiBjYW4gYmUgc3RhcnRlZCB0byBzdXBwb3J0IHg4 NiBndWVzdC9ob3N0IHdpdGggbGl0dGxlCj4+Pj4gZW5kaWFuIGFuZCB3ZWFrIG1lbW9yeSBvcmRl cmluZyBmaXJzdC4gVGhlIHJlYWwgdHJpY2sgaXMgdG8gZGV0ZWN0Cj4+Pj4gbGVnYWN5IGd1ZXN0 IC0gSSBhbSBub3Qgc3VyZSBpZiBpdCdzIGZlYXNpYmxlIHRvIHNoaWZ0IGFsbCB0aGUgbGVnYWN5 Cj4+Pj4gZGV0ZWN0aW9uIHdvcmsgdG8gUUVNVSwgb3IgdGhlIGtlcm5lbCBoYXMgdG8gYmUgcGFy dCBvZiB0aGUgZGV0ZWN0aW9uCj4+Pj4gKGUuZy4gdGhlIGtpY2sgYmVmb3JlIERSSVZFUl9PSyB0 aGluZyB3ZSBoYXZlIHRvIGR1cGxpY2F0ZSB0aGUgdHJhY2tpbmcKPj4+PiBlZmZvcnQgaW4gUUVN VSkgYXMgd2VsbC4gTGV0IG1lIHRha2UgYSBmdXJ0aGVyIGxvb2sgYW5kIGdldCBiYWNrLgo+Pj4g TWljaGFlbCBtYXkgdGhpbmsgZGlmZmVyZW50bHkgYnV0IEkgdGhpbmsgZG9pbmcgdGhpcyBpbiBR ZW11IGlzIG11Y2ggZWFzaWVyLgo+PiBJIHRoaW5rIHRoZSBrZXkgaXMgd2hldGhlciB3ZSBwb3Np dGlvbiBlbXVsYXRpbmcgbGVnYWN5IGludGVyZmFjZXMgaW4KPj4gUUVNVSBkb2luZyB0cmFuc2xh dGlvbiBvbiB0b3Agb2YgYSB2MS4wIG1vZGVybi1vbmx5IGRldmljZSBpbiB0aGUKPj4ga2VybmVs LCBvciB3ZSBhbGxvdyB2ZHBhIGNvcmUgKG9yIHlvdSBjYW4gc2F5IHZob3N0LXZkcGEpIGFuZCB2 ZW5kb3IKPj4gZHJpdmVyIHRvIHN1cHBvcnQgYSB0cmFuc2l0aW9uYWwgbW9kZWwgaW4gdGhlIGtl cm5lbCB0aGF0IGlzIGFibGUgdG8KPj4gd29yayBmb3IgYm90aCB2MC45NSBhbmQgdjEuMCBkcml2 ZXJzLCB3aXRoIHNvbWUgc2xpZ2h0IGFpZCBmcm9tIFFFTVUgZm9yCj4+IGRldGVjdGluZy9lbXVs YXRpb24vc2hhZG93aW5nIChmb3IgZS5nIENWUSwgSS9PIHBvcnQgcmVsYXkpLiBJIGd1ZXNzIGZv cgo+PiB0aGUgZm9ybWVyIHdlIHN0aWxsIHJlbHkgb24gdmVuZG9yIGZvciBhIHBlcmZvcm1hbnQg ZGF0YSB2cXMKPj4gaW1wbGVtZW50YXRpb24sIGxlYXZpbmcgdGhlIHF1ZXN0aW9uIHRvIHdoYXQg aXQgbWF5IGVuZCB1cCBldmVudHVhbGx5IGluCj4+IHRoZSBrZXJuZWwgaXMgZWZmZWN0aXZlbHkg dGhlIGxhdHRlcikuCj4gSSB0aGluayB3ZSBjYW4gZG8gdGhlIGxlZ2FjeSBpbnRlcmZhY2UgZW11 bGF0aW9uIG9uIHRvcCBvZiB0aGUgc2hhZG93Cj4gVlEuIEFuZCB3ZSBrbm93IGl0IHdvcmtzIGZv ciBzdXJlLiBCdXQgSSBhZ3JlZSwgaXQgd291bGQgYmUgbXVjaAo+IGVhc2llciBpZiB3ZSBkZXBl bmQgb24gdGhlIHZlbmRvciB0byBpbXBsZW1lbnQgYSB0cmFuc2l0aW9uYWwgZGV2aWNlLgpGaXJz dCBJIGFtIG5vdCBzdXJlIGlmIHRoZXJlJ3MgYSBjb252aW5jaW5nIGNhc2UgZm9yIHVzZXJzIHRv IGRlcGxveSAKdkRQQSB3aXRoIHNoYWRvdyAoZGF0YSkgVlEgYWdhaW5zdCB0aGUgcHVyZSBzb2Z0 d2FyZSBiYXNlZCBiYWNrZW5kLiAKUGxlYXNlIGVubGlnaHRlbiBtZSBpZiB0aGVyZSBpcy4KCkZv ciB1cywgdGhlIHBvaW50IHRvIGRlcGxveSB2RFBBIGZvciBsZWdhY3kgZ3Vlc3QgaXMgdGhlIGFj Y2VsZXJhdGlvbiAKKHdoYXQgIkEiIHN0YW5kcyBmb3IgaW4gInZEUEEiKSBwYXJ0IG9mIGl0IHNv IHRoYXQgd2UgY2FuIGxldmVyYWdlIHRoZSAKaGFyZHdhcmUgcG90ZW50aWFsIGlmIGF0IGFsbCBw b3NzaWJsZS4gTm90IHN1cmUgaG93IHRoZSBzaGFkb3cgVlEgCmltcGxlbWVudGF0aW9uIGNhbiBl YXNpbHkgZGVhbCB3aXRoIGRhdGFwYXRoIGFjY2VsZXJhdGlvbiB3aXRob3V0IGxvc2luZyAKdG9v IG11Y2ggcGVyZm9ybWFuY2U/Cgo+IFNvIGFzc3VtaW5nIHdlIGRlcGVuZCBvbiB0aGUgdmVuZG9y LCBJIGRvbid0IHNlZSBhbnl0aGluZyB0aGF0IGlzCj4gc3RyaWN0bHkgbmVlZGVkIGluIHRoZSBr ZXJuZWwsIHRoZSBraWNrIG9yIGNvbmZpZyBhY2Nlc3MgYmVmb3JlCj4gRFJJVkVSX09LIGNhbiBh bGwgYmUgaGFuZGxlZCBlYXNpbHkgaW4gUWVtdSB1bmxlc3MgSSBtaXNzIHNvbWV0aGluZy4KUmln aHQsIHRoYXQncyB3aGF0IEkgdGhpbmsgdG9vIGl0J3Mgbm90IHF1aXRlIGEgbG90IG9mIHdvcmsg aW4gdGhlIAprZXJuZWwgaWYgdmVuZG9yIGRldmljZSBvZmZlcnMgdGhlIGFpZC9zdXBwb3J0IGZv ciB0cmFuc2l0aW9uYWwuIFRoZSAKa2VybmVsIG9ubHkgcHJvdmlkZXMgdGhlIGFic3RyYWN0aW9u IG9mIGRldmljZSBtb2RlbCAodHJhbnNpdGlvbmFsIG9yIAptb2Rlcm4tb25seSksIHdoaWxlIHZl bmRvciBkcml2ZXIgbWF5IGltcGxlbWVudCBlYXJseSBwbGF0Zm9ybSBmZWF0dXJlIApkaXNjb3Zl cnkgYW5kIGFwcGx5IGxlZ2FjeSBzcGVjaWZpYyBxdWlya3MgKHVuc3VwcG9ydGVkIGVuZGlhbm5l c3MsIAptaXNtYXRjaGVkIHBhZ2Ugc2l6ZSwgdW5zdXBwb3J0ZWQgaG9zdCBtZW1vcnkgb3JkZXJp bmcgbW9kZWwpIHRoYXQgdGhlIApkZXZpY2UgY2FuJ3QgYWRhcHQgdG8uIEkgZG9uJ3Qgc2F5IHdl IGhhdmUgdG8gZGVwZW5kIG9uIHRoZSB2ZW5kb3IsIGJ1dCAKdGhlIHBvaW50IGlzIHRoYXQgd2Ug bXVzdCBhc3N1bWUgZnVsbHkgc3BlYyBjb21wbGlhbnQgdHJhbnNpdGlvbmFsIApzdXBwb3J0ICh0 aGUgZGF0YXBhdGggaW4gcGFydGljdWxhcikgZnJvbSB0aGUgdmVuZG9yIHRvIGdldCBzdGFydGVk LCBhcyAKSSBndWVzcyBpdCdzIHByb2JhYmx5IHRoZSBtYWluIG1vdGl2YXRpb24gZm9yIHVzZXJz IHRvIGRlcGxveSBpdCAtIAphY2NlbGVyYXRpb24gb2YgbGVnYWN5IGd1ZXN0IHdvcmtsb2FkIHdp dGhvdXQgZXhoYXVzdGluZyBob3N0IGNvbXB1dGluZyAKcmVzb3VyY2UuIEV2ZW4gaWYgd2UgZ2V0 IHN0YXJ0ZWQgd2l0aCBzaGFkb3cgVlEgdG8gbWVkaWF0ZSBhbmQgdHJhbnNsYXRlIAp0aGUgZGF0 YXBhdGgsIGV2ZW50dWFsbHkgaXQgbWF5IGV2b2x2ZSB0b3dhcmRzIGxldmVyYWdpbmcgZGF0YXBh dGggCm9mZmxvYWQgdG8gaGFyZHdhcmUgaWYgYWNjZWxlcmF0aW9uIGlzIHRoZSBvbmx5IGNvbnZp bmNpbmcgdXNlIGNhc2UgZm9yIApsZWdhY3kgc3VwcG9ydC4KClRoYW5rcywKLVNpd2VpCj4gVGhl IG9ubHkgdmFsdWUgdG8gZG8gdGhhdCBpbiB0aGUga2VybmVsIGlzIHRoYXQgaXQgY2FuIHdvcmsg Zm9yCj4gdmlydGlvLXZkcGEsIGJ1dCBtb2Rlcm4gb25seSB2aXJpdG8tdnBkYSBpcyBzdWZmaWNp ZW50OyB3ZSBkb24ndCBuZWVkCj4gYW55IGxlZ2FjeSBzdHVmZiBmb3IgdGhhdC4KPgo+IFRoYW5r cwo+Cj4+IFRoYW5rcywKPj4gLVNpd2VpCj4+Cj4+PiBUaGFua3MKPj4+Cj4+Pgo+Pj4KPj4+PiBN ZWFud2hpbGUsIEknbGwgY2hlY2sgaW50ZXJuYWxseSB0byBzZWUgaWYgYSBsZWdhY3kgb25seSBt b2RlbCB3b3VsZAo+Pj4+IHdvcmsuIFRoYW5rcy4KPj4+Pgo+Pj4+IFRoYW5rcywKPj4+PiAtU2l3 ZWkKPj4+Pgo+Pj4+Cj4+Pj4+Pj4+IE9uIDMvMi8yMDIxIDI6NTMgQU0sIEphc29uIFdhbmcgd3Jv dGU6Cj4+Pj4+Pj4+PiBPbiAyMDIxLzMvMiA1OjQ3IOS4i+WNiCwgTWljaGFlbCBTLiBUc2lya2lu IHdyb3RlOgo+Pj4+Pj4+Pj4+IE9uIE1vbiwgTWFyIDAxLCAyMDIxIGF0IDExOjU2OjUwQU0gKzA4 MDAsIEphc29uIFdhbmcgd3JvdGU6Cj4+Pj4+Pj4+Pj4+IE9uIDIwMjEvMy8xIDU6MzQg5LiK5Y2I LCBNaWNoYWVsIFMuIFRzaXJraW4gd3JvdGU6Cj4+Pj4+Pj4+Pj4+PiBPbiBXZWQsIEZlYiAyNCwg MjAyMSBhdCAxMDoyNDo0MUFNIC0wODAwLCBTaS1XZWkgTGl1IHdyb3RlOgo+Pj4+Pj4+Pj4+Pj4+ PiBEZXRlY3RpbmcgaXQgaXNuJ3QgZW5vdWdoIHRob3VnaCwgd2Ugd2lsbCBuZWVkIGEgbmV3IGlv Y3RsIHRvIG5vdGlmeQo+Pj4+Pj4+Pj4+Pj4+PiB0aGUga2VybmVsIHRoYXQgaXQncyBhIGxlZ2Fj eSBndWVzdC4gVWdoIDooCj4+Pj4+Pj4+Pj4+Pj4gV2VsbCwgYWx0aG91Z2ggSSB0aGluayBhZGRp bmcgYW4gaW9jdGwgaXMgZG9hYmxlLCBtYXkgSQo+Pj4+Pj4+Pj4+Pj4+IGtub3cgd2hhdCB0aGUg dXNlCj4+Pj4+Pj4+Pj4+Pj4gY2FzZSB0aGVyZSB3aWxsIGJlIGZvciBrZXJuZWwgdG8gbGV2ZXJh Z2Ugc3VjaCBpbmZvCj4+Pj4+Pj4+Pj4+Pj4gZGlyZWN0bHk/IElzIHRoZXJlIGEKPj4+Pj4+Pj4+ Pj4+PiBjYXNlIFFFTVUgY2FuJ3QgZG8gd2l0aCBkZWRpY2F0ZSBpb2N0bHMgbGF0ZXIgaWYgdGhl cmUncyBpbmRlZWQKPj4+Pj4+Pj4+Pj4+PiBkaWZmZXJlbnRpYXRpb24gKGxlZ2FjeSB2LnMuIG1v ZGVybikgbmVlZGVkPwo+Pj4+Pj4+Pj4+Pj4gQlRXIGEgZ29vZCBBUEkgY291bGQgYmUKPj4+Pj4+ Pj4+Pj4+Cj4+Pj4+Pj4+Pj4+PiAjZGVmaW5lIFZIT1NUX1NFVF9FTkRJQU4gX0lPVyhWSE9TVF9W SVJUSU8sID8sIGludCkKPj4+Pj4+Pj4+Pj4+ICNkZWZpbmUgVkhPU1RfR0VUX0VORElBTiBfSU9X KFZIT1NUX1ZJUlRJTywgPywgaW50KQo+Pj4+Pj4+Pj4+Pj4KPj4+Pj4+Pj4+Pj4+IHdlIGRpZCBp dCBwZXIgdnJpbmcgYnV0IG1heWJlIHRoYXQgd2FzIGEgbWlzdGFrZSAuLi4KPj4+Pj4+Pj4+Pj4g QWN0dWFsbHksIEkgd29uZGVyIHdoZXRoZXIgaXQncyBnb29kIHRpbWUgdG8ganVzdCBub3Qgc3Vw cG9ydAo+Pj4+Pj4+Pj4+PiBsZWdhY3kgZHJpdmVyCj4+Pj4+Pj4+Pj4+IGZvciB2RFBBLiBDb25z aWRlcjoKPj4+Pj4+Pj4+Pj4KPj4+Pj4+Pj4+Pj4gMSkgSXQncyBkZWZpbml0aW9uIGlzIG5vLW5v cm1hdGl2ZQo+Pj4+Pj4+Pj4+PiAyKSBBIGxvdCBvZiBidWRyZW4gb2YgY29kZXMKPj4+Pj4+Pj4+ Pj4KPj4+Pj4+Pj4+Pj4gU28gcWVtdSBjYW4gc3RpbGwgcHJlc2VudCB0aGUgbGVnYWN5IGRldmlj ZSBzaW5jZSB0aGUgY29uZmlnCj4+Pj4+Pj4+Pj4+IHNwYWNlIG9yIG90aGVyCj4+Pj4+Pj4+Pj4+ IHN0dWZmcyB0aGF0IGlzIHByZXNlbnRlZCBieSB2aG9zdC12RFBBIGlzIG5vdCBleHBlY3RlZCB0 byBiZQo+Pj4+Pj4+Pj4+PiBhY2Nlc3NlZCBieQo+Pj4+Pj4+Pj4+PiBndWVzdCBkaXJlY3RseS4g UWVtdSBjYW4gZG8gdGhlIGVuZGlhbiBjb252ZXJzaW9uIHdoZW4gbmVjZXNzYXJ5Cj4+Pj4+Pj4+ Pj4+IGluIHRoaXMKPj4+Pj4+Pj4+Pj4gY2FzZT8KPj4+Pj4+Pj4+Pj4KPj4+Pj4+Pj4+Pj4gVGhh bmtzCj4+Pj4+Pj4+Pj4+Cj4+Pj4+Pj4+Pj4gT3ZlcmFsbCBJIHdvdWxkIGJlIGZpbmUgd2l0aCB0 aGlzIGFwcHJvYWNoIGJ1dCB3ZSBuZWVkIHRvIGF2b2lkIGJyZWFraW5nCj4+Pj4+Pj4+Pj4gd29y a2luZyB1c2Vyc3BhY2UsIHFlbXUgcmVsZWFzZXMgd2l0aCB2ZHBhIHN1cHBvcnQgYXJlIG91dCB0 aGVyZSBhbmQKPj4+Pj4+Pj4+PiBzZWVtIHRvIHdvcmsgZm9yIHBlb3BsZS4gQW55IGNoYW5nZXMg bmVlZCB0byB0YWtlIHRoYXQgaW50byBhY2NvdW50Cj4+Pj4+Pj4+Pj4gYW5kIGRvY3VtZW50IGNv bXBhdGliaWxpdHkgY29uY2VybnMuCj4+Pj4+Pj4+PiBBZ3JlZSwgbGV0IG1lIGNoZWNrLgo+Pj4+ Pj4+Pj4KPj4+Pj4+Pj4+Cj4+Pj4+Pj4+Pj4gICAgICAgSSBub3RlIHRoYXQgYW55IGhhcmR3YXJl Cj4+Pj4+Pj4+Pj4gaW1wbGVtZW50YXRpb24gaXMgYWxyZWFkeSBicm9rZW4gZm9yIGxlZ2FjeSBl eGNlcHQgb24gcGxhdGZvcm1zIHdpdGgKPj4+Pj4+Pj4+PiBzdHJvbmcgb3JkZXJpbmcgd2hpY2gg bWlnaHQgYmUgaGVscGZ1bCBpbiByZWR1Y2luZyB0aGUgc2NvcGUuCj4+Pj4+Pj4+PiBZZXMuCj4+ Pj4+Pj4+Pgo+Pj4+Pj4+Pj4gVGhhbmtzCj4+Pj4+Pj4+Pgo+Pj4+Pj4+Pj4KCl9fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fClZpcnR1YWxpemF0aW9uIG1haWxp bmcgbGlzdApWaXJ0dWFsaXphdGlvbkBsaXN0cy5saW51eC1mb3VuZGF0aW9uLm9yZwpodHRwczov L2xpc3RzLmxpbnV4Zm91bmRhdGlvbi5vcmcvbWFpbG1hbi9saXN0aW5mby92aXJ0dWFsaXphdGlv bg==