From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.6 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_2 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 209C9C4708F for ; Wed, 2 Jun 2021 20:37:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0028861008 for ; Wed, 2 Jun 2021 20:37:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229667AbhFBUjX (ORCPT ); Wed, 2 Jun 2021 16:39:23 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:47477 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229587AbhFBUjW (ORCPT ); Wed, 2 Jun 2021 16:39:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622666258; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TL5RCeD6URaQW+iNoSmRFKX2AmrwBTs64Rz3fDPVQnU=; b=hkUcOwerbOUHron5RguhWuS74t2Y5bsGy79ijh9Ij3FQmR/SUTJGoUizwaqQGWc+qW4rBk k/Sp70lFXXBybLxVYsFSwUUxK/w4H1/yWzjVRc0QaeMoOWHolXVVjUKF4VY6Y6j49rCzxa yFJRwVb6Pz4hIVl1HqpWc6LXJmIveMk= Received: from mail-oo1-f70.google.com (mail-oo1-f70.google.com [209.85.161.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-104-4nxbtnyeP6qDzjNP-db3Ww-1; Wed, 02 Jun 2021 16:37:37 -0400 X-MC-Unique: 4nxbtnyeP6qDzjNP-db3Ww-1 Received: by mail-oo1-f70.google.com with SMTP id c25-20020a4ad7990000b029020e67cc1879so2137529oou.18 for ; Wed, 02 Jun 2021 13:37:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TL5RCeD6URaQW+iNoSmRFKX2AmrwBTs64Rz3fDPVQnU=; b=B0yIv7TSuP/qYnIdfHYxpyGyhbqmRGNHw/W+RnWXF7wyMY9zWLIknZcVZdZnfr2+9Z J7MMd3EQ1W5oysYdJajvCcgyFtgzZNuyxG2H2e/RHxQk2kchYa3Ydi/yfRvkjAsDHJNR 3/sx2BSRvTHt4wHoFvHLDaDFnbKO7xoeUZJfaEXSzvOiSD497XaI28+x9wN6q/qAIFCB IMxLf31tiMpCQYUacBIYqqnRpoZhkkT9PSxSq0HuwRPS2f+ftT5pDZDM04Gb3T0gnAGU OFf2PFM1S5TEYkvTH2Rd68dNwXApfeXbobkY+vg0qC4lVpXO0KD0STooLx4afFMjRe7t UZ0A== X-Gm-Message-State: AOAM531Zv9M1fMjUC3AhbjoXftJWM3Tduo1pL6V31rxuP2ggM3Kh/7Bh IRgBr5Od/o1qFx7HEE4YqA6T3jo9cOvt2mCl/Ng1SvhzMiHwKj7k91Ufw0C/2hvbhil7HvYax+y Z67xLB2d0k3actdfQIz2qzaUU X-Received: by 2002:a05:6830:18ee:: with SMTP id d14mr2615010otf.347.1622666256092; Wed, 02 Jun 2021 13:37:36 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxTGOm/+QC96xO7+b3hYQUsHRLgYBQr4fcc4vLi2KEeMHHCtYSbE2eryQSUu9BcVRPF7Kuerw== X-Received: by 2002:a05:6830:18ee:: with SMTP id d14mr2614981otf.347.1622666255750; Wed, 02 Jun 2021 13:37:35 -0700 (PDT) Received: from redhat.com ([198.99.80.109]) by smtp.gmail.com with ESMTPSA id t39sm195560ooi.42.2021.06.02.13.37.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Jun 2021 13:37:35 -0700 (PDT) Date: Wed, 2 Jun 2021 14:37:34 -0600 From: Alex Williamson To: Jason Gunthorpe Cc: "Tian, Kevin" , Jean-Philippe Brucker , "Jiang, Dave" , "Raj, Ashok" , "kvm@vger.kernel.org" , Jonathan Corbet , Robin Murphy , LKML , "iommu@lists.linux-foundation.org" , David Gibson , Kirti Wankhede , David Woodhouse , Jason Wang Subject: Re: [RFC] /dev/ioasid uAPI proposal Message-ID: <20210602143734.72fb4fa4.alex.williamson@redhat.com> In-Reply-To: <20210602195404.GI1002214@nvidia.com> References: <20210528200311.GP1002214@nvidia.com> <20210601162225.259923bc.alex.williamson@redhat.com> <20210602160140.GV1002214@nvidia.com> <20210602111117.026d4a26.alex.williamson@redhat.com> <20210602173510.GE1002214@nvidia.com> <20210602120111.5e5bcf93.alex.williamson@redhat.com> <20210602180925.GH1002214@nvidia.com> <20210602130053.615db578.alex.williamson@redhat.com> <20210602195404.GI1002214@nvidia.com> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; x86_64-redhat-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, 2 Jun 2021 16:54:04 -0300 Jason Gunthorpe wrote: > On Wed, Jun 02, 2021 at 01:00:53PM -0600, Alex Williamson wrote: > >=20 > > Right, the device can generate the no-snoop transactions, but it's the > > IOMMU that essentially determines whether those transactions are > > actually still cache coherent, AIUI. =20 >=20 > Wow, this is really confusing stuff in the code. >=20 > At the PCI level there is a TLP bit called no-snoop that is platform > specific. The general intention is to allow devices to selectively > bypass the CPU caching for DMAs. GPUs like to use this feature for > performance. Yes > I assume there is some exciting security issues here. Looks like > allowing cache bypass does something bad inside VMs? Looks like > allowing the VM to use the cache clear instruction that is mandatory > with cache bypass DMA causes some QOS issues? OK. IIRC, largely a DoS issue if userspace gets to choose when to emulate wbinvd rather than it being demanded for correct operation. > So how does it work? >=20 > What I see in the intel/iommu.c is that some domains support "snoop > control" or not, based on some HW flag. This indicates if the > DMA_PTE_SNP bit is supported on a page by page basis or not. >=20 > Since x86 always leans toward "DMA cache coherent" I'm reading some > tea leaves here: >=20 > IOMMU_CAP_CACHE_COHERENCY, /* IOMMU can enforce cache coherent DMA > transactions */ >=20 > And guessing that IOMMUs that implement DMA_PTE_SNP will ignore the > snoop bit in TLPs for IOVA's that have DMA_PTE_SNP set? That's my understanding as well. > Further, I guess IOMMUs that don't support PTE_SNP, or have > DMA_PTE_SNP clear will always honour the snoop bit. (backwards compat > and all) Yes. > So, IOMMU_CAP_CACHE_COHERENCY does not mean the IOMMU is DMA > incoherent with the CPU caches, it just means that that snoop bit in > the TLP cannot be enforced. ie the device *could* do no-shoop DMA > if it wants. Devices that never do no-snoop remain DMA coherent on > x86, as they always have been. Yes, IOMMU_CAP_CACHE_COHERENCY=3Dfalse means we cannot force the device DMA to be coherent via the IOMMU. > IOMMU_CACHE does not mean the IOMMU is DMA cache coherent, it means > the PCI device is blocked from using no-snoop in its TLPs. >=20 > I wonder if ARM implemented this consistently? I see VDPA is > confused.. I was confused. What a terrible set of names. >=20 > In VFIO generic code I see it always sets IOMMU_CACHE: >=20 > if (iommu_capable(bus, IOMMU_CAP_CACHE_COHERENCY)) > domain->prot |=3D IOMMU_CACHE; >=20 > And thus also always provides IOMMU_CACHE to iommu_map: >=20 > ret =3D iommu_map(d->domain, iova, (phys_addr_t)pfn << PA= GE_SHIFT, > npage << PAGE_SHIFT, prot | d->prot); >=20 > So when the IOMMU supports the no-snoop blocking security feature VFIO > turns it on and blocks no-snoop to all pages? Ok.. Yep, I'd forgotten this nuance that we need to enable it via the mapping flags. > But I must be missing something big because *something* in the IOVA > map should work with no-snoopable DMA, right? Otherwise what is the > point of exposing the invalidate instruction to the guest? >=20 > I would think userspace should be relaying the DMA_PTE_SNP bit from > the guest's page tables up to here?? >=20 > The KVM hookup is driven by IOMMU_CACHE which is driven by > IOMMU_CAP_CACHE_COHERENCY. So we turn on the special KVM support only > if the IOMMU can block the SNP bit? And then we map all the pages to > block the snoop bit? Huh? Right. I don't follow where you're jumping to relaying DMA_PTE_SNP from the guest page table... what page table? We don't necessarily have a vIOMMU to expose such things, I don't think it even existed when this we added. Essentially if we can ignore no-snoop at the IOMMU, then KVM doesn't need to worry about emulating wbinvd because of an assigned device, whether that device uses it or not. Win-win. > Your explanation makes perfect sense: Block guests from using the > dangerous cache invalidate instruction unless a device that uses > no-snoop is plugged in. Block devices from using no-snoop because > something about it is insecure. Ok. No-snoop itself is not insecure, but to support no-snoop in a VM KVM can't ignore wbinvd, which has overhead and abuse implications. > But the conditions I'm looking for "device that uses no-snoop" is: > - The device will issue no-snoop TLPs at all We can't really know this generically. We can try to set the enable bit to see if the device is capable of no-snoop, but that doesn't mean it will use no-snoop. > - The IOMMU will let no-snoop through > - The platform will honor no-snoop >=20 > Only if all three are met we should allow the dangerous instruction in > KVM, right? We test at the IOMMU and assume that the IOMMU knowledge encompasses whether the platform honors no-snoop (note for example how amd and arm report true for IOMMU_CAP_CACHE_COHERENCY but seem to ignore the IOMMU_CACHE flag). We could probably use an iommu_group_for_each_dev to test if any devices within the group are capable of no-snoop if the IOMMU can't protect us, but at the time it didn't seem worthwhile. I'm still not sure if it is. =20 > Which brings me back to my original point - this is at least partially > a device specific behavior. It depends on the content of the IOMMU > page table, it depends if the device even supports no-snoop at all. >=20 > My guess is this works correctly for the mdev Intel kvmgt which > probably somehow allows no-snoop DMA throught the mdev SW iommu > mappings. (assuming I didn't miss a tricky iommu_map without > IOMMU_CACHe set in the type1 code?) This support existed before mdev, IIRC we needed it for direct assignment of NVIDIA GPUs. =20 > But why is vfio-pci using it? Hmm? Use the IOMMU to reduce hypervisor overhead, let the hypervisor learn about it, ignore the subtleties of whether the device actually uses no-snoop as imprecise and poor ROI given the apparent direction of hardware. =C2=AF\_(=E3=83=84)_/=C2=AF, Alex From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_2 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6D79C4708F for ; Wed, 2 Jun 2021 20:37:44 +0000 (UTC) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 62CC261008 for ; Wed, 2 Jun 2021 20:37:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 62CC261008 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 28B6640544; Wed, 2 Jun 2021 20:37:44 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CghHXDuFGbKK; Wed, 2 Jun 2021 20:37:43 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTP id 863FF4054A; Wed, 2 Jun 2021 20:37:42 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 5F72BC000E; Wed, 2 Jun 2021 20:37:42 +0000 (UTC) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id C707CC0001 for ; Wed, 2 Jun 2021 20:37:40 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id AD84183BD6 for ; Wed, 2 Jun 2021 20:37:40 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id P1rQdIsy7pPi for ; Wed, 2 Jun 2021 20:37:39 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id 846D982D85 for ; Wed, 2 Jun 2021 20:37:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622666258; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iJlh51bjI+f1OqmoNY6RCwcvjRmGzSQVTBsY3gR9pCY=; b=Gc45c6gv4R+E/WR7JCYX04i8cPkSwdF7ggoPYA6pOX0kG2jcDkYI3v0gE/+ZGI1brtbN5i hBZh/vNItx5RSkL3dDVChaN1FRPEsTKa91L7wdnSdxQTMc3H0ieclUwzRYfucrfoOHubZe e20ZijD2MjLRAHI+6lRPFes0AY8Y7WQ= Received: from mail-oi1-f199.google.com (mail-oi1-f199.google.com [209.85.167.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-490-ZQIGyCh8O9-QHAo1wDTjyg-1; Wed, 02 Jun 2021 16:37:37 -0400 X-MC-Unique: ZQIGyCh8O9-QHAo1wDTjyg-1 Received: by mail-oi1-f199.google.com with SMTP id e5-20020aca23050000b02901f0c4e00a29so1762270oie.18 for ; Wed, 02 Jun 2021 13:37:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TL5RCeD6URaQW+iNoSmRFKX2AmrwBTs64Rz3fDPVQnU=; b=DQyY5FbnSEaajkMPzkE62IQR29dg0FpARsfTqFj+2ELRJ8dpbjNjwVTr5uAmMe/01f o+JXd/nmEvi++Q1DWn716l3P4sYVqPeuznCBd6TCBsX33dnv3A0VxVbAPrxxD4bdcAOV edHnqoFs0qeT+FjeZAVtdgpbVAPAlO7xNiMAxaRlKKPLaZ6iRWZevw0Z3c6wBJz2aNEN f0vCoRVAK9eJljXy7RmwEMDjiYszdkVAY62q8/jQKfaI24CdgafmHEB5CbfZTbtDsf0/ ZflU1My/ZxuQVTg8hqwfPo5/L061y2gmHtn2lxEUtQd/4ypkAY0wEtzVc5piC0hcoB6d vFCA== X-Gm-Message-State: AOAM530oaSbcZymuneBaKK4uNeJC5eYP27IGJQl0sdAYkviwO4uWYBuj T9NyzMTDNk0k+mCjWo9czJ9PTnuo0zN3RoxL3jigkozvX0nKihubq62nYO67FIq7OGbgx+GItpK sB4bg0McONC9E1lG4eAOEIFOYiCFuNg== X-Received: by 2002:a05:6830:18ee:: with SMTP id d14mr2615013otf.347.1622666256093; Wed, 02 Jun 2021 13:37:36 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxTGOm/+QC96xO7+b3hYQUsHRLgYBQr4fcc4vLi2KEeMHHCtYSbE2eryQSUu9BcVRPF7Kuerw== X-Received: by 2002:a05:6830:18ee:: with SMTP id d14mr2614981otf.347.1622666255750; Wed, 02 Jun 2021 13:37:35 -0700 (PDT) Received: from redhat.com ([198.99.80.109]) by smtp.gmail.com with ESMTPSA id t39sm195560ooi.42.2021.06.02.13.37.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Jun 2021 13:37:35 -0700 (PDT) Date: Wed, 2 Jun 2021 14:37:34 -0600 From: Alex Williamson To: Jason Gunthorpe Subject: Re: [RFC] /dev/ioasid uAPI proposal Message-ID: <20210602143734.72fb4fa4.alex.williamson@redhat.com> In-Reply-To: <20210602195404.GI1002214@nvidia.com> References: <20210528200311.GP1002214@nvidia.com> <20210601162225.259923bc.alex.williamson@redhat.com> <20210602160140.GV1002214@nvidia.com> <20210602111117.026d4a26.alex.williamson@redhat.com> <20210602173510.GE1002214@nvidia.com> <20210602120111.5e5bcf93.alex.williamson@redhat.com> <20210602180925.GH1002214@nvidia.com> <20210602130053.615db578.alex.williamson@redhat.com> <20210602195404.GI1002214@nvidia.com> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; x86_64-redhat-linux-gnu) MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=alex.williamson@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: Jean-Philippe Brucker , "Tian, Kevin" , "Jiang, Dave" , "Raj, Ashok" , "kvm@vger.kernel.org" , Jonathan Corbet , David Woodhouse , Jason Wang , LKML , Kirti Wankhede , "iommu@lists.linux-foundation.org" , Robin Murphy , David Gibson X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" T24gV2VkLCAyIEp1biAyMDIxIDE2OjU0OjA0IC0wMzAwCkphc29uIEd1bnRob3JwZSA8amdnQG52 aWRpYS5jb20+IHdyb3RlOgoKPiBPbiBXZWQsIEp1biAwMiwgMjAyMSBhdCAwMTowMDo1M1BNIC0w NjAwLCBBbGV4IFdpbGxpYW1zb24gd3JvdGU6Cj4gPiAKPiA+IFJpZ2h0LCB0aGUgZGV2aWNlIGNh biBnZW5lcmF0ZSB0aGUgbm8tc25vb3AgdHJhbnNhY3Rpb25zLCBidXQgaXQncyB0aGUKPiA+IElP TU1VIHRoYXQgZXNzZW50aWFsbHkgZGV0ZXJtaW5lcyB3aGV0aGVyIHRob3NlIHRyYW5zYWN0aW9u cyBhcmUKPiA+IGFjdHVhbGx5IHN0aWxsIGNhY2hlIGNvaGVyZW50LCBBSVVJLiAgCj4gCj4gV293 LCB0aGlzIGlzIHJlYWxseSBjb25mdXNpbmcgc3R1ZmYgaW4gdGhlIGNvZGUuCj4gCj4gQXQgdGhl IFBDSSBsZXZlbCB0aGVyZSBpcyBhIFRMUCBiaXQgY2FsbGVkIG5vLXNub29wIHRoYXQgaXMgcGxh dGZvcm0KPiBzcGVjaWZpYy4gVGhlIGdlbmVyYWwgaW50ZW50aW9uIGlzIHRvIGFsbG93IGRldmlj ZXMgdG8gc2VsZWN0aXZlbHkKPiBieXBhc3MgdGhlIENQVSBjYWNoaW5nIGZvciBETUFzLiBHUFVz IGxpa2UgdG8gdXNlIHRoaXMgZmVhdHVyZSBmb3IKPiBwZXJmb3JtYW5jZS4KClllcwoKPiBJIGFz c3VtZSB0aGVyZSBpcyBzb21lIGV4Y2l0aW5nIHNlY3VyaXR5IGlzc3VlcyBoZXJlLiBMb29rcyBs aWtlCj4gYWxsb3dpbmcgY2FjaGUgYnlwYXNzIGRvZXMgc29tZXRoaW5nIGJhZCBpbnNpZGUgVk1z PyBMb29rcyBsaWtlCj4gYWxsb3dpbmcgdGhlIFZNIHRvIHVzZSB0aGUgY2FjaGUgY2xlYXIgaW5z dHJ1Y3Rpb24gdGhhdCBpcyBtYW5kYXRvcnkKPiB3aXRoIGNhY2hlIGJ5cGFzcyBETUEgY2F1c2Vz IHNvbWUgUU9TIGlzc3Vlcz8gT0suCgpJSVJDLCBsYXJnZWx5IGEgRG9TIGlzc3VlIGlmIHVzZXJz cGFjZSBnZXRzIHRvIGNob29zZSB3aGVuIHRvIGVtdWxhdGUKd2JpbnZkIHJhdGhlciB0aGFuIGl0 IGJlaW5nIGRlbWFuZGVkIGZvciBjb3JyZWN0IG9wZXJhdGlvbi4KCj4gU28gaG93IGRvZXMgaXQg d29yaz8KPiAKPiBXaGF0IEkgc2VlIGluIHRoZSBpbnRlbC9pb21tdS5jIGlzIHRoYXQgc29tZSBk b21haW5zIHN1cHBvcnQgInNub29wCj4gY29udHJvbCIgb3Igbm90LCBiYXNlZCBvbiBzb21lIEhX IGZsYWcuIFRoaXMgaW5kaWNhdGVzIGlmIHRoZQo+IERNQV9QVEVfU05QIGJpdCBpcyBzdXBwb3J0 ZWQgb24gYSBwYWdlIGJ5IHBhZ2UgYmFzaXMgb3Igbm90Lgo+IAo+IFNpbmNlIHg4NiBhbHdheXMg bGVhbnMgdG93YXJkICJETUEgY2FjaGUgY29oZXJlbnQiIEknbSByZWFkaW5nIHNvbWUKPiB0ZWEg bGVhdmVzIGhlcmU6Cj4gCj4gCUlPTU1VX0NBUF9DQUNIRV9DT0hFUkVOQ1ksCS8qIElPTU1VIGNh biBlbmZvcmNlIGNhY2hlIGNvaGVyZW50IERNQQo+IAkJCQkJICAgdHJhbnNhY3Rpb25zICovCj4g Cj4gQW5kIGd1ZXNzaW5nIHRoYXQgSU9NTVVzIHRoYXQgaW1wbGVtZW50IERNQV9QVEVfU05QIHdp bGwgaWdub3JlIHRoZQo+IHNub29wIGJpdCBpbiBUTFBzIGZvciBJT1ZBJ3MgdGhhdCBoYXZlIERN QV9QVEVfU05QIHNldD8KClRoYXQncyBteSB1bmRlcnN0YW5kaW5nIGFzIHdlbGwuCgo+IEZ1cnRo ZXIsIEkgZ3Vlc3MgSU9NTVVzIHRoYXQgZG9uJ3Qgc3VwcG9ydCBQVEVfU05QLCBvciBoYXZlCj4g RE1BX1BURV9TTlAgY2xlYXIgd2lsbCBhbHdheXMgaG9ub3VyIHRoZSBzbm9vcCBiaXQuIChiYWNr d2FyZHMgY29tcGF0Cj4gYW5kIGFsbCkKClllcy4KCj4gU28sIElPTU1VX0NBUF9DQUNIRV9DT0hF UkVOQ1kgZG9lcyBub3QgbWVhbiB0aGUgSU9NTVUgaXMgRE1BCj4gaW5jb2hlcmVudCB3aXRoIHRo ZSBDUFUgY2FjaGVzLCBpdCBqdXN0IG1lYW5zIHRoYXQgdGhhdCBzbm9vcCBiaXQgaW4KPiB0aGUg VExQIGNhbm5vdCBiZSBlbmZvcmNlZC4gaWUgdGhlIGRldmljZSAqY291bGQqIGRvIG5vLXNob29w IERNQQo+IGlmIGl0IHdhbnRzLiBEZXZpY2VzIHRoYXQgbmV2ZXIgZG8gbm8tc25vb3AgcmVtYWlu IERNQSBjb2hlcmVudCBvbgo+IHg4NiwgYXMgdGhleSBhbHdheXMgaGF2ZSBiZWVuLgoKWWVzLCBJ T01NVV9DQVBfQ0FDSEVfQ09IRVJFTkNZPWZhbHNlIG1lYW5zIHdlIGNhbm5vdCBmb3JjZSB0aGUg ZGV2aWNlCkRNQSB0byBiZSBjb2hlcmVudCB2aWEgdGhlIElPTU1VLgoKPiBJT01NVV9DQUNIRSBk b2VzIG5vdCBtZWFuIHRoZSBJT01NVSBpcyBETUEgY2FjaGUgY29oZXJlbnQsIGl0IG1lYW5zCj4g dGhlIFBDSSBkZXZpY2UgaXMgYmxvY2tlZCBmcm9tIHVzaW5nIG5vLXNub29wIGluIGl0cyBUTFBz Lgo+IAo+IEkgd29uZGVyIGlmIEFSTSBpbXBsZW1lbnRlZCB0aGlzIGNvbnNpc3RlbnRseT8gSSBz ZWUgVkRQQSBpcwo+IGNvbmZ1c2VkLi4gSSB3YXMgY29uZnVzZWQuIFdoYXQgYSB0ZXJyaWJsZSBz ZXQgb2YgbmFtZXMuCj4gCj4gSW4gVkZJTyBnZW5lcmljIGNvZGUgSSBzZWUgaXQgYWx3YXlzIHNl dHMgSU9NTVVfQ0FDSEU6Cj4gCj4gICAgICAgICBpZiAoaW9tbXVfY2FwYWJsZShidXMsIElPTU1V X0NBUF9DQUNIRV9DT0hFUkVOQ1kpKQo+ICAgICAgICAgICAgICAgICBkb21haW4tPnByb3QgfD0g SU9NTVVfQ0FDSEU7Cj4gCj4gQW5kIHRodXMgYWxzbyBhbHdheXMgcHJvdmlkZXMgSU9NTVVfQ0FD SEUgdG8gaW9tbXVfbWFwOgo+IAo+ICAgICAgICAgICAgICAgICByZXQgPSBpb21tdV9tYXAoZC0+ ZG9tYWluLCBpb3ZhLCAocGh5c19hZGRyX3QpcGZuIDw8IFBBR0VfU0hJRlQsCj4gICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICBucGFnZSA8PCBQQUdFX1NISUZULCBwcm90IHwgZC0+cHJv dCk7Cj4gCj4gU28gd2hlbiB0aGUgSU9NTVUgc3VwcG9ydHMgdGhlIG5vLXNub29wIGJsb2NraW5n IHNlY3VyaXR5IGZlYXR1cmUgVkZJTwo+IHR1cm5zIGl0IG9uIGFuZCBibG9ja3Mgbm8tc25vb3Ag dG8gYWxsIHBhZ2VzPyBPay4uCgpZZXAsIEknZCBmb3Jnb3R0ZW4gdGhpcyBudWFuY2UgdGhhdCB3 ZSBuZWVkIHRvIGVuYWJsZSBpdCB2aWEgdGhlCm1hcHBpbmcgZmxhZ3MuCgo+IEJ1dCBJIG11c3Qg YmUgbWlzc2luZyBzb21ldGhpbmcgYmlnIGJlY2F1c2UgKnNvbWV0aGluZyogaW4gdGhlIElPVkEK PiBtYXAgc2hvdWxkIHdvcmsgd2l0aCBuby1zbm9vcGFibGUgRE1BLCByaWdodD8gT3RoZXJ3aXNl IHdoYXQgaXMgdGhlCj4gcG9pbnQgb2YgZXhwb3NpbmcgdGhlIGludmFsaWRhdGUgaW5zdHJ1Y3Rp b24gdG8gdGhlIGd1ZXN0Pwo+IAo+IEkgd291bGQgdGhpbmsgdXNlcnNwYWNlIHNob3VsZCBiZSBy ZWxheWluZyB0aGUgRE1BX1BURV9TTlAgYml0IGZyb20KPiB0aGUgZ3Vlc3QncyBwYWdlIHRhYmxl cyB1cCB0byBoZXJlPz8KPiAKPiBUaGUgS1ZNIGhvb2t1cCBpcyBkcml2ZW4gYnkgSU9NTVVfQ0FD SEUgd2hpY2ggaXMgZHJpdmVuIGJ5Cj4gSU9NTVVfQ0FQX0NBQ0hFX0NPSEVSRU5DWS4gU28gd2Ug dHVybiBvbiB0aGUgc3BlY2lhbCBLVk0gc3VwcG9ydCBvbmx5Cj4gaWYgdGhlIElPTU1VIGNhbiBi bG9jayB0aGUgU05QIGJpdD8gQW5kIHRoZW4gd2UgbWFwIGFsbCB0aGUgcGFnZXMgdG8KPiBibG9j ayB0aGUgc25vb3AgYml0PyBIdWg/CgpSaWdodC4gIEkgZG9uJ3QgZm9sbG93IHdoZXJlIHlvdSdy ZSBqdW1waW5nIHRvIHJlbGF5aW5nIERNQV9QVEVfU05QCmZyb20gdGhlIGd1ZXN0IHBhZ2UgdGFi bGUuLi4gd2hhdCBwYWdlIHRhYmxlPyAgV2UgZG9uJ3QgbmVjZXNzYXJpbHkKaGF2ZSBhIHZJT01N VSB0byBleHBvc2Ugc3VjaCB0aGluZ3MsIEkgZG9uJ3QgdGhpbmsgaXQgZXZlbiBleGlzdGVkIHdo ZW4KdGhpcyB3ZSBhZGRlZC4gIEVzc2VudGlhbGx5IGlmIHdlIGNhbiBpZ25vcmUgbm8tc25vb3Ag YXQgdGhlIElPTU1VLAp0aGVuIEtWTSBkb2Vzbid0IG5lZWQgdG8gd29ycnkgYWJvdXQgZW11bGF0 aW5nIHdiaW52ZCBiZWNhdXNlIG9mIGFuCmFzc2lnbmVkIGRldmljZSwgd2hldGhlciB0aGF0IGRl dmljZSB1c2VzIGl0IG9yIG5vdC4gIFdpbi13aW4uCgo+IFlvdXIgZXhwbGFuYXRpb24gbWFrZXMg cGVyZmVjdCBzZW5zZTogQmxvY2sgZ3Vlc3RzIGZyb20gdXNpbmcgdGhlCj4gZGFuZ2Vyb3VzIGNh Y2hlIGludmFsaWRhdGUgaW5zdHJ1Y3Rpb24gdW5sZXNzIGEgZGV2aWNlIHRoYXQgdXNlcwo+IG5v LXNub29wIGlzIHBsdWdnZWQgaW4uIEJsb2NrIGRldmljZXMgZnJvbSB1c2luZyBuby1zbm9vcCBi ZWNhdXNlCj4gc29tZXRoaW5nIGFib3V0IGl0IGlzIGluc2VjdXJlLiBPay4KCk5vLXNub29wIGl0 c2VsZiBpcyBub3QgaW5zZWN1cmUsIGJ1dCB0byBzdXBwb3J0IG5vLXNub29wIGluIGEgVk0gS1ZN CmNhbid0IGlnbm9yZSB3YmludmQsIHdoaWNoIGhhcyBvdmVyaGVhZCBhbmQgYWJ1c2UgaW1wbGlj YXRpb25zLgoKPiBCdXQgdGhlIGNvbmRpdGlvbnMgSSdtIGxvb2tpbmcgZm9yICJkZXZpY2UgdGhh dCB1c2VzIG5vLXNub29wIiBpczoKPiAgLSBUaGUgZGV2aWNlIHdpbGwgaXNzdWUgbm8tc25vb3Ag VExQcyBhdCBhbGwKCldlIGNhbid0IHJlYWxseSBrbm93IHRoaXMgZ2VuZXJpY2FsbHkuICBXZSBj YW4gdHJ5IHRvIHNldCB0aGUgZW5hYmxlCmJpdCB0byBzZWUgaWYgdGhlIGRldmljZSBpcyBjYXBh YmxlIG9mIG5vLXNub29wLCBidXQgdGhhdCBkb2Vzbid0IG1lYW4KaXQgd2lsbCB1c2Ugbm8tc25v b3AuCgo+ICAtIFRoZSBJT01NVSB3aWxsIGxldCBuby1zbm9vcCB0aHJvdWdoCj4gIC0gVGhlIHBs YXRmb3JtIHdpbGwgaG9ub3Igbm8tc25vb3AKPiAKPiBPbmx5IGlmIGFsbCB0aHJlZSBhcmUgbWV0 IHdlIHNob3VsZCBhbGxvdyB0aGUgZGFuZ2Vyb3VzIGluc3RydWN0aW9uIGluCj4gS1ZNLCByaWdo dD8KCldlIHRlc3QgYXQgdGhlIElPTU1VIGFuZCBhc3N1bWUgdGhhdCB0aGUgSU9NTVUga25vd2xl ZGdlIGVuY29tcGFzc2VzCndoZXRoZXIgdGhlIHBsYXRmb3JtIGhvbm9ycyBuby1zbm9vcCAobm90 ZSBmb3IgZXhhbXBsZSBob3cgYW1kIGFuZCBhcm0KcmVwb3J0IHRydWUgZm9yIElPTU1VX0NBUF9D QUNIRV9DT0hFUkVOQ1kgYnV0IHNlZW0gdG8gaWdub3JlIHRoZQpJT01NVV9DQUNIRSBmbGFnKS4g IFdlIGNvdWxkIHByb2JhYmx5IHVzZSBhbiBpb21tdV9ncm91cF9mb3JfZWFjaF9kZXYKdG8gdGVz dCBpZiBhbnkgZGV2aWNlcyB3aXRoaW4gdGhlIGdyb3VwIGFyZSBjYXBhYmxlIG9mIG5vLXNub29w IGlmIHRoZQpJT01NVSBjYW4ndCBwcm90ZWN0IHVzLCBidXQgYXQgdGhlIHRpbWUgaXQgZGlkbid0 IHNlZW0gd29ydGh3aGlsZS4gIEknbQpzdGlsbCBub3Qgc3VyZSBpZiBpdCBpcy4KIAo+IFdoaWNo IGJyaW5ncyBtZSBiYWNrIHRvIG15IG9yaWdpbmFsIHBvaW50IC0gdGhpcyBpcyBhdCBsZWFzdCBw YXJ0aWFsbHkKPiBhIGRldmljZSBzcGVjaWZpYyBiZWhhdmlvci4gSXQgZGVwZW5kcyBvbiB0aGUg Y29udGVudCBvZiB0aGUgSU9NTVUKPiBwYWdlIHRhYmxlLCBpdCBkZXBlbmRzIGlmIHRoZSBkZXZp Y2UgZXZlbiBzdXBwb3J0cyBuby1zbm9vcCBhdCBhbGwuCj4gCj4gTXkgZ3Vlc3MgaXMgdGhpcyB3 b3JrcyBjb3JyZWN0bHkgZm9yIHRoZSBtZGV2IEludGVsIGt2bWd0IHdoaWNoCj4gcHJvYmFibHkg c29tZWhvdyBhbGxvd3Mgbm8tc25vb3AgRE1BIHRocm91Z2h0IHRoZSBtZGV2IFNXIGlvbW11Cj4g bWFwcGluZ3MuIChhc3N1bWluZyBJIGRpZG4ndCBtaXNzIGEgdHJpY2t5IGlvbW11X21hcCB3aXRo b3V0Cj4gSU9NTVVfQ0FDSGUgc2V0IGluIHRoZSB0eXBlMSBjb2RlPykKClRoaXMgc3VwcG9ydCBl eGlzdGVkIGJlZm9yZSBtZGV2LCBJSVJDIHdlIG5lZWRlZCBpdCBmb3IgZGlyZWN0CmFzc2lnbm1l bnQgb2YgTlZJRElBIEdQVXMuCiAKPiBCdXQgd2h5IGlzIHZmaW8tcGNpIHVzaW5nIGl0PyBIbW0/ CgpVc2UgdGhlIElPTU1VIHRvIHJlZHVjZSBoeXBlcnZpc29yIG92ZXJoZWFkLCBsZXQgdGhlIGh5 cGVydmlzb3IgbGVhcm4KYWJvdXQgaXQsIGlnbm9yZSB0aGUgc3VidGxldGllcyBvZiB3aGV0aGVy IHRoZSBkZXZpY2UgYWN0dWFsbHkgdXNlcwpuby1zbm9vcCBhcyBpbXByZWNpc2UgYW5kIHBvb3Ig Uk9JIGdpdmVuIHRoZSBhcHBhcmVudCBkaXJlY3Rpb24gb2YKaGFyZHdhcmUuCgrCr1xfKOODhClf L8KvLApBbGV4CgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f Xwppb21tdSBtYWlsaW5nIGxpc3QKaW9tbXVAbGlzdHMubGludXgtZm91bmRhdGlvbi5vcmcKaHR0 cHM6Ly9saXN0cy5saW51eGZvdW5kYXRpb24ub3JnL21haWxtYW4vbGlzdGluZm8vaW9tbXU=