From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.3 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,MSGID_FROM_MTA_HEADER,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66339C433B4 for ; Thu, 6 May 2021 16:10:28 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 09053610A1 for ; Thu, 6 May 2021 16:10:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 09053610A1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 6CC716E1EE; Thu, 6 May 2021 16:10:24 +0000 (UTC) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2089.outbound.protection.outlook.com [40.107.223.89]) by gabe.freedesktop.org (Postfix) with ESMTPS id 338266E1EE; Thu, 6 May 2021 16:10:23 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cXu4FRnSpg+isk31hKiyFiK1mawXCDnzAtuyI+GdD7EEPPxRZ+CMwMyvkcU4WvhW6K3avk4k1poCJOSgxAXTye+JGiMMSa4KjyHvofbI4Y7IfMOJ63W/jwHChPEYcJ6K9TdSUGY3uE5+B1dI0rtUqB9K+9mQFqhqc7WJmoIy027nBDIQoPx5S+hOKxyt9RZupO1X7JOL5Yeo6wasfmySjEt8x4chpaqJpLY/UV9IjVApvcloOOZOAWkQ72jvOXhHBnAq+UvBnofqBVVUPby5ZfGcW7Ray1Dqvk5LXT1Ef3UAMfucTv4OT3KOhiH7AFnRuDfY46/jziXuEE8Qpy3k1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=D/UwVeZpRepIO2X98iq0Qp4lRuvUKtzaPoy1V9Jpe7o=; b=Qyt1a7FKTjDpIOYjBrFTArTKdOuKlruftOBtox4c8IsrpAQrgmKBSTEQXAyD87En7V3ObYm785Ef56ckDIxw8BeXMgx2Nsb30TmNSLEsxdvRGzIrM9CYjzipasLIObWtUksZBAKtKjPMWBDqPeWpgrWAr9O3qqafwqJfYDIst8R5PLZE+QUmFQfvcFnvHPLMYGINaoHy1UgDREY/zpNUGU9PfAKVTMKv+KjWYiL2FrtHXON76Cfqob5ZFIuclArjZTreT9sUjE20jRGui4HBCpCDA+juqHwwz8fz78u58836aL7CcYxQxP7q44ZcgNIUDlxjjEtc/qHk2oRrhDBLhw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=D/UwVeZpRepIO2X98iq0Qp4lRuvUKtzaPoy1V9Jpe7o=; b=S9qqIZPiyQsXJS7Hdb68F3J42HKv6q9mwyxSRaUXAdX1uPjGJsTc4d88Z2Dl/kZICFWRA6j+nfDxUEJDde/t4BqSLwr/PYeQ22h5Ug0tHV3h1qmk3LrW7ODBicMfOo1K0bfCxbpz4Zy3jupoR6DZ4SphrpmNl60L80oa7P8RNcs= Authentication-Results: lisas.de; dkim=none (message not signed) header.d=none;lisas.de; dmarc=none action=none header.from=amd.com; Received: from BN9PR12MB5129.namprd12.prod.outlook.com (2603:10b6:408:136::12) by BN9PR12MB5292.namprd12.prod.outlook.com (2603:10b6:408:105::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4108.24; Thu, 6 May 2021 16:10:18 +0000 Received: from BN9PR12MB5129.namprd12.prod.outlook.com ([fe80::3c78:e58b:fba7:b8dd]) by BN9PR12MB5129.namprd12.prod.outlook.com ([fe80::3c78:e58b:fba7:b8dd%6]) with mapi id 15.20.4087.039; Thu, 6 May 2021 16:10:18 +0000 Subject: Re: [RFC] CRIU support for ROCm To: Daniel Vetter , "airlied@gmail.com" , Jason Ekstrand References: <9245171d-ecc9-1bdf-3ecd-cf776dc17855@amd.com> From: Felix Kuehling Message-ID: <232d301f-cf57-85b1-6bbf-11bddc634189@amd.com> Date: Thu, 6 May 2021 12:10:15 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.8.1 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Originating-IP: [142.116.138.207] X-ClientProxiedBy: YT2PR01CA0017.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:38::22) To BN9PR12MB5129.namprd12.prod.outlook.com (2603:10b6:408:136::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [192.168.2.100] (142.116.138.207) by YT2PR01CA0017.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:38::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4087.37 via Frontend Transport; Thu, 6 May 2021 16:10:17 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3188938b-0f25-454b-4771-08d910a9710f X-MS-TrafficTypeDiagnostic: BN9PR12MB5292: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5797; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Uy9zc59f3YW0IvyzYfBd07zUcw0LZWY7DcvdYyXPbbNVpRuNjYm694mg/i2hWXZ+xngfpxRvEc8KmNH9tZb/aoI6BD4bTa2U2omeayVs83IBDERq2a5d+/379MTJMw9ykEcuZRO/8dbgh4xDBYeJDYCHL+bfzk3CDx4BuriPjtN+dL6/UfdP1kYuiaaqzVPz1miahb6DDq1/aySXmSyoLHJMbLKAmM3ZzYepVPVzuluUT6zMa6EfxWKg+GxDt2I0TQL1rJfPM4himW+mtHMZkNiGKcbmRGf+EyCbQfM6vdxhXkvbT0BnY+e1dp07OVRxo+9T8TUxJqBvKZd0W+sCBCsWXJ3mmFhEb4Dz1glUWsNLc3zIycMJKi3m4LaJhZi495kZ3hNGiHptUL8G27Saa/NFGBq/OBrVFNe4oeJJojM/pe5XqqZPvhDcaJYsxYMGh2kEf4GwoxZhuCkCT/ZDhdtCwy/QT2BBzMWvlBW77oJJONOifp1gYYVv3rz5igfHGpFBp/0mYnQe2B4HoxDdeu/R49VQghG/mXa0cfzvw0+zbNq9liboxXSrA5hdL9tYxW1MgSJT07TvFNKQ2zrk56tdovSqoPMdUQKYg66ODn1XHrANmJUlVakysQZdqzZoIBW3Nzb8AXU2ttaNI0QR+mMpcOc/OWLc/JXeninsBHyrvnog78mntAMgVy5GUSGdejQcxTL6SIolr/woFFFY9psZ/6cFus0LYBsIGd33PRGfkxKQj6PC7FatlDfzBK1aVzVept642Nx1MkB+IR+QSw== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN9PR12MB5129.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(136003)(39860400002)(396003)(376002)(346002)(31696002)(110136005)(2906002)(30864003)(16576012)(54906003)(4326008)(44832011)(8936002)(966005)(6486002)(5660300002)(316002)(36756003)(31686004)(7416002)(86362001)(478600001)(956004)(83380400001)(2616005)(8676002)(16526019)(186003)(66476007)(66946007)(38100700002)(66556008)(26005)(45980500001)(43740500002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?utf-8?B?LzRmc0pjeWl4aml3NmNEODNNNmVCS0ZhRHR5VEM1K3FWYVJGcFhybXdQU25z?= =?utf-8?B?Wjg5M0ExMkl6M2VSM3l6LzJ4NzgrM2pJMit5d1czUzUwUkI1MXZLZlZDOEpW?= =?utf-8?B?cnk3WXZCbzh2L2h3OGlPem1BZHp0RVpxWHRYY1ZvdElBMnBjV1pNd2VXOWIy?= =?utf-8?B?amMrTjNCQVRFVUY0RVhBN2djZkxlbUJZb2RlcE5Gai9OUWV1QU5yOWJaZGtF?= =?utf-8?B?TmJwQllNQkxzR2p1cEZhN1lxVGlUZTlkTjFMWE9DTndxaGdLS20rZ3Y4aUV5?= =?utf-8?B?TW1uQmtadlF2a29UTFIveXFYcDNzZi95emxWN3pZSGJQODRvakhPcVJidXFu?= =?utf-8?B?THREZFFrSHk2MFpjc1E0TXVPQXNldWVJTG1iYWRESTl4aFk2ZzN3L1ZGcE9C?= =?utf-8?B?RHphMHpEblRVWGthajZYWXJ0cDhJOTJ5eE1lSVFDOGJYKzhrYVBBeHIzNkE0?= =?utf-8?B?c0Q3cGJBRkc3VW5tWVA2blJ1d1ZPZlR4b3hXRmlTN2MyeTBQRk9lTVRZN2RT?= =?utf-8?B?ZERnUVZWbVlWS1JkRzBqaVJnTE1PVGxkUzBiNGZyTDJLVWFoRkJqRkcyL3dR?= =?utf-8?B?Smkrc2xmeHBUODQ0R3JDaWJ4YjUzVDJlRXRQcEp2d0dISFZ3b2E4dTl5cG9q?= =?utf-8?B?eUlUMmxaNHdPckFQdCtCTndRTXVwSC9aWmg3Tzh0RVZLT3BsOGlJTmFlQkth?= =?utf-8?B?VGJ6K3RkUXpxMzBra0NoZXFsblIwZG0wNHpHNnZRbHRqbXordkxzVTZheUhl?= =?utf-8?B?THdBTEViNktHdTJmZWZCOXNaSWFtZDZ3UTduS3dKQnFVZ09Lb0w5UDVpVGYw?= =?utf-8?B?OFY5WmVaajdMSnVtTjZuSFpWR0lsNkZjQm1BUmNwYk1UMEx1b21kS1pDeHdV?= =?utf-8?B?UUVuUXlRTW1naFN1bkQzK3lyT0JDSFpPallxRlFXNlFlbXZXLy96dmtrbllm?= =?utf-8?B?TFEyalU2dklVWTFBTWJyQk82cHduVHU2UGN1NzJRQmpIa2NqTXpiaUhzTjFO?= =?utf-8?B?SU9GNDJIVnJxSmY1eU5lbFl1T0Q0UkhVZE9VQWVDcmZodnVuQ0Y5aWFOeWRX?= =?utf-8?B?NXppQnMyc2JHRE0xTmZOVjFFYUpySmZCQW55SjloZGJLVVlCTkJUb1VweWI5?= =?utf-8?B?Qzkwc2tQbnB6eFlGUFk2dERZZzVhMkVNMkl6MWJFeklqMk5SaHVuM2NjVGFO?= =?utf-8?B?c2pJd084L2VHVjZYVWgySHltaXZVVnI2NHArbnRyS2dUWE5YdzNHaTkzMmVr?= =?utf-8?B?WFVmcjFKUThhdjlPU0RvN1RGNmdacDdZMmZYekpMcmtPTG9OQmhJbXorZ1lu?= =?utf-8?B?c1p3RXBCS1A0SjA2b290UlZyUEthRkZHZE9Pa1RhdGlaRi9Oc1N0U2hMRHZY?= =?utf-8?B?cVFKWGZxdUxVdjl6Tll4eEpqMVVqeWVZaVUxc1BFVDBLNURHNlkvcXZVMFZT?= =?utf-8?B?OHI4M3N2Qk9VZVpQOGVOR3J2M1Z4Wk55Rlg5UlhQQnhpQXJBK25HZUYwNWhk?= =?utf-8?B?cm9QTzRaRlE4SVhmMkR5TS9YYUs2anVFcFlMdGo0bjJrWDd4TVVYa01RTjgv?= =?utf-8?B?L3V6M2wxV3B3Tm5PTmUwMUdFd3h2Mk9aeUxKWHR0RkhkOTVoS0dyS0UrekdF?= =?utf-8?B?bkZOWTd2dWg0YWNGMFd2aGdiM0IwLzFYa3hNQXBqbS9qTCt4aDBBcHBzMlhP?= =?utf-8?B?Z2s4SVZoaXIxLzd2N0E5azhlNG5ySU40anU4ajF5amU1UnI0dHRuT3U3VlZk?= =?utf-8?B?VGl2aW84cjJrVmNxbUVOTWlrL0EvMTBtZWlHT3FybGhrSXBWN0M1N1RXWDc2?= =?utf-8?B?SmV2RDIwdEg3aEljN0thUT09?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3188938b-0f25-454b-4771-08d910a9710f X-MS-Exchange-CrossTenant-AuthSource: BN9PR12MB5129.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 May 2021 16:10:18.4459 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7sy2ob3GsQ1KPOVTZJ6MubdJYcY4DVdxJYQEzWjd2HGZXaOgOf1tlAuRwguPgWhSBibuL8/4IFg3DQp6snO0mA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR12MB5292 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alexander Mihalicyn , Pavel Emelyanov , "Bhardwaj, Rajneesh" , DRI Development , Pavel Tikhomirov , criu@openvz.org, "Yat Sin, David" , amd-gfx list , Adrian Reber Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Am 2021-05-04 um 9:00 a.m. schrieb Daniel Vetter: > On Fri, Apr 30, 2021 at 09:57:45PM -0400, Felix Kuehling wrote: >> We have been working on a prototype supporting CRIU (Checkpoint/Restore >> In Userspace) for accelerated compute applications running on AMD GPUs >> using ROCm (Radeon Open Compute Platform). We're happy to finally share >> this work publicly to solicit feedback and advice. The end-goal is to >> get this work included upstream in Linux and CRIU. A short whitepaper >> describing our design and intention can be found on Github: >> https://github.com/RadeonOpenCompute/criu/tree/criu-dev/test/others/ext-kfd/README.md >> >> We have RFC patch series for the kernel (based on Alex Deucher's >> amd-staging-drm-next branch) and for CRIU including a new plugin and a >> few core CRIU changes. I will send those to the respective mailing lists >> separately in a minute. They can also be found on Github. >> >> CRIU+plugin: https://github.com/RadeonOpenCompute/criu/commits/criu-dev >> Kernel (KFD): >> https://github.com/RadeonOpenCompute/ROCK-Kernel-Driver/commits/fxkamd/criu-wip >> >> At this point this is very much a work in progress and not ready for >> upstream inclusion. There are still several missing features, known >> issues, and open questions that we would like to start addressing with >> your feedback. > Since the thread is a bit split I'm dumping the big thoughts here on this > RFC. > > We've discussed this in the past, but I'm once more (insert meme here) > asking whether continuing to walk down the amdgpu vs amdkfd split is > really the right choice. It starts to feel a bit much like sunk cost > fallacy ... Hi Daniel, Thanks for the feedback. I have some comments to your specific points below. This is my own opinion at this point and may not reflect AMDs position. I'm starting some internal discussions about unifying the KFD and graphics APIs in the long run. But IMO this is going to take years and won't be supported on our current compute GPUs, including Aldebaran which isn't even released yet. > > - From the big thread we're having right now on dri-devel it's clear that > 3d will also move towards more and more a userspace submit model. I'll need to start following dri-devel more closely and take a more active role in those discussions. If there is an opportunity for a unified memory management and command submission model for graphics and compute on future hardware, I want to be sure that our compute requirements are understood early on. > But > due to backwards compat issues it will be a mixed model, and in some > cases we need to pick at runtime which model we're picking. A hard split > between the amdgpu and the amdkfd world gets in the way here. Backwards compatibility will force us to maintain KFD at least for GFXv9 and older AMD GPUs. As I understand it, the new user mode command submission model will only be viable on GFXv10 or even newer GPUs. GFXv9 is our architecture for current compute GPUs (including Aldebaran which is still in development), so this is going to be important for us for years to come. > > - There's use-cases for doing compute in vulkan (that was a discussion > from Feb that I kicked again in private, since I think still > unresolved). So you need a vulkan stack that runs on both amdgpu and > amdvlk. By amdvlk, do you mean AMD's "pro" driver that supports our non-RADV Vulcan driver? > > - Maybe not yet on amd's radar, but there's a lot of cloud computing. And > maybe they also want CRIU for migrating their containers around. So that > means CRIU for amdgpu too, not just amdkf. Our strategy for compute support (OpenCL, HIP, OpenMP and mayby future C++ standards) is to use ROCm on all current and future GPUs starting with GFXv9. As long as ROCm is based on KFD, that means our compute stack runs on KFD. We don't see KFD as something mutually exclusive with graphics. They are different ioctl APIs, but they're sharing the hardware and memory and are meant to work together. We are already planning to add CRIU support to our render node API because video acceleration on our compute stack still relies on Mesa and the render node CS API. I admit, that we are not currently considering Vulcan as part of our compute stack. I don't think it makes sense to build Vulcan on top of ROCm because we have no plans to add graphics support to ROCm. What commonality there is between Vulcan and other compute APIs will need to be addressed at a lower level. If graphics is moving to user mode command submission and a shared virtual memory model, I see an opportunity to move ROCm to a common ioctl API on those future hardware generations. > > - What's much worse, and I don't think anyone in amd has realized this yet > (at least not in a public thread I've seen). In vulkan you need to be > able to switch from compute mode to dma-fence mode after > pipelines/devices have been created already. This is because winsys are > only initialized in a second step, until that's done you have to > pessimistically assume that the user does pure compute. What's worse for > buffer sharing you don't even have a clear signal on this stuff. So > either > > - you figure out how to migrate all the buffers and state from amdkfd to > amdgpu at runtime, and duplicate all the features. Which is rather > pointless. > > - or you duplicate all the compute features to amdgpu so that vk can use > them, and still reasonably easy migrate to winsys/dma-fence mode, > which makes amdkfd rather redundant. The basic "compute features" that ROCm depends on are a shared-virtual-memory model and user mode command submission. If those become part of the graphics ioctl API, I see no problem with implementing proper Vulcan compute on top of it. For more consistent support for tools such as debugger, profilers, CRIU, etc. it would make sense to have a common ioctl API for ROCm and Vulcan, which would effectively deprecate the KFD API. And I'm definitely open to working on that for future HW generations that can support it. We'll probably have to pick a target GPU where we make such a switch, and use the time until then to get things ready, prototyping in current GFXv10 or newer GPUs. This feels to me a bit like the transition from radeon to amdgpu, where GFXv7 was supported by both for some time.   > > I've discussed this problem extensively with Jason Ekstrand, and it's > really nasty. > > So purely from a technical pov, only looking at the AMD perspective here, > this doesn't make much sense to me. The only reason to keep doubling down > on amdkfd I'm seeing is that you've built your compute rocm stack on top > of it, and because of that the only option is to keep doing that. Which > stops making sense eventually, and we're getting to that point for sure. As long as our compute GPUs are based on the GFXv9 architecture, I think we don't really have an alternative than to double down on KFD. We can't just stop work on KFD and tell our customers that they'll not get any new features for the next 2 or 3 years that it takes to build a common upstream memory management and scheduling API and the HW that can support it. > > The other side is a bit the upstream side, but that's a lot smaller: > > - vulkan compute is one of the more reasonable ways to get cross vendor > compute ecosystem off the ground. At least from what I know from > background chatter, which you guys probably haven't all heard. amdkfd > being the single very odd driver here requiring entirely different uapi > for compute mode is not going to be great. Like I said, KFD is not exclusive with anything else. Vulcan may be a good option for desktops. For HPC uses or general portable non-graphics compute code, I think something like OpenMP or future C++ standards would be a better way forward. There is no technical reason why both cannot coexist in the same application. > > - CRIU will need new access rights handling (for the save/restore/resume > stuff you're adding). Generally we standardize access rights checks > across drivers, and leave everything else to render drivers (command > submission, memory management, ...). By adding CRIU support to amdkfd > we pretty much guarantee that we wont be able to standardize CRIU access > rights across drivers. Which just plains sucks from an > upstream/cross-vendor ecosystem pov. By access rights, do you mean requiring root for some ioctls, or being ptrace-attached for others? These are driven by how CRIU works and interacts with its target processes. I think they will apply equally to any driver implementing CRIU support. I don't see how graphics drivers specifically drive standardization of CRIU access rights. > > And yes we'd still need a per-driver criu plugin in userspace, but the > same is true for amdvlk/radv/anv/ and all the other drivers we have: > Driver is different, access right management is still the same. > > And secondly, just because nvidia refuses to collaborate in any > standards around gpu compute doesn't mean that's a good reason for us to > do the same in upstream. > > Thirdly, it sounds like this is the first device-driver CRIU support, so I > think we need a solid agreement/standard here to set as an example for > everyone else. There's all the AI accel chips and fpga-for-compute stuff > that I expect will eventually also gain CRIU support. Sure. My experience with CRIU so far is, that CRIU requires root (even without our plugin). Regardless of that, CRIU attaches to the target process with ptrace. The kernel's security policy for allowing ptrace access is influenced by many factors. We chose not to duplicate this in the driver. Therefore we check whether the caller of our CRIU dump ioctl is ptrace attached. If it is, it means it has passed all the kernel's security checks. We deliberately do not want to implement our own security policy in the driver. There would be ways to remove the root requirement from our CRIU restore-ioctl, if that becomes a requirement for CRIU. It would mean some sanity checking of the HW state that's getting restored. > > So I'm asking once more, is this _really_ the right path forward? Both for > amd (at least long term), but also somewhat for upstream. I think long-term it makes sense to plan for a unified API. For current hardware I don't see a way to get there for multiple reasons: 1. Maintaining backwards compatibility with existing user mode 2. Lacking HW capabilities for graphics user mode command submission and HW scheduling Therefore KFD will be important for AMD's compute GPU strategy and our customers for years to come. Not being able to continue developing it upstream would be a major impediment for AMD. Again, this is my personal opinion. Regards,   Felix > > Cheers, Daniel > > >> What's working and tested at this point: >> >> * Checkpoint and restore accelerated machine learning apps: PyTorch >> running Bert on systems with 1 or 2 GPUs (MI50 or MI100), 100% >> unmodified user mode stack >> * Checkpoint on one system, restore on a different system >> * Checkpoint on one GPU, restore on a different GPU >> >> Major Known issues: >> >> * The KFD ioctl API is not final: Needs a complete redesign to allow >> future extension without breaking the ABI >> * Very slow: Need to implement DMA to dump VRAM contents >> >> Missing or incomplete features: >> >> * Support for the new KFD SVM API >> * Check device topology during restore >> * Checkpoint and restore multiple processes >> * Support for applications using Mesa for video decode/encode >> * Testing with more different GPUs and workloads >> >> Big Open questions: >> >> * What's the preferred way to publish our CRIU plugin? In-tree or >> out-of-tree? >> * What's the preferred way to distribute our CRIU plugin? Source? >> Binary .so? Whole CRIU? Just in-box support? >> * If our plugin can be upstreamed in the CRIU tree, what would be the >> right directory? >> >> Best regards, >> Felix >> >> >> _______________________________________________ >> dri-devel mailing list >> dri-devel@lists.freedesktop.org >> https://lists.freedesktop.org/mailman/listinfo/dri-devel > -- > Daniel Vetter > Software Engineer, Intel Corporation > http://blog.ffwll.ch/ From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.1 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,MSGID_FROM_MTA_HEADER,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BADCDC433ED for ; Thu, 6 May 2021 16:10:25 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2787F610A1 for ; Thu, 6 May 2021 16:10:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2787F610A1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=amd-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 9D5DE6E5A0; Thu, 6 May 2021 16:10:24 +0000 (UTC) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2089.outbound.protection.outlook.com [40.107.223.89]) by gabe.freedesktop.org (Postfix) with ESMTPS id 338266E1EE; Thu, 6 May 2021 16:10:23 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cXu4FRnSpg+isk31hKiyFiK1mawXCDnzAtuyI+GdD7EEPPxRZ+CMwMyvkcU4WvhW6K3avk4k1poCJOSgxAXTye+JGiMMSa4KjyHvofbI4Y7IfMOJ63W/jwHChPEYcJ6K9TdSUGY3uE5+B1dI0rtUqB9K+9mQFqhqc7WJmoIy027nBDIQoPx5S+hOKxyt9RZupO1X7JOL5Yeo6wasfmySjEt8x4chpaqJpLY/UV9IjVApvcloOOZOAWkQ72jvOXhHBnAq+UvBnofqBVVUPby5ZfGcW7Ray1Dqvk5LXT1Ef3UAMfucTv4OT3KOhiH7AFnRuDfY46/jziXuEE8Qpy3k1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=D/UwVeZpRepIO2X98iq0Qp4lRuvUKtzaPoy1V9Jpe7o=; b=Qyt1a7FKTjDpIOYjBrFTArTKdOuKlruftOBtox4c8IsrpAQrgmKBSTEQXAyD87En7V3ObYm785Ef56ckDIxw8BeXMgx2Nsb30TmNSLEsxdvRGzIrM9CYjzipasLIObWtUksZBAKtKjPMWBDqPeWpgrWAr9O3qqafwqJfYDIst8R5PLZE+QUmFQfvcFnvHPLMYGINaoHy1UgDREY/zpNUGU9PfAKVTMKv+KjWYiL2FrtHXON76Cfqob5ZFIuclArjZTreT9sUjE20jRGui4HBCpCDA+juqHwwz8fz78u58836aL7CcYxQxP7q44ZcgNIUDlxjjEtc/qHk2oRrhDBLhw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=D/UwVeZpRepIO2X98iq0Qp4lRuvUKtzaPoy1V9Jpe7o=; b=S9qqIZPiyQsXJS7Hdb68F3J42HKv6q9mwyxSRaUXAdX1uPjGJsTc4d88Z2Dl/kZICFWRA6j+nfDxUEJDde/t4BqSLwr/PYeQ22h5Ug0tHV3h1qmk3LrW7ODBicMfOo1K0bfCxbpz4Zy3jupoR6DZ4SphrpmNl60L80oa7P8RNcs= Authentication-Results: lisas.de; dkim=none (message not signed) header.d=none;lisas.de; dmarc=none action=none header.from=amd.com; Received: from BN9PR12MB5129.namprd12.prod.outlook.com (2603:10b6:408:136::12) by BN9PR12MB5292.namprd12.prod.outlook.com (2603:10b6:408:105::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4108.24; Thu, 6 May 2021 16:10:18 +0000 Received: from BN9PR12MB5129.namprd12.prod.outlook.com ([fe80::3c78:e58b:fba7:b8dd]) by BN9PR12MB5129.namprd12.prod.outlook.com ([fe80::3c78:e58b:fba7:b8dd%6]) with mapi id 15.20.4087.039; Thu, 6 May 2021 16:10:18 +0000 Subject: Re: [RFC] CRIU support for ROCm To: Daniel Vetter , "airlied@gmail.com" , Jason Ekstrand References: <9245171d-ecc9-1bdf-3ecd-cf776dc17855@amd.com> From: Felix Kuehling Message-ID: <232d301f-cf57-85b1-6bbf-11bddc634189@amd.com> Date: Thu, 6 May 2021 12:10:15 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.8.1 In-Reply-To: Content-Language: en-US X-Originating-IP: [142.116.138.207] X-ClientProxiedBy: YT2PR01CA0017.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:38::22) To BN9PR12MB5129.namprd12.prod.outlook.com (2603:10b6:408:136::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [192.168.2.100] (142.116.138.207) by YT2PR01CA0017.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:38::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4087.37 via Frontend Transport; Thu, 6 May 2021 16:10:17 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3188938b-0f25-454b-4771-08d910a9710f X-MS-TrafficTypeDiagnostic: BN9PR12MB5292: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5797; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Uy9zc59f3YW0IvyzYfBd07zUcw0LZWY7DcvdYyXPbbNVpRuNjYm694mg/i2hWXZ+xngfpxRvEc8KmNH9tZb/aoI6BD4bTa2U2omeayVs83IBDERq2a5d+/379MTJMw9ykEcuZRO/8dbgh4xDBYeJDYCHL+bfzk3CDx4BuriPjtN+dL6/UfdP1kYuiaaqzVPz1miahb6DDq1/aySXmSyoLHJMbLKAmM3ZzYepVPVzuluUT6zMa6EfxWKg+GxDt2I0TQL1rJfPM4himW+mtHMZkNiGKcbmRGf+EyCbQfM6vdxhXkvbT0BnY+e1dp07OVRxo+9T8TUxJqBvKZd0W+sCBCsWXJ3mmFhEb4Dz1glUWsNLc3zIycMJKi3m4LaJhZi495kZ3hNGiHptUL8G27Saa/NFGBq/OBrVFNe4oeJJojM/pe5XqqZPvhDcaJYsxYMGh2kEf4GwoxZhuCkCT/ZDhdtCwy/QT2BBzMWvlBW77oJJONOifp1gYYVv3rz5igfHGpFBp/0mYnQe2B4HoxDdeu/R49VQghG/mXa0cfzvw0+zbNq9liboxXSrA5hdL9tYxW1MgSJT07TvFNKQ2zrk56tdovSqoPMdUQKYg66ODn1XHrANmJUlVakysQZdqzZoIBW3Nzb8AXU2ttaNI0QR+mMpcOc/OWLc/JXeninsBHyrvnog78mntAMgVy5GUSGdejQcxTL6SIolr/woFFFY9psZ/6cFus0LYBsIGd33PRGfkxKQj6PC7FatlDfzBK1aVzVept642Nx1MkB+IR+QSw== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN9PR12MB5129.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(136003)(39860400002)(396003)(376002)(346002)(31696002)(110136005)(2906002)(30864003)(16576012)(54906003)(4326008)(44832011)(8936002)(966005)(6486002)(5660300002)(316002)(36756003)(31686004)(7416002)(86362001)(478600001)(956004)(83380400001)(2616005)(8676002)(16526019)(186003)(66476007)(66946007)(38100700002)(66556008)(26005)(45980500001)(43740500002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?utf-8?B?LzRmc0pjeWl4aml3NmNEODNNNmVCS0ZhRHR5VEM1K3FWYVJGcFhybXdQU25z?= =?utf-8?B?Wjg5M0ExMkl6M2VSM3l6LzJ4NzgrM2pJMit5d1czUzUwUkI1MXZLZlZDOEpW?= =?utf-8?B?cnk3WXZCbzh2L2h3OGlPem1BZHp0RVpxWHRYY1ZvdElBMnBjV1pNd2VXOWIy?= =?utf-8?B?amMrTjNCQVRFVUY0RVhBN2djZkxlbUJZb2RlcE5Gai9OUWV1QU5yOWJaZGtF?= =?utf-8?B?TmJwQllNQkxzR2p1cEZhN1lxVGlUZTlkTjFMWE9DTndxaGdLS20rZ3Y4aUV5?= =?utf-8?B?TW1uQmtadlF2a29UTFIveXFYcDNzZi95emxWN3pZSGJQODRvakhPcVJidXFu?= =?utf-8?B?THREZFFrSHk2MFpjc1E0TXVPQXNldWVJTG1iYWRESTl4aFk2ZzN3L1ZGcE9C?= =?utf-8?B?RHphMHpEblRVWGthajZYWXJ0cDhJOTJ5eE1lSVFDOGJYKzhrYVBBeHIzNkE0?= =?utf-8?B?c0Q3cGJBRkc3VW5tWVA2blJ1d1ZPZlR4b3hXRmlTN2MyeTBQRk9lTVRZN2RT?= =?utf-8?B?ZERnUVZWbVlWS1JkRzBqaVJnTE1PVGxkUzBiNGZyTDJLVWFoRkJqRkcyL3dR?= =?utf-8?B?Smkrc2xmeHBUODQ0R3JDaWJ4YjUzVDJlRXRQcEp2d0dISFZ3b2E4dTl5cG9q?= =?utf-8?B?eUlUMmxaNHdPckFQdCtCTndRTXVwSC9aWmg3Tzh0RVZLT3BsOGlJTmFlQkth?= =?utf-8?B?VGJ6K3RkUXpxMzBra0NoZXFsblIwZG0wNHpHNnZRbHRqbXordkxzVTZheUhl?= =?utf-8?B?THdBTEViNktHdTJmZWZCOXNaSWFtZDZ3UTduS3dKQnFVZ09Lb0w5UDVpVGYw?= =?utf-8?B?OFY5WmVaajdMSnVtTjZuSFpWR0lsNkZjQm1BUmNwYk1UMEx1b21kS1pDeHdV?= =?utf-8?B?UUVuUXlRTW1naFN1bkQzK3lyT0JDSFpPallxRlFXNlFlbXZXLy96dmtrbllm?= =?utf-8?B?TFEyalU2dklVWTFBTWJyQk82cHduVHU2UGN1NzJRQmpIa2NqTXpiaUhzTjFO?= =?utf-8?B?SU9GNDJIVnJxSmY1eU5lbFl1T0Q0UkhVZE9VQWVDcmZodnVuQ0Y5aWFOeWRX?= =?utf-8?B?NXppQnMyc2JHRE0xTmZOVjFFYUpySmZCQW55SjloZGJLVVlCTkJUb1VweWI5?= =?utf-8?B?Qzkwc2tQbnB6eFlGUFk2dERZZzVhMkVNMkl6MWJFeklqMk5SaHVuM2NjVGFO?= =?utf-8?B?c2pJd084L2VHVjZYVWgySHltaXZVVnI2NHArbnRyS2dUWE5YdzNHaTkzMmVr?= =?utf-8?B?WFVmcjFKUThhdjlPU0RvN1RGNmdacDdZMmZYekpMcmtPTG9OQmhJbXorZ1lu?= =?utf-8?B?c1p3RXBCS1A0SjA2b290UlZyUEthRkZHZE9Pa1RhdGlaRi9Oc1N0U2hMRHZY?= =?utf-8?B?cVFKWGZxdUxVdjl6Tll4eEpqMVVqeWVZaVUxc1BFVDBLNURHNlkvcXZVMFZT?= =?utf-8?B?OHI4M3N2Qk9VZVpQOGVOR3J2M1Z4Wk55Rlg5UlhQQnhpQXJBK25HZUYwNWhk?= =?utf-8?B?cm9QTzRaRlE4SVhmMkR5TS9YYUs2anVFcFlMdGo0bjJrWDd4TVVYa01RTjgv?= =?utf-8?B?L3V6M2wxV3B3Tm5PTmUwMUdFd3h2Mk9aeUxKWHR0RkhkOTVoS0dyS0UrekdF?= =?utf-8?B?bkZOWTd2dWg0YWNGMFd2aGdiM0IwLzFYa3hNQXBqbS9qTCt4aDBBcHBzMlhP?= =?utf-8?B?Z2s4SVZoaXIxLzd2N0E5azhlNG5ySU40anU4ajF5amU1UnI0dHRuT3U3VlZk?= =?utf-8?B?VGl2aW84cjJrVmNxbUVOTWlrL0EvMTBtZWlHT3FybGhrSXBWN0M1N1RXWDc2?= =?utf-8?B?SmV2RDIwdEg3aEljN0thUT09?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3188938b-0f25-454b-4771-08d910a9710f X-MS-Exchange-CrossTenant-AuthSource: BN9PR12MB5129.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 May 2021 16:10:18.4459 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7sy2ob3GsQ1KPOVTZJ6MubdJYcY4DVdxJYQEzWjd2HGZXaOgOf1tlAuRwguPgWhSBibuL8/4IFg3DQp6snO0mA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR12MB5292 X-BeenThere: amd-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion list for AMD gfx List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alexander Mihalicyn , Pavel Emelyanov , "Bhardwaj, Rajneesh" , DRI Development , Pavel Tikhomirov , criu@openvz.org, "Yat Sin, David" , amd-gfx list , Adrian Reber Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: amd-gfx-bounces@lists.freedesktop.org Sender: "amd-gfx" QW0gMjAyMS0wNS0wNCB1bSA5OjAwIGEubS4gc2NocmllYiBEYW5pZWwgVmV0dGVyOgo+IE9uIEZy aSwgQXByIDMwLCAyMDIxIGF0IDA5OjU3OjQ1UE0gLTA0MDAsIEZlbGl4IEt1ZWhsaW5nIHdyb3Rl Ogo+PiBXZSBoYXZlIGJlZW4gd29ya2luZyBvbiBhIHByb3RvdHlwZSBzdXBwb3J0aW5nIENSSVUg KENoZWNrcG9pbnQvUmVzdG9yZQo+PiBJbiBVc2Vyc3BhY2UpIGZvciBhY2NlbGVyYXRlZCBjb21w dXRlIGFwcGxpY2F0aW9ucyBydW5uaW5nIG9uIEFNRCBHUFVzCj4+IHVzaW5nIFJPQ20gKFJhZGVv biBPcGVuIENvbXB1dGUgUGxhdGZvcm0pLiBXZSdyZSBoYXBweSB0byBmaW5hbGx5IHNoYXJlCj4+ IHRoaXMgd29yayBwdWJsaWNseSB0byBzb2xpY2l0IGZlZWRiYWNrIGFuZCBhZHZpY2UuIFRoZSBl bmQtZ29hbCBpcyB0bwo+PiBnZXQgdGhpcyB3b3JrIGluY2x1ZGVkIHVwc3RyZWFtIGluIExpbnV4 IGFuZCBDUklVLiBBIHNob3J0IHdoaXRlcGFwZXIKPj4gZGVzY3JpYmluZyBvdXIgZGVzaWduIGFu ZCBpbnRlbnRpb24gY2FuIGJlIGZvdW5kIG9uIEdpdGh1YjoKPj4gaHR0cHM6Ly9naXRodWIuY29t L1JhZGVvbk9wZW5Db21wdXRlL2NyaXUvdHJlZS9jcml1LWRldi90ZXN0L290aGVycy9leHQta2Zk L1JFQURNRS5tZAo+Pgo+PiBXZSBoYXZlIFJGQyBwYXRjaCBzZXJpZXMgZm9yIHRoZSBrZXJuZWwg KGJhc2VkIG9uIEFsZXggRGV1Y2hlcidzCj4+IGFtZC1zdGFnaW5nLWRybS1uZXh0IGJyYW5jaCkg YW5kIGZvciBDUklVIGluY2x1ZGluZyBhIG5ldyBwbHVnaW4gYW5kIGEKPj4gZmV3IGNvcmUgQ1JJ VSBjaGFuZ2VzLiBJIHdpbGwgc2VuZCB0aG9zZSB0byB0aGUgcmVzcGVjdGl2ZSBtYWlsaW5nIGxp c3RzCj4+IHNlcGFyYXRlbHkgaW4gYSBtaW51dGUuIFRoZXkgY2FuIGFsc28gYmUgZm91bmQgb24g R2l0aHViLgo+Pgo+PiAgICAgQ1JJVStwbHVnaW46IGh0dHBzOi8vZ2l0aHViLmNvbS9SYWRlb25P cGVuQ29tcHV0ZS9jcml1L2NvbW1pdHMvY3JpdS1kZXYKPj4gICAgIEtlcm5lbCAoS0ZEKToKPj4g ICAgIGh0dHBzOi8vZ2l0aHViLmNvbS9SYWRlb25PcGVuQ29tcHV0ZS9ST0NLLUtlcm5lbC1Ecml2 ZXIvY29tbWl0cy9meGthbWQvY3JpdS13aXAKPj4KPj4gQXQgdGhpcyBwb2ludCB0aGlzIGlzIHZl cnkgbXVjaCBhIHdvcmsgaW4gcHJvZ3Jlc3MgYW5kIG5vdCByZWFkeSBmb3IKPj4gdXBzdHJlYW0g aW5jbHVzaW9uLiBUaGVyZSBhcmUgc3RpbGwgc2V2ZXJhbCBtaXNzaW5nIGZlYXR1cmVzLCBrbm93 bgo+PiBpc3N1ZXMsIGFuZCBvcGVuIHF1ZXN0aW9ucyB0aGF0IHdlIHdvdWxkIGxpa2UgdG8gc3Rh cnQgYWRkcmVzc2luZyB3aXRoCj4+IHlvdXIgZmVlZGJhY2suCj4gU2luY2UgdGhlIHRocmVhZCBp cyBhIGJpdCBzcGxpdCBJJ20gZHVtcGluZyB0aGUgYmlnIHRob3VnaHRzIGhlcmUgb24gdGhpcwo+ IFJGQy4KPgo+IFdlJ3ZlIGRpc2N1c3NlZCB0aGlzIGluIHRoZSBwYXN0LCBidXQgSSdtIG9uY2Ug bW9yZSAoaW5zZXJ0IG1lbWUgaGVyZSkKPiBhc2tpbmcgd2hldGhlciBjb250aW51aW5nIHRvIHdh bGsgZG93biB0aGUgYW1kZ3B1IHZzIGFtZGtmZCBzcGxpdCBpcwo+IHJlYWxseSB0aGUgcmlnaHQg Y2hvaWNlLiBJdCBzdGFydHMgdG8gZmVlbCBhIGJpdCBtdWNoIGxpa2Ugc3VuayBjb3N0Cj4gZmFs bGFjeSAuLi4KCkhpIERhbmllbCwKClRoYW5rcyBmb3IgdGhlIGZlZWRiYWNrLiBJIGhhdmUgc29t ZSBjb21tZW50cyB0byB5b3VyIHNwZWNpZmljIHBvaW50cwpiZWxvdy4gVGhpcyBpcyBteSBvd24g b3BpbmlvbiBhdCB0aGlzIHBvaW50IGFuZCBtYXkgbm90IHJlZmxlY3QgQU1Ecwpwb3NpdGlvbi4g SSdtIHN0YXJ0aW5nIHNvbWUgaW50ZXJuYWwgZGlzY3Vzc2lvbnMgYWJvdXQgdW5pZnlpbmcgdGhl IEtGRAphbmQgZ3JhcGhpY3MgQVBJcyBpbiB0aGUgbG9uZyBydW4uIEJ1dCBJTU8gdGhpcyBpcyBn b2luZyB0byB0YWtlIHllYXJzCmFuZCB3b24ndCBiZSBzdXBwb3J0ZWQgb24gb3VyIGN1cnJlbnQg Y29tcHV0ZSBHUFVzLCBpbmNsdWRpbmcgQWxkZWJhcmFuCndoaWNoIGlzbid0IGV2ZW4gcmVsZWFz ZWQgeWV0LgoKCj4KPiAtIEZyb20gdGhlIGJpZyB0aHJlYWQgd2UncmUgaGF2aW5nIHJpZ2h0IG5v dyBvbiBkcmktZGV2ZWwgaXQncyBjbGVhciB0aGF0Cj4gICAzZCB3aWxsIGFsc28gbW92ZSB0b3dh cmRzIG1vcmUgYW5kIG1vcmUgYSB1c2Vyc3BhY2Ugc3VibWl0IG1vZGVsLgoKSSdsbCBuZWVkIHRv IHN0YXJ0IGZvbGxvd2luZyBkcmktZGV2ZWwgbW9yZSBjbG9zZWx5IGFuZCB0YWtlIGEgbW9yZQph Y3RpdmUgcm9sZSBpbiB0aG9zZSBkaXNjdXNzaW9ucy4gSWYgdGhlcmUgaXMgYW4gb3Bwb3J0dW5p dHkgZm9yIGEKdW5pZmllZCBtZW1vcnkgbWFuYWdlbWVudCBhbmQgY29tbWFuZCBzdWJtaXNzaW9u IG1vZGVsIGZvciBncmFwaGljcyBhbmQKY29tcHV0ZSBvbiBmdXR1cmUgaGFyZHdhcmUsIEkgd2Fu dCB0byBiZSBzdXJlIHRoYXQgb3VyIGNvbXB1dGUKcmVxdWlyZW1lbnRzIGFyZSB1bmRlcnN0b29k IGVhcmx5IG9uLgoKCj4gIEJ1dAo+ICAgZHVlIHRvIGJhY2t3YXJkcyBjb21wYXQgaXNzdWVzIGl0 IHdpbGwgYmUgYSBtaXhlZCBtb2RlbCwgYW5kIGluIHNvbWUKPiAgIGNhc2VzIHdlIG5lZWQgdG8g cGljayBhdCBydW50aW1lIHdoaWNoIG1vZGVsIHdlJ3JlIHBpY2tpbmcuIEEgaGFyZCBzcGxpdAo+ ICAgYmV0d2VlbiB0aGUgYW1kZ3B1IGFuZCB0aGUgYW1ka2ZkIHdvcmxkIGdldHMgaW4gdGhlIHdh eSBoZXJlLgoKQmFja3dhcmRzIGNvbXBhdGliaWxpdHkgd2lsbCBmb3JjZSB1cyB0byBtYWludGFp biBLRkQgYXQgbGVhc3QgZm9yIEdGWHY5CmFuZCBvbGRlciBBTUQgR1BVcy4gQXMgSSB1bmRlcnN0 YW5kIGl0LCB0aGUgbmV3IHVzZXIgbW9kZSBjb21tYW5kCnN1Ym1pc3Npb24gbW9kZWwgd2lsbCBv bmx5IGJlIHZpYWJsZSBvbiBHRlh2MTAgb3IgZXZlbiBuZXdlciBHUFVzLiBHRlh2OQppcyBvdXIg YXJjaGl0ZWN0dXJlIGZvciBjdXJyZW50IGNvbXB1dGUgR1BVcyAoaW5jbHVkaW5nIEFsZGViYXJh biB3aGljaAppcyBzdGlsbCBpbiBkZXZlbG9wbWVudCksIHNvIHRoaXMgaXMgZ29pbmcgdG8gYmUg aW1wb3J0YW50IGZvciB1cyBmb3IKeWVhcnMgdG8gY29tZS4KCgo+Cj4gLSBUaGVyZSdzIHVzZS1j YXNlcyBmb3IgZG9pbmcgY29tcHV0ZSBpbiB2dWxrYW4gKHRoYXQgd2FzIGEgZGlzY3Vzc2lvbgo+ ICAgZnJvbSBGZWIgdGhhdCBJIGtpY2tlZCBhZ2FpbiBpbiBwcml2YXRlLCBzaW5jZSBJIHRoaW5r IHN0aWxsCj4gICB1bnJlc29sdmVkKS4gU28geW91IG5lZWQgYSB2dWxrYW4gc3RhY2sgdGhhdCBy dW5zIG9uIGJvdGggYW1kZ3B1IGFuZAo+ICAgYW1kdmxrLgoKQnkgYW1kdmxrLCBkbyB5b3UgbWVh biBBTUQncyAicHJvIiBkcml2ZXIgdGhhdCBzdXBwb3J0cyBvdXIgbm9uLVJBRFYKVnVsY2FuIGRy aXZlcj8KCgo+Cj4gLSBNYXliZSBub3QgeWV0IG9uIGFtZCdzIHJhZGFyLCBidXQgdGhlcmUncyBh IGxvdCBvZiBjbG91ZCBjb21wdXRpbmcuIEFuZAo+ICAgbWF5YmUgdGhleSBhbHNvIHdhbnQgQ1JJ VSBmb3IgbWlncmF0aW5nIHRoZWlyIGNvbnRhaW5lcnMgYXJvdW5kLiBTbyB0aGF0Cj4gICBtZWFu cyBDUklVIGZvciBhbWRncHUgdG9vLCBub3QganVzdCBhbWRrZi4KCk91ciBzdHJhdGVneSBmb3Ig Y29tcHV0ZSBzdXBwb3J0IChPcGVuQ0wsIEhJUCwgT3Blbk1QIGFuZCBtYXlieSBmdXR1cmUKQysr IHN0YW5kYXJkcykgaXMgdG8gdXNlIFJPQ20gb24gYWxsIGN1cnJlbnQgYW5kIGZ1dHVyZSBHUFVz IHN0YXJ0aW5nCndpdGggR0ZYdjkuIEFzIGxvbmcgYXMgUk9DbSBpcyBiYXNlZCBvbiBLRkQsIHRo YXQgbWVhbnMgb3VyIGNvbXB1dGUKc3RhY2sgcnVucyBvbiBLRkQuIFdlIGRvbid0IHNlZSBLRkQg YXMgc29tZXRoaW5nIG11dHVhbGx5IGV4Y2x1c2l2ZSB3aXRoCmdyYXBoaWNzLiBUaGV5IGFyZSBk aWZmZXJlbnQgaW9jdGwgQVBJcywgYnV0IHRoZXkncmUgc2hhcmluZyB0aGUKaGFyZHdhcmUgYW5k IG1lbW9yeSBhbmQgYXJlIG1lYW50IHRvIHdvcmsgdG9nZXRoZXIuCgpXZSBhcmUgYWxyZWFkeSBw bGFubmluZyB0byBhZGQgQ1JJVSBzdXBwb3J0IHRvIG91ciByZW5kZXIgbm9kZSBBUEkKYmVjYXVz ZSB2aWRlbyBhY2NlbGVyYXRpb24gb24gb3VyIGNvbXB1dGUgc3RhY2sgc3RpbGwgcmVsaWVzIG9u IE1lc2EgYW5kCnRoZSByZW5kZXIgbm9kZSBDUyBBUEkuCgpJIGFkbWl0LCB0aGF0IHdlIGFyZSBu b3QgY3VycmVudGx5IGNvbnNpZGVyaW5nIFZ1bGNhbiBhcyBwYXJ0IG9mIG91cgpjb21wdXRlIHN0 YWNrLiBJIGRvbid0IHRoaW5rIGl0IG1ha2VzIHNlbnNlIHRvIGJ1aWxkIFZ1bGNhbiBvbiB0b3Ag b2YKUk9DbSBiZWNhdXNlIHdlIGhhdmUgbm8gcGxhbnMgdG8gYWRkIGdyYXBoaWNzIHN1cHBvcnQg dG8gUk9DbS4gV2hhdApjb21tb25hbGl0eSB0aGVyZSBpcyBiZXR3ZWVuIFZ1bGNhbiBhbmQgb3Ro ZXIgY29tcHV0ZSBBUElzIHdpbGwgbmVlZCB0bwpiZSBhZGRyZXNzZWQgYXQgYSBsb3dlciBsZXZl bC4gSWYgZ3JhcGhpY3MgaXMgbW92aW5nIHRvIHVzZXIgbW9kZQpjb21tYW5kIHN1Ym1pc3Npb24g YW5kIGEgc2hhcmVkIHZpcnR1YWwgbWVtb3J5IG1vZGVsLCBJIHNlZSBhbgpvcHBvcnR1bml0eSB0 byBtb3ZlIFJPQ20gdG8gYSBjb21tb24gaW9jdGwgQVBJIG9uIHRob3NlIGZ1dHVyZSBoYXJkd2Fy ZQpnZW5lcmF0aW9ucy4KCgo+Cj4gLSBXaGF0J3MgbXVjaCB3b3JzZSwgYW5kIEkgZG9uJ3QgdGhp bmsgYW55b25lIGluIGFtZCBoYXMgcmVhbGl6ZWQgdGhpcyB5ZXQKPiAgIChhdCBsZWFzdCBub3Qg aW4gYSBwdWJsaWMgdGhyZWFkIEkndmUgc2VlbikuIEluIHZ1bGthbiB5b3UgbmVlZCB0byBiZQo+ ICAgYWJsZSB0byBzd2l0Y2ggZnJvbSBjb21wdXRlIG1vZGUgdG8gZG1hLWZlbmNlIG1vZGUgYWZ0 ZXIKPiAgIHBpcGVsaW5lcy9kZXZpY2VzIGhhdmUgYmVlbiBjcmVhdGVkIGFscmVhZHkuIFRoaXMg aXMgYmVjYXVzZSB3aW5zeXMgYXJlCj4gICBvbmx5IGluaXRpYWxpemVkIGluIGEgc2Vjb25kIHN0 ZXAsIHVudGlsIHRoYXQncyBkb25lIHlvdSBoYXZlIHRvCj4gICBwZXNzaW1pc3RpY2FsbHkgYXNz dW1lIHRoYXQgdGhlIHVzZXIgZG9lcyBwdXJlIGNvbXB1dGUuIFdoYXQncyB3b3JzZSBmb3IKPiAg IGJ1ZmZlciBzaGFyaW5nIHlvdSBkb24ndCBldmVuIGhhdmUgYSBjbGVhciBzaWduYWwgb24gdGhp cyBzdHVmZi4gU28KPiAgIGVpdGhlcgo+Cj4gICAtIHlvdSBmaWd1cmUgb3V0IGhvdyB0byBtaWdy YXRlIGFsbCB0aGUgYnVmZmVycyBhbmQgc3RhdGUgZnJvbSBhbWRrZmQgdG8KPiAgICAgYW1kZ3B1 IGF0IHJ1bnRpbWUsIGFuZCBkdXBsaWNhdGUgYWxsIHRoZSBmZWF0dXJlcy4gV2hpY2ggaXMgcmF0 aGVyCj4gICAgIHBvaW50bGVzcy4KPgo+ICAgLSBvciB5b3UgZHVwbGljYXRlIGFsbCB0aGUgY29t cHV0ZSBmZWF0dXJlcyB0byBhbWRncHUgc28gdGhhdCB2ayBjYW4gdXNlCj4gICAgIHRoZW0sIGFu ZCBzdGlsbCByZWFzb25hYmx5IGVhc3kgbWlncmF0ZSB0byB3aW5zeXMvZG1hLWZlbmNlIG1vZGUs Cj4gICAgIHdoaWNoIG1ha2VzIGFtZGtmZCByYXRoZXIgcmVkdW5kYW50LgoKVGhlIGJhc2ljICJj b21wdXRlIGZlYXR1cmVzIiB0aGF0IFJPQ20gZGVwZW5kcyBvbiBhcmUgYQpzaGFyZWQtdmlydHVh bC1tZW1vcnkgbW9kZWwgYW5kIHVzZXIgbW9kZSBjb21tYW5kIHN1Ym1pc3Npb24uIElmIHRob3Nl CmJlY29tZSBwYXJ0IG9mIHRoZSBncmFwaGljcyBpb2N0bCBBUEksIEkgc2VlIG5vIHByb2JsZW0g d2l0aAppbXBsZW1lbnRpbmcgcHJvcGVyIFZ1bGNhbiBjb21wdXRlIG9uIHRvcCBvZiBpdC4KCkZv ciBtb3JlIGNvbnNpc3RlbnQgc3VwcG9ydCBmb3IgdG9vbHMgc3VjaCBhcyBkZWJ1Z2dlciwgcHJv ZmlsZXJzLCBDUklVLApldGMuIGl0IHdvdWxkIG1ha2Ugc2Vuc2UgdG8gaGF2ZSBhIGNvbW1vbiBp b2N0bCBBUEkgZm9yIFJPQ20gYW5kIFZ1bGNhbiwKd2hpY2ggd291bGQgZWZmZWN0aXZlbHkgZGVw cmVjYXRlIHRoZSBLRkQgQVBJLiBBbmQgSSdtIGRlZmluaXRlbHkgb3Blbgp0byB3b3JraW5nIG9u IHRoYXQgZm9yIGZ1dHVyZSBIVyBnZW5lcmF0aW9ucyB0aGF0IGNhbiBzdXBwb3J0IGl0LgoKV2Un bGwgcHJvYmFibHkgaGF2ZSB0byBwaWNrIGEgdGFyZ2V0IEdQVSB3aGVyZSB3ZSBtYWtlIHN1Y2gg YSBzd2l0Y2gsCmFuZCB1c2UgdGhlIHRpbWUgdW50aWwgdGhlbiB0byBnZXQgdGhpbmdzIHJlYWR5 LCBwcm90b3R5cGluZyBpbiBjdXJyZW50CkdGWHYxMCBvciBuZXdlciBHUFVzLiBUaGlzIGZlZWxz IHRvIG1lIGEgYml0IGxpa2UgdGhlIHRyYW5zaXRpb24gZnJvbQpyYWRlb24gdG8gYW1kZ3B1LCB3 aGVyZSBHRlh2NyB3YXMgc3VwcG9ydGVkIGJ5IGJvdGggZm9yIHNvbWUgdGltZS4KCsKgCgo+Cj4g ICBJJ3ZlIGRpc2N1c3NlZCB0aGlzIHByb2JsZW0gZXh0ZW5zaXZlbHkgd2l0aCBKYXNvbiBFa3N0 cmFuZCwgYW5kIGl0J3MKPiAgIHJlYWxseSBuYXN0eS4KPgo+IFNvIHB1cmVseSBmcm9tIGEgdGVj aG5pY2FsIHBvdiwgb25seSBsb29raW5nIGF0IHRoZSBBTUQgcGVyc3BlY3RpdmUgaGVyZSwKPiB0 aGlzIGRvZXNuJ3QgbWFrZSBtdWNoIHNlbnNlIHRvIG1lLiBUaGUgb25seSByZWFzb24gdG8ga2Vl cCBkb3VibGluZyBkb3duCj4gb24gYW1ka2ZkIEknbSBzZWVpbmcgaXMgdGhhdCB5b3UndmUgYnVp bHQgeW91ciBjb21wdXRlIHJvY20gc3RhY2sgb24gdG9wCj4gb2YgaXQsIGFuZCBiZWNhdXNlIG9m IHRoYXQgdGhlIG9ubHkgb3B0aW9uIGlzIHRvIGtlZXAgZG9pbmcgdGhhdC4gV2hpY2gKPiBzdG9w cyBtYWtpbmcgc2Vuc2UgZXZlbnR1YWxseSwgYW5kIHdlJ3JlIGdldHRpbmcgdG8gdGhhdCBwb2lu dCBmb3Igc3VyZS4KCkFzIGxvbmcgYXMgb3VyIGNvbXB1dGUgR1BVcyBhcmUgYmFzZWQgb24gdGhl IEdGWHY5IGFyY2hpdGVjdHVyZSwgSSB0aGluawp3ZSBkb24ndCByZWFsbHkgaGF2ZSBhbiBhbHRl cm5hdGl2ZSB0aGFuIHRvIGRvdWJsZSBkb3duIG9uIEtGRC4gV2UgY2FuJ3QKanVzdCBzdG9wIHdv cmsgb24gS0ZEIGFuZCB0ZWxsIG91ciBjdXN0b21lcnMgdGhhdCB0aGV5J2xsIG5vdCBnZXQgYW55 Cm5ldyBmZWF0dXJlcyBmb3IgdGhlIG5leHQgMiBvciAzIHllYXJzIHRoYXQgaXQgdGFrZXMgdG8g YnVpbGQgYSBjb21tb24KdXBzdHJlYW0gbWVtb3J5IG1hbmFnZW1lbnQgYW5kIHNjaGVkdWxpbmcg QVBJIGFuZCB0aGUgSFcgdGhhdCBjYW4Kc3VwcG9ydCBpdC4KCgo+Cj4gVGhlIG90aGVyIHNpZGUg aXMgYSBiaXQgdGhlIHVwc3RyZWFtIHNpZGUsIGJ1dCB0aGF0J3MgYSBsb3Qgc21hbGxlcjoKPgo+ IC0gdnVsa2FuIGNvbXB1dGUgaXMgb25lIG9mIHRoZSBtb3JlIHJlYXNvbmFibGUgd2F5cyB0byBn ZXQgY3Jvc3MgdmVuZG9yCj4gICBjb21wdXRlIGVjb3N5c3RlbSBvZmYgdGhlIGdyb3VuZC4gQXQg bGVhc3QgZnJvbSB3aGF0IEkga25vdyBmcm9tCj4gICBiYWNrZ3JvdW5kIGNoYXR0ZXIsIHdoaWNo IHlvdSBndXlzIHByb2JhYmx5IGhhdmVuJ3QgYWxsIGhlYXJkLiBhbWRrZmQKPiAgIGJlaW5nIHRo ZSBzaW5nbGUgdmVyeSBvZGQgZHJpdmVyIGhlcmUgcmVxdWlyaW5nIGVudGlyZWx5IGRpZmZlcmVu dCB1YXBpCj4gICBmb3IgY29tcHV0ZSBtb2RlIGlzIG5vdCBnb2luZyB0byBiZSBncmVhdC4KCkxp a2UgSSBzYWlkLCBLRkQgaXMgbm90IGV4Y2x1c2l2ZSB3aXRoIGFueXRoaW5nIGVsc2UuCgpWdWxj YW4gbWF5IGJlIGEgZ29vZCBvcHRpb24gZm9yIGRlc2t0b3BzLiBGb3IgSFBDIHVzZXMgb3IgZ2Vu ZXJhbApwb3J0YWJsZSBub24tZ3JhcGhpY3MgY29tcHV0ZSBjb2RlLCBJIHRoaW5rIHNvbWV0aGlu ZyBsaWtlIE9wZW5NUCBvcgpmdXR1cmUgQysrIHN0YW5kYXJkcyB3b3VsZCBiZSBhIGJldHRlciB3 YXkgZm9yd2FyZC4gVGhlcmUgaXMgbm8KdGVjaG5pY2FsIHJlYXNvbiB3aHkgYm90aCBjYW5ub3Qg Y29leGlzdCBpbiB0aGUgc2FtZSBhcHBsaWNhdGlvbi4KCgo+Cj4gLSBDUklVIHdpbGwgbmVlZCBu ZXcgYWNjZXNzIHJpZ2h0cyBoYW5kbGluZyAoZm9yIHRoZSBzYXZlL3Jlc3RvcmUvcmVzdW1lCj4g ICBzdHVmZiB5b3UncmUgYWRkaW5nKS4gR2VuZXJhbGx5IHdlIHN0YW5kYXJkaXplIGFjY2VzcyBy aWdodHMgY2hlY2tzCj4gICBhY3Jvc3MgZHJpdmVycywgYW5kIGxlYXZlIGV2ZXJ5dGhpbmcgZWxz ZSB0byByZW5kZXIgZHJpdmVycyAoY29tbWFuZAo+ICAgc3VibWlzc2lvbiwgbWVtb3J5IG1hbmFn ZW1lbnQsIC4uLikuIEJ5IGFkZGluZyBDUklVIHN1cHBvcnQgdG8gYW1ka2ZkCj4gICB3ZSBwcmV0 dHkgbXVjaCBndWFyYW50ZWUgdGhhdCB3ZSB3b250IGJlIGFibGUgdG8gc3RhbmRhcmRpemUgQ1JJ VSBhY2Nlc3MKPiAgIHJpZ2h0cyBhY3Jvc3MgZHJpdmVycy4gV2hpY2gganVzdCBwbGFpbnMgc3Vj a3MgZnJvbSBhbgo+ICAgdXBzdHJlYW0vY3Jvc3MtdmVuZG9yIGVjb3N5c3RlbSBwb3YuCgpCeSBh Y2Nlc3MgcmlnaHRzLCBkbyB5b3UgbWVhbiByZXF1aXJpbmcgcm9vdCBmb3Igc29tZSBpb2N0bHMs IG9yIGJlaW5nCnB0cmFjZS1hdHRhY2hlZCBmb3Igb3RoZXJzPyBUaGVzZSBhcmUgZHJpdmVuIGJ5 IGhvdyBDUklVIHdvcmtzIGFuZAppbnRlcmFjdHMgd2l0aCBpdHMgdGFyZ2V0IHByb2Nlc3Nlcy4g SSB0aGluayB0aGV5IHdpbGwgYXBwbHkgZXF1YWxseSB0bwphbnkgZHJpdmVyIGltcGxlbWVudGlu ZyBDUklVIHN1cHBvcnQuIEkgZG9uJ3Qgc2VlIGhvdyBncmFwaGljcyBkcml2ZXJzCnNwZWNpZmlj YWxseSBkcml2ZSBzdGFuZGFyZGl6YXRpb24gb2YgQ1JJVSBhY2Nlc3MgcmlnaHRzLgoKCj4KPiBB bmQgeWVzIHdlJ2Qgc3RpbGwgbmVlZCBhIHBlci1kcml2ZXIgY3JpdSBwbHVnaW4gaW4gdXNlcnNw YWNlLCBidXQgdGhlCj4gc2FtZSBpcyB0cnVlIGZvciBhbWR2bGsvcmFkdi9hbnYvIGFuZCBhbGwg dGhlIG90aGVyIGRyaXZlcnMgd2UgaGF2ZToKPiBEcml2ZXIgaXMgZGlmZmVyZW50LCBhY2Nlc3Mg cmlnaHQgbWFuYWdlbWVudCBpcyBzdGlsbCB0aGUgc2FtZS4KPgo+IEFuZCBzZWNvbmRseSwganVz dCBiZWNhdXNlIG52aWRpYSByZWZ1c2VzIHRvIGNvbGxhYm9yYXRlIGluIGFueQo+IHN0YW5kYXJk cyBhcm91bmQgZ3B1IGNvbXB1dGUgZG9lc24ndCBtZWFuIHRoYXQncyBhIGdvb2QgcmVhc29uIGZv ciB1cyB0bwo+IGRvIHRoZSBzYW1lIGluIHVwc3RyZWFtLgo+Cj4gVGhpcmRseSwgaXQgc291bmRz IGxpa2UgdGhpcyBpcyB0aGUgZmlyc3QgZGV2aWNlLWRyaXZlciBDUklVIHN1cHBvcnQsIHNvIEkK PiB0aGluayB3ZSBuZWVkIGEgc29saWQgYWdyZWVtZW50L3N0YW5kYXJkIGhlcmUgdG8gc2V0IGFz IGFuIGV4YW1wbGUgZm9yCj4gZXZlcnlvbmUgZWxzZS4gVGhlcmUncyBhbGwgdGhlIEFJIGFjY2Vs IGNoaXBzIGFuZCBmcGdhLWZvci1jb21wdXRlIHN0dWZmCj4gdGhhdCBJIGV4cGVjdCB3aWxsIGV2 ZW50dWFsbHkgYWxzbyBnYWluIENSSVUgc3VwcG9ydC4KClN1cmUuIE15IGV4cGVyaWVuY2Ugd2l0 aCBDUklVIHNvIGZhciBpcywgdGhhdCBDUklVIHJlcXVpcmVzIHJvb3QgKGV2ZW4Kd2l0aG91dCBv dXIgcGx1Z2luKS4gUmVnYXJkbGVzcyBvZiB0aGF0LCBDUklVIGF0dGFjaGVzIHRvIHRoZSB0YXJn ZXQKcHJvY2VzcyB3aXRoIHB0cmFjZS4gVGhlIGtlcm5lbCdzIHNlY3VyaXR5IHBvbGljeSBmb3Ig YWxsb3dpbmcgcHRyYWNlCmFjY2VzcyBpcyBpbmZsdWVuY2VkIGJ5IG1hbnkgZmFjdG9ycy4gV2Ug Y2hvc2Ugbm90IHRvIGR1cGxpY2F0ZSB0aGlzIGluCnRoZSBkcml2ZXIuIFRoZXJlZm9yZSB3ZSBj aGVjayB3aGV0aGVyIHRoZSBjYWxsZXIgb2Ygb3VyIENSSVUgZHVtcCBpb2N0bAppcyBwdHJhY2Ug YXR0YWNoZWQuIElmIGl0IGlzLCBpdCBtZWFucyBpdCBoYXMgcGFzc2VkIGFsbCB0aGUga2VybmVs J3MKc2VjdXJpdHkgY2hlY2tzLiBXZSBkZWxpYmVyYXRlbHkgZG8gbm90IHdhbnQgdG8gaW1wbGVt ZW50IG91ciBvd24Kc2VjdXJpdHkgcG9saWN5IGluIHRoZSBkcml2ZXIuCgpUaGVyZSB3b3VsZCBi ZSB3YXlzIHRvIHJlbW92ZSB0aGUgcm9vdCByZXF1aXJlbWVudCBmcm9tIG91ciBDUklVCnJlc3Rv cmUtaW9jdGwsIGlmIHRoYXQgYmVjb21lcyBhIHJlcXVpcmVtZW50IGZvciBDUklVLiBJdCB3b3Vs ZCBtZWFuCnNvbWUgc2FuaXR5IGNoZWNraW5nIG9mIHRoZSBIVyBzdGF0ZSB0aGF0J3MgZ2V0dGlu ZyByZXN0b3JlZC4KCgo+Cj4gU28gSSdtIGFza2luZyBvbmNlIG1vcmUsIGlzIHRoaXMgX3JlYWxs eV8gdGhlIHJpZ2h0IHBhdGggZm9yd2FyZD8gQm90aCBmb3IKPiBhbWQgKGF0IGxlYXN0IGxvbmcg dGVybSksIGJ1dCBhbHNvIHNvbWV3aGF0IGZvciB1cHN0cmVhbS4KCkkgdGhpbmsgbG9uZy10ZXJt IGl0IG1ha2VzIHNlbnNlIHRvIHBsYW4gZm9yIGEgdW5pZmllZCBBUEkuIEZvciBjdXJyZW50Cmhh cmR3YXJlIEkgZG9uJ3Qgc2VlIGEgd2F5IHRvIGdldCB0aGVyZSBmb3IgbXVsdGlwbGUgcmVhc29u czoKCiAxLiBNYWludGFpbmluZyBiYWNrd2FyZHMgY29tcGF0aWJpbGl0eSB3aXRoIGV4aXN0aW5n IHVzZXIgbW9kZQogMi4gTGFja2luZyBIVyBjYXBhYmlsaXRpZXMgZm9yIGdyYXBoaWNzIHVzZXIg bW9kZSBjb21tYW5kIHN1Ym1pc3Npb24KICAgIGFuZCBIVyBzY2hlZHVsaW5nCgpUaGVyZWZvcmUg S0ZEIHdpbGwgYmUgaW1wb3J0YW50IGZvciBBTUQncyBjb21wdXRlIEdQVSBzdHJhdGVneSBhbmQg b3VyCmN1c3RvbWVycyBmb3IgeWVhcnMgdG8gY29tZS4gTm90IGJlaW5nIGFibGUgdG8gY29udGlu dWUgZGV2ZWxvcGluZyBpdAp1cHN0cmVhbSB3b3VsZCBiZSBhIG1ham9yIGltcGVkaW1lbnQgZm9y IEFNRC4gQWdhaW4sIHRoaXMgaXMgbXkgcGVyc29uYWwKb3Bpbmlvbi4KClJlZ2FyZHMsCsKgIEZl bGl4CgoKPgo+IENoZWVycywgRGFuaWVsCj4KPgo+PiBXaGF0J3Mgd29ya2luZyBhbmQgdGVzdGVk IGF0IHRoaXMgcG9pbnQ6Cj4+Cj4+ICAgKiBDaGVja3BvaW50IGFuZCByZXN0b3JlIGFjY2VsZXJh dGVkIG1hY2hpbmUgbGVhcm5pbmcgYXBwczogUHlUb3JjaAo+PiAgICAgcnVubmluZyBCZXJ0IG9u IHN5c3RlbXMgd2l0aCAxIG9yIDIgR1BVcyAoTUk1MCBvciBNSTEwMCksIDEwMCUKPj4gICAgIHVu bW9kaWZpZWQgdXNlciBtb2RlIHN0YWNrCj4+ICAgKiBDaGVja3BvaW50IG9uIG9uZSBzeXN0ZW0s IHJlc3RvcmUgb24gYSBkaWZmZXJlbnQgc3lzdGVtCj4+ICAgKiBDaGVja3BvaW50IG9uIG9uZSBH UFUsIHJlc3RvcmUgb24gYSBkaWZmZXJlbnQgR1BVCj4+Cj4+IE1ham9yIEtub3duIGlzc3VlczoK Pj4KPj4gICAqIFRoZSBLRkQgaW9jdGwgQVBJIGlzIG5vdCBmaW5hbDogTmVlZHMgYSBjb21wbGV0 ZSByZWRlc2lnbiB0byBhbGxvdwo+PiAgICAgZnV0dXJlIGV4dGVuc2lvbiB3aXRob3V0IGJyZWFr aW5nIHRoZSBBQkkKPj4gICAqIFZlcnkgc2xvdzogTmVlZCB0byBpbXBsZW1lbnQgRE1BIHRvIGR1 bXAgVlJBTSBjb250ZW50cwo+Pgo+PiBNaXNzaW5nIG9yIGluY29tcGxldGUgZmVhdHVyZXM6Cj4+ Cj4+ICAgKiBTdXBwb3J0IGZvciB0aGUgbmV3IEtGRCBTVk0gQVBJCj4+ICAgKiBDaGVjayBkZXZp Y2UgdG9wb2xvZ3kgZHVyaW5nIHJlc3RvcmUKPj4gICAqIENoZWNrcG9pbnQgYW5kIHJlc3RvcmUg bXVsdGlwbGUgcHJvY2Vzc2VzCj4+ICAgKiBTdXBwb3J0IGZvciBhcHBsaWNhdGlvbnMgdXNpbmcg TWVzYSBmb3IgdmlkZW8gZGVjb2RlL2VuY29kZQo+PiAgICogVGVzdGluZyB3aXRoIG1vcmUgZGlm ZmVyZW50IEdQVXMgYW5kIHdvcmtsb2Fkcwo+Pgo+PiBCaWcgT3BlbiBxdWVzdGlvbnM6Cj4+Cj4+ ICAgKiBXaGF0J3MgdGhlIHByZWZlcnJlZCB3YXkgdG8gcHVibGlzaCBvdXIgQ1JJVSBwbHVnaW4/ IEluLXRyZWUgb3IKPj4gICAgIG91dC1vZi10cmVlPwo+PiAgICogV2hhdCdzIHRoZSBwcmVmZXJy ZWQgd2F5IHRvIGRpc3RyaWJ1dGUgb3VyIENSSVUgcGx1Z2luPyBTb3VyY2U/Cj4+ICAgICBCaW5h cnkgLnNvPyBXaG9sZSBDUklVPyBKdXN0IGluLWJveCBzdXBwb3J0Pwo+PiAgICogSWYgb3VyIHBs dWdpbiBjYW4gYmUgdXBzdHJlYW1lZCBpbiB0aGUgQ1JJVSB0cmVlLCB3aGF0IHdvdWxkIGJlIHRo ZQo+PiAgICAgcmlnaHQgZGlyZWN0b3J5Pwo+Pgo+PiBCZXN0IHJlZ2FyZHMsCj4+ICAgRmVsaXgK Pj4KPj4KPj4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18K Pj4gZHJpLWRldmVsIG1haWxpbmcgbGlzdAo+PiBkcmktZGV2ZWxAbGlzdHMuZnJlZWRlc2t0b3Au b3JnCj4+IGh0dHBzOi8vbGlzdHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4vbGlzdGluZm8vZHJp LWRldmVsCj4gLS0KPiBEYW5pZWwgVmV0dGVyCj4gU29mdHdhcmUgRW5naW5lZXIsIEludGVsIENv cnBvcmF0aW9uCj4gaHR0cDovL2Jsb2cuZmZ3bGwuY2gvCl9fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fCmFtZC1nZnggbWFpbGluZyBsaXN0CmFtZC1nZnhAbGlz dHMuZnJlZWRlc2t0b3Aub3JnCmh0dHBzOi8vbGlzdHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4v bGlzdGluZm8vYW1kLWdmeAo=