From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8EE84C43460 for ; Mon, 10 May 2021 16:36:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6992761107 for ; Mon, 10 May 2021 16:36:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231281AbhEJQhs (ORCPT ); Mon, 10 May 2021 12:37:48 -0400 Received: from mail-co1nam11on2042.outbound.protection.outlook.com ([40.107.220.42]:58336 "EHLO NAM11-CO1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S230497AbhEJQhr (ORCPT ); Mon, 10 May 2021 12:37:47 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RLQ7lNFzHwMmiZdt8qAMYmQJBXCdFNft6y8/EOc1nAhs415oFtRqGXlrhEgL3Cd/TQ9aNOmElceD9yix0/EZ4kphj9MLWZZ+7Ggo12Rx+qLDLKgASpqG7arFqMBQB6RfIii8tC9tDdR6ErqolqgYw8pMcLQpqxGw0eTqe3x07PLxVlyb7O0HVfvYQe/IKaIwpWJlAKeBd/kbZlZO56fTQ6bWT0VLStws41OSqgy9PnIfyJaWjNwx+abgKxNAgi8h4ieRIc+krAmsA98+8rV/1fAwlhHjqHAkSPMMJAOOHndb4hvboZFU1PO7iQLRtnXHXkKFSrV8VGcBsI6lT+XPgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WH9mV/4G+pvjgm69H/sukJ+57rWHS6bSAKI6Wq2WwqI=; b=TRIjNJZXfjXKcKg0XR/Pon6Sne5iP+gglrTAclgwzMJNqTD8HGXou3qcV+3UZugJd8+Em4kXWX2nCxNfy1vTjnyHF7AJuXsJIblCQMZ+O/+St2FgTOzhw6Vbq0djSHqTa6Kp9DP1fzjEYXCOIPcsA9uLwGus00W96rgQ2mhm2SGkXEZbjwW8BlHIp3OIh4ggw3Rir2J4Srowzf5ocEP2tWPNn5YJvPEUe9Lx5TgGJUs9Me3DBHG1cV9SgFIqgNZvY/ppJVmkCrmkvrfNgI2Roxeg7XKzj3XbaX9Sx7eJXOqpJ4NKTjU/OVdcrq7QU4gHKaE5F6L9ZY+0GywbF57OBg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WH9mV/4G+pvjgm69H/sukJ+57rWHS6bSAKI6Wq2WwqI=; b=fLD2k7OtJ+7J+/1Y8V8ZjfYE7uM8dYtaUOLaBl2Rlft+ypuXMUmyEOOtQo5JxP5iyKhTGsVzvR9deVh11THQG7QfxNy3HN/gRz8mfwwN9g6AmMTQB70n57lI5z8p+zVhkmsI0lveMwt+hZCu9bl0WX4sUlMm/GY+sMlZoQqv4bI= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none;lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from SN6PR12MB4623.namprd12.prod.outlook.com (2603:10b6:805:e9::17) by SA0PR12MB4462.namprd12.prod.outlook.com (2603:10b6:806:95::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4108.25; Mon, 10 May 2021 16:36:41 +0000 Received: from SN6PR12MB4623.namprd12.prod.outlook.com ([fe80::ad51:8c49:b171:856c]) by SN6PR12MB4623.namprd12.prod.outlook.com ([fe80::ad51:8c49:b171:856c%7]) with mapi id 15.20.4108.031; Mon, 10 May 2021 16:36:41 +0000 From: Andrey Grodzovsky To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, linux-pci@vger.kernel.org, ckoenig.leichtzumerken@gmail.com, daniel.vetter@ffwll.ch, Harry.Wentland@amd.com Cc: ppaalanen@gmail.com, Alexander.Deucher@amd.com, gregkh@linuxfoundation.org, helgaas@kernel.org, Felix.Kuehling@amd.com, Andrey Grodzovsky Subject: [PATCH v6 00/16] RFC Support hot device unplug in amdgpu Date: Mon, 10 May 2021 12:36:09 -0400 Message-Id: <20210510163625.407105-1-andrey.grodzovsky@amd.com> X-Mailer: git-send-email 2.25.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Originating-IP: [2607:fea8:3edf:49b0:6a5:47b8:e610:f6a3] X-ClientProxiedBy: YT2PR01CA0020.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:38::25) To SN6PR12MB4623.namprd12.prod.outlook.com (2603:10b6:805:e9::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from agrodzovsky-All-Series.hitronhub.home (2607:fea8:3edf:49b0:6a5:47b8:e610:f6a3) by YT2PR01CA0020.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:38::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4108.31 via Frontend Transport; Mon, 10 May 2021 16:36:40 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ee1b6dff-ae9c-49a8-d82f-08d913d1ca14 X-MS-TrafficTypeDiagnostic: SA0PR12MB4462: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: TIQxZkXXTman37N440jTguZDI5anXlTIleZ9e7Ci34SiXFUwgfcZHWnX3eT9Rxpvpgo9swllWMHBRdds6fveymthIt031gnGC0/NaxSve3YEXtHqTLRUlobpNEsxcw86dEq92ndkEm9DYuiw3ddsl4LnNCywmbFd0KiDKVzK62b9CJkRw0mygDazpIwC65X7Vxrzz/mPZEAhu6QsXHi9BJl2PPAJV6OzNjJ2u+lMSous6FhQ/5ZBWhuq7BEdl7QKMaFhyBHGjmFyBCW1NGWCi9pAnWc+hS8jCrCYczYIpHmBwtpYHQdzRRWuraIY7ThrS8h7+wmbwico2kCxuuq3kD6vbnSURm1JZWsoidVCtnRCku2RUK4UiqBIJ8w7JJvGMiE3+JuHXn2DxzjiP/6qNBuUvMA+fIfDoWSW5lRikLvyg5VsqIC2VJTftb+yHdZkSZFWufNx09kOWSFZfwdezqdbnYFxT+Clz4QExNZqPF+VLqhbvyYxJ+8vuN+FFarni31eRrtEIwRMjQHHGQBcuGBUz4u8Y4iX8SeVmO0GCjaLpr45am9Spq2k9uK0Vd37F82/a7k5D8+YvTUfbRDt15YD8EtwhnJl4PPjU0PsyH0hpY5eGES847FeoDA7/BNedA6fFAgik+VupxMqicLdjixl6AcmuclhJLrMdlGp7hk+wtz1LKJVECJxLkx6AFFgXjsYv2WZRi5MlpJatv0pCQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SN6PR12MB4623.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(366004)(136003)(39860400002)(396003)(376002)(346002)(86362001)(6506007)(38100700002)(52116002)(966005)(6666004)(44832011)(8676002)(36756003)(5660300002)(478600001)(8936002)(2906002)(6486002)(66946007)(2616005)(66476007)(66556008)(1076003)(316002)(6636002)(6512007)(16526019)(186003)(83380400001)(4326008);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?utf-8?B?YkR4NUJlaHBjQVNQMU9hQVFMMFFEUXdFdU5kbVB3eHhJeHZ3MHNtMFVCblRK?= =?utf-8?B?UmIwRUN4UzJ4RENhTk1vWWRNbWREeDhZN2JBU3ZLeEh4Ry83L0ZNeW9ocG5L?= =?utf-8?B?eWcxakUxL3dta2pFODVBbFQzdkU4bDlZYnU0MXhYanY0RW1WenZiK1ZnUmtp?= =?utf-8?B?NFhWcmZnbmdJNUNQa2RzY2wyRysyNVQ3SjE5eW95VS9FRzk4L3hhL01hb1d2?= =?utf-8?B?aEpBbnNXdkF6S3hQajkwUHB0a1Q5bWRNSmlJU3dHRTBwbFN4OFZCWFN1djFS?= =?utf-8?B?YzdOT1lvRnJ4cjI2eFNhUVVJbXZYYklSMCtVSXVudHV4OVJmZ0M2WE1pdEZJ?= =?utf-8?B?aXBhRU5EMndvbDltOEtyanFPQ0k3NXVybVVSRzdlVnJ3R2cvdXhHa1E3dkU4?= =?utf-8?B?YjNEOFNiOTZuK0V2MklKRmQyR1ZJRUI4RWQ2OEw2WTArcG43N2hFaGptUk9m?= =?utf-8?B?WlRTQ0YrYWhBN29scDFzWElPaG5oNndpbXFXNUl3WDNmRzJKUnZ5enZ5ajNB?= =?utf-8?B?Yk80SGYxWkQ5ZTdxcXJNSW1HTjhQczBLYUk0Zm1YTHE4Z1RRK21NaXd0dllD?= =?utf-8?B?YUZvazRpb1FsaGtGeUtPeXdHcVZxdUZkTW9obVY5VkV6ZTBCSWx6QnJNd2Q3?= =?utf-8?B?clVtTnFvSmFDK3dZNzBOZFRUUGZ3TStkQUY5QkQ1OXFIRHJPa1RuYUJ3QWh6?= =?utf-8?B?QWJpRmZSYmRLendYYmlSeE4yNlUwRHpScFVWMDU5RDU1U1pRRVM3OTUrYXJY?= =?utf-8?B?ZTRFZlZDSk9mcHlNbkpvQ2huOER4RStFZVRUOTV0djYzR3VJNWxwNXhFK1N0?= =?utf-8?B?dTByUXJoVUdHeFNoOFBCSkdVWFV5NW1kdlNNbnAybndqVFgzbjVHMnJSaDY0?= =?utf-8?B?a0c0dXl6SlhLcjQ0L2JPL1RGOGdDcE1PR0VBeWx5dU9aM3NBRjVjZDJMNWd4?= =?utf-8?B?S2o1clphTEpFRDdXTnRYdXFVa3pKRjJ2Y2s4cGg1bVN2Y1dyWXpXZVE0OXBx?= =?utf-8?B?NmRqekE0aHhjTWc1MWl3QXNPT0ZoSm1LUmQxUUdDNkFMQTc3L3BOTXFnczRu?= =?utf-8?B?MU03d0Z3UVdHK0JrSlAyYzZ3S1lKVDlHR2g1aVdXdTZPa1E3SGNzWlQ2MXgy?= =?utf-8?B?Ty9XNEpXMmxmWlhnL2YxTmxBZzgzTU5iNWRFN2hIY0xKaDJ1SmhmZlo2MjBx?= =?utf-8?B?SlNvYndnMkZvSWZGOXZMdmtXSHVPYUVSc09XS2NnalVFK0c0ejdlbFB2ZnVD?= =?utf-8?B?UnRUUmtydGRKRE9OdzJmd2hxUWI5aEZRem5XS3FQTW9KZUhiRXhwT2ZWVjhp?= =?utf-8?B?bGNsVjFEUForejVJZE93aUtKTjgxV1EzTmZKZWhNK0JUS2JJNnpUbFdQWFVI?= =?utf-8?B?SkNOYnp4QTgzeW5WcGQyRENzMzJtZmF5NExNNHV3N1VPdHhYV0pLU1dDc2Iw?= =?utf-8?B?NVk5RzZvMmlEdTlZbVNCVkNFNjBHRW9YK1BSU0s3RVVqWjJCZExqMnp2eXlU?= =?utf-8?B?N2lLaUtENGxVN0RrOU4rd1hLZW5tdWY4UFc1QlE3V3BPKzJQL3I2bTE5OTJU?= =?utf-8?B?RWIxbDR3WVlDWVZjYVJjTjFHS3oxOWdWOFF1U1htWE9CWjdLQXplVzJFblEv?= =?utf-8?B?WnR2MDBnd0VSSXF1TWJoTmxqOHdZR3JLNkhScWFidkMzTEtuUWNiaTRaLzJP?= =?utf-8?B?YnhXdm4wVFMyYkZBbm5BOEx1d0RBbEhZSkZHSHkwOWNpeVNCcXJyZDJTempn?= =?utf-8?B?SkJJYTVhaXJzV0FORzZ0NStObmNTVDZEQjNxVGNBaGlYZjE1aEpQOHN3cjdh?= =?utf-8?B?RGQybXU3UDNESVBkR09qYmNpd2gwSm84UloyMmdJb0kzeGJmcVNUOFNYU1NF?= =?utf-8?Q?6FlReWY/dBT+5?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: ee1b6dff-ae9c-49a8-d82f-08d913d1ca14 X-MS-Exchange-CrossTenant-AuthSource: SN6PR12MB4623.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 May 2021 16:36:41.0761 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: gg2/cqAcVOkyV5by/Vh5xQyiFz1Po2KRvwU7imuk5QeMgJgAAx1f97+nyO00cH2m5lOTeGnD5ReSTQ3YbIwzeg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB4462 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Until now extracting a card either by physical extraction (e.g. eGPU with thunderbolt connection or by emulation through  sysfs -> /sys/bus/pci/devices/device_id/remove) would cause random crashes in user apps. The random crashes in apps were mostly due to the app having mapped a device backed BO into its address space and was still trying to access the BO while the backing device was gone. To answer this first problem Christian suggested fixing the handling of mapped memory in the clients when the device goes away by forcibly unmapping all buffers the user processes have by clearing their respective VMAs mapping the device BOs. Then when the VMAs try to fill in the page tables again we check in the fault handler if the device is removed and if so, return an error. This will generate a SIGBUS to the application which can then cleanly terminate. This indeed was done but this in turn created a problem of kernel OOPs where the OOPSes were due to the fact that while the app was terminating because of the SIGBUS it would trigger use after free in the driver by calling to access device structures that were already released from the pci remove sequence. This was handled by introducing a 'flush' sequence during device removal where we wait for drm file reference to drop to 0 meaning all user clients directly using this device terminated. v2: Based on discussions in the mailing list with Daniel and Pekka [1] and based on the document produced by Pekka from those discussions [2] the whole approach with returning SIGBUS and waiting for all user clients having CPU mapping of device BOs to die was dropped. Instead as per the document suggestion the device structures are kept alive until the last reference to the device is dropped by user client and in the meanwhile all existing and new CPU mappings of the BOs belonging to the device directly or by dma-buf import are rerouted to per user process dummy rw page.Also, I skipped the 'Requirements for KMS UAPI' section of [2] since i am trying to get the minimal set of requirements that still give useful solution to work and this is the'Requirements for Render and Cross-Device UAPI' section and so my test case is removing a secondary device, which is render only and is not involved in KMS. v3: More updates following comments from v2 such as removing loop to find DRM file when rerouting page faults to dummy page,getting rid of unnecessary sysfs handling refactoring and moving prevention of GPU recovery post device unplug from amdgpu to scheduler layer. On top of that added unplug support for the IOMMU enabled system. v4: Drop last sysfs hack and use sysfs default attribute. Guard against write accesses after device removal to avoid modifying released memory. Update dummy pages handling to on demand allocation and release through drm managed framework. Add return value to scheduler job TO handler (by Luben Tuikov) and use this in amdgpu for prevention of GPU recovery post device unplug Also rebase on top of drm-misc-mext instead of amd-staging-drm-next v5: The most significant in this series is the improved protection from kernel driver accessing MMIO ranges that were allocated for the device once the device is gone. To do this, first a patch 'drm/amdgpu: Unmap all MMIO mappings' is introduced. This patch unamps all MMIO mapped into the kernel address space in the form of BARs and kernel BOs with CPU visible VRAM mappings. This way it helped to discover multiple such access points because a page fault would be immediately generated on access. Most of them were solved by moving HW fini code into pci_remove stage (patch drm/amdgpu: Add early fini callback) and for some who were harder to unwind drm_dev_enter/exit scoping was used. In addition all the IOCTLs and all background work and timers are now protected with drm_dev_enter/exit at their root in an attempt that after drm_dev_unplug is finished none of them run anymore and the pci_remove thread is the only thread executing which might touch the HW. To prevent deadlocks in such case against threads stuck on various HW or SW fences patches 'drm/amdgpu: Finalise device fences on device remove' and drm/amdgpu: Add rw_sem to pushing job into sched queue' take care of force signaling all such existing fences and rejecting any newly added ones. v6: Drop using drm_dev_enter/exit in conjunction with signalling HW fences before setting drm_dev_unplug. We need to devise a more robust cros DRM approach to the problem of dma fence waits falling inside drm_dev_enter/exit scopes -> move to TODO. With these patches I am able to gracefully remove the secondary card using sysfs remove hook while glxgears is running off of secondary card (DRI_PRIME=1) without kernel oopses or hangs and keep working with the primary card or soft reset the device without hangs or oopses. Also as per Daniel's comment I added 3 tests to IGT [4] to core_hotunplug test suite - remove device while commands are submitted, exported BO and exported fence (not pushed yet). Also now it's possible to plug back the device after unplug Also some users now can successfully use those patches with eGPU boxes[3]. TODOs for followup work: Convert AMDGPU code to use devm (for hw stuff) and drmm (for sw stuff and allocations) (Daniel) Add support for 'Requirements for KMS UAPI' section of [2] - unplugging primary, display connected card. Annotate drm_dev_enter/exit against dma_fence_waits as first in deciding where to use drm_dev_enter/exit in code for device unplug. [1] - Discussions during v5 of the patchset https://lore.kernel.org/amd-gfx/20210428151207.1212258-1-andrey.grodzovsky@amd.com/ [2] - drm/doc: device hot-unplug for userspace https://www.spinics.net/lists/dri-devel/msg259755.html [3] - Related gitlab ticket https://gitlab.freedesktop.org/drm/amd/-/issues/1081 [4] - Related IGT tests https://gitlab.freedesktop.org/agrodzov/igt-gpu-tools/-/commits/master Andrey Grodzovsky (16): drm/ttm: Remap all page faults to per process dummy page. drm/ttm: Expose ttm_tt_unpopulate for driver use drm/amdgpu: Split amdgpu_device_fini into early and late drm/amdkfd: Split kfd suspend from devie exit drm/amdgpu: Add early fini callback drm/amdgpu: Handle IOMMU enabled case. drm/amdgpu: Remap all page faults to per process dummy page. PCI: Add support for dev_groups to struct pci_device_driver drm/amdgpu: Convert driver sysfs attributes to static attributes drm/amdgpu: Guard against write accesses after device removal drm/sched: Make timeout timer rearm conditional. drm/amdgpu: Prevent any job recoveries after device is unplugged. drm/amdgpu: Fix hang on device removal. drm/scheduler: Fix hang when sched_entity released drm/amd/display: Remove superflous drm_mode_config_cleanup drm/amdgpu: Verify DMA opearations from device are done drivers/gpu/drm/amd/amdgpu/amdgpu.h | 6 +- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_atombios.c | 17 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 98 +++++++++++++------ drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 26 ++++- drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c | 31 ++++-- drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c | 3 +- drivers/gpu/drm/amd/amdgpu/amdgpu_gart.h | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 9 ++ drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.c | 25 ++--- drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c | 17 +++- drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c | 35 +++++-- drivers/gpu/drm/amd/amdgpu/amdgpu_irq.h | 3 +- drivers/gpu/drm/amd/amdgpu/amdgpu_job.c | 19 +++- drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 12 ++- drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c | 63 +++++++----- drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h | 2 + drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c | 70 +++++++++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h | 52 ++-------- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 21 +++- drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c | 31 ++++-- drivers/gpu/drm/amd/amdgpu/amdgpu_vce.c | 11 ++- drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.c | 22 +++-- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 7 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 14 +-- drivers/gpu/drm/amd/amdgpu/cik_ih.c | 3 +- drivers/gpu/drm/amd/amdgpu/cz_ih.c | 3 +- drivers/gpu/drm/amd/amdgpu/iceland_ih.c | 3 +- drivers/gpu/drm/amd/amdgpu/navi10_ih.c | 5 +- drivers/gpu/drm/amd/amdgpu/psp_v11_0.c | 44 ++++----- drivers/gpu/drm/amd/amdgpu/psp_v12_0.c | 8 +- drivers/gpu/drm/amd/amdgpu/psp_v3_1.c | 8 +- drivers/gpu/drm/amd/amdgpu/si_ih.c | 3 +- drivers/gpu/drm/amd/amdgpu/tonga_ih.c | 3 +- drivers/gpu/drm/amd/amdgpu/vce_v4_0.c | 26 +++-- drivers/gpu/drm/amd/amdgpu/vcn_v3_0.c | 22 +++-- drivers/gpu/drm/amd/amdgpu/vega10_ih.c | 5 +- drivers/gpu/drm/amd/amdgpu/vega20_ih.c | 2 +- drivers/gpu/drm/amd/amdkfd/kfd_device.c | 3 +- .../gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 13 ++- drivers/gpu/drm/amd/include/amd_shared.h | 2 + .../drm/amd/pm/powerplay/smumgr/smu7_smumgr.c | 2 + drivers/gpu/drm/scheduler/sched_entity.c | 3 +- drivers/gpu/drm/scheduler/sched_main.c | 35 ++++++- drivers/gpu/drm/ttm/ttm_bo_vm.c | 57 ++++++++++- drivers/gpu/drm/ttm/ttm_tt.c | 1 + drivers/pci/pci-driver.c | 1 + include/drm/ttm/ttm_bo_api.h | 2 + include/linux/pci.h | 3 + 51 files changed, 585 insertions(+), 272 deletions(-) -- 2.25.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82437C433B4 for ; Mon, 10 May 2021 16:36:44 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3A56E6115C for ; Mon, 10 May 2021 16:36:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3A56E6115C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 8D1D06E89A; Mon, 10 May 2021 16:36:43 +0000 (UTC) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2055.outbound.protection.outlook.com [40.107.94.55]) by gabe.freedesktop.org (Postfix) with ESMTPS id AD2926E89A; Mon, 10 May 2021 16:36:42 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RLQ7lNFzHwMmiZdt8qAMYmQJBXCdFNft6y8/EOc1nAhs415oFtRqGXlrhEgL3Cd/TQ9aNOmElceD9yix0/EZ4kphj9MLWZZ+7Ggo12Rx+qLDLKgASpqG7arFqMBQB6RfIii8tC9tDdR6ErqolqgYw8pMcLQpqxGw0eTqe3x07PLxVlyb7O0HVfvYQe/IKaIwpWJlAKeBd/kbZlZO56fTQ6bWT0VLStws41OSqgy9PnIfyJaWjNwx+abgKxNAgi8h4ieRIc+krAmsA98+8rV/1fAwlhHjqHAkSPMMJAOOHndb4hvboZFU1PO7iQLRtnXHXkKFSrV8VGcBsI6lT+XPgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WH9mV/4G+pvjgm69H/sukJ+57rWHS6bSAKI6Wq2WwqI=; b=TRIjNJZXfjXKcKg0XR/Pon6Sne5iP+gglrTAclgwzMJNqTD8HGXou3qcV+3UZugJd8+Em4kXWX2nCxNfy1vTjnyHF7AJuXsJIblCQMZ+O/+St2FgTOzhw6Vbq0djSHqTa6Kp9DP1fzjEYXCOIPcsA9uLwGus00W96rgQ2mhm2SGkXEZbjwW8BlHIp3OIh4ggw3Rir2J4Srowzf5ocEP2tWPNn5YJvPEUe9Lx5TgGJUs9Me3DBHG1cV9SgFIqgNZvY/ppJVmkCrmkvrfNgI2Roxeg7XKzj3XbaX9Sx7eJXOqpJ4NKTjU/OVdcrq7QU4gHKaE5F6L9ZY+0GywbF57OBg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WH9mV/4G+pvjgm69H/sukJ+57rWHS6bSAKI6Wq2WwqI=; b=fLD2k7OtJ+7J+/1Y8V8ZjfYE7uM8dYtaUOLaBl2Rlft+ypuXMUmyEOOtQo5JxP5iyKhTGsVzvR9deVh11THQG7QfxNy3HN/gRz8mfwwN9g6AmMTQB70n57lI5z8p+zVhkmsI0lveMwt+hZCu9bl0WX4sUlMm/GY+sMlZoQqv4bI= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from SN6PR12MB4623.namprd12.prod.outlook.com (2603:10b6:805:e9::17) by SA0PR12MB4462.namprd12.prod.outlook.com (2603:10b6:806:95::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4108.25; Mon, 10 May 2021 16:36:41 +0000 Received: from SN6PR12MB4623.namprd12.prod.outlook.com ([fe80::ad51:8c49:b171:856c]) by SN6PR12MB4623.namprd12.prod.outlook.com ([fe80::ad51:8c49:b171:856c%7]) with mapi id 15.20.4108.031; Mon, 10 May 2021 16:36:41 +0000 From: Andrey Grodzovsky To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, linux-pci@vger.kernel.org, ckoenig.leichtzumerken@gmail.com, daniel.vetter@ffwll.ch, Harry.Wentland@amd.com Subject: [PATCH v6 00/16] RFC Support hot device unplug in amdgpu Date: Mon, 10 May 2021 12:36:09 -0400 Message-Id: <20210510163625.407105-1-andrey.grodzovsky@amd.com> X-Mailer: git-send-email 2.25.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Originating-IP: [2607:fea8:3edf:49b0:6a5:47b8:e610:f6a3] X-ClientProxiedBy: YT2PR01CA0020.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:38::25) To SN6PR12MB4623.namprd12.prod.outlook.com (2603:10b6:805:e9::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from agrodzovsky-All-Series.hitronhub.home (2607:fea8:3edf:49b0:6a5:47b8:e610:f6a3) by YT2PR01CA0020.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:38::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4108.31 via Frontend Transport; Mon, 10 May 2021 16:36:40 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ee1b6dff-ae9c-49a8-d82f-08d913d1ca14 X-MS-TrafficTypeDiagnostic: SA0PR12MB4462: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: TIQxZkXXTman37N440jTguZDI5anXlTIleZ9e7Ci34SiXFUwgfcZHWnX3eT9Rxpvpgo9swllWMHBRdds6fveymthIt031gnGC0/NaxSve3YEXtHqTLRUlobpNEsxcw86dEq92ndkEm9DYuiw3ddsl4LnNCywmbFd0KiDKVzK62b9CJkRw0mygDazpIwC65X7Vxrzz/mPZEAhu6QsXHi9BJl2PPAJV6OzNjJ2u+lMSous6FhQ/5ZBWhuq7BEdl7QKMaFhyBHGjmFyBCW1NGWCi9pAnWc+hS8jCrCYczYIpHmBwtpYHQdzRRWuraIY7ThrS8h7+wmbwico2kCxuuq3kD6vbnSURm1JZWsoidVCtnRCku2RUK4UiqBIJ8w7JJvGMiE3+JuHXn2DxzjiP/6qNBuUvMA+fIfDoWSW5lRikLvyg5VsqIC2VJTftb+yHdZkSZFWufNx09kOWSFZfwdezqdbnYFxT+Clz4QExNZqPF+VLqhbvyYxJ+8vuN+FFarni31eRrtEIwRMjQHHGQBcuGBUz4u8Y4iX8SeVmO0GCjaLpr45am9Spq2k9uK0Vd37F82/a7k5D8+YvTUfbRDt15YD8EtwhnJl4PPjU0PsyH0hpY5eGES847FeoDA7/BNedA6fFAgik+VupxMqicLdjixl6AcmuclhJLrMdlGp7hk+wtz1LKJVECJxLkx6AFFgXjsYv2WZRi5MlpJatv0pCQ== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SN6PR12MB4623.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(136003)(39860400002)(396003)(376002)(346002)(86362001)(6506007)(38100700002)(52116002)(966005)(6666004)(44832011)(8676002)(36756003)(5660300002)(478600001)(8936002)(2906002)(6486002)(66946007)(2616005)(66476007)(66556008)(1076003)(316002)(6636002)(6512007)(16526019)(186003)(83380400001)(4326008); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?utf-8?B?YkR4NUJlaHBjQVNQMU9hQVFMMFFEUXdFdU5kbVB3eHhJeHZ3MHNtMFVCblRK?= =?utf-8?B?UmIwRUN4UzJ4RENhTk1vWWRNbWREeDhZN2JBU3ZLeEh4Ry83L0ZNeW9ocG5L?= =?utf-8?B?eWcxakUxL3dta2pFODVBbFQzdkU4bDlZYnU0MXhYanY0RW1WenZiK1ZnUmtp?= =?utf-8?B?NFhWcmZnbmdJNUNQa2RzY2wyRysyNVQ3SjE5eW95VS9FRzk4L3hhL01hb1d2?= =?utf-8?B?aEpBbnNXdkF6S3hQajkwUHB0a1Q5bWRNSmlJU3dHRTBwbFN4OFZCWFN1djFS?= =?utf-8?B?YzdOT1lvRnJ4cjI2eFNhUVVJbXZYYklSMCtVSXVudHV4OVJmZ0M2WE1pdEZJ?= =?utf-8?B?aXBhRU5EMndvbDltOEtyanFPQ0k3NXVybVVSRzdlVnJ3R2cvdXhHa1E3dkU4?= =?utf-8?B?YjNEOFNiOTZuK0V2MklKRmQyR1ZJRUI4RWQ2OEw2WTArcG43N2hFaGptUk9m?= =?utf-8?B?WlRTQ0YrYWhBN29scDFzWElPaG5oNndpbXFXNUl3WDNmRzJKUnZ5enZ5ajNB?= =?utf-8?B?Yk80SGYxWkQ5ZTdxcXJNSW1HTjhQczBLYUk0Zm1YTHE4Z1RRK21NaXd0dllD?= =?utf-8?B?YUZvazRpb1FsaGtGeUtPeXdHcVZxdUZkTW9obVY5VkV6ZTBCSWx6QnJNd2Q3?= =?utf-8?B?clVtTnFvSmFDK3dZNzBOZFRUUGZ3TStkQUY5QkQ1OXFIRHJPa1RuYUJ3QWh6?= =?utf-8?B?QWJpRmZSYmRLendYYmlSeE4yNlUwRHpScFVWMDU5RDU1U1pRRVM3OTUrYXJY?= =?utf-8?B?ZTRFZlZDSk9mcHlNbkpvQ2huOER4RStFZVRUOTV0djYzR3VJNWxwNXhFK1N0?= =?utf-8?B?dTByUXJoVUdHeFNoOFBCSkdVWFV5NW1kdlNNbnAybndqVFgzbjVHMnJSaDY0?= =?utf-8?B?a0c0dXl6SlhLcjQ0L2JPL1RGOGdDcE1PR0VBeWx5dU9aM3NBRjVjZDJMNWd4?= =?utf-8?B?S2o1clphTEpFRDdXTnRYdXFVa3pKRjJ2Y2s4cGg1bVN2Y1dyWXpXZVE0OXBx?= =?utf-8?B?NmRqekE0aHhjTWc1MWl3QXNPT0ZoSm1LUmQxUUdDNkFMQTc3L3BOTXFnczRu?= =?utf-8?B?MU03d0Z3UVdHK0JrSlAyYzZ3S1lKVDlHR2g1aVdXdTZPa1E3SGNzWlQ2MXgy?= =?utf-8?B?Ty9XNEpXMmxmWlhnL2YxTmxBZzgzTU5iNWRFN2hIY0xKaDJ1SmhmZlo2MjBx?= =?utf-8?B?SlNvYndnMkZvSWZGOXZMdmtXSHVPYUVSc09XS2NnalVFK0c0ejdlbFB2ZnVD?= =?utf-8?B?UnRUUmtydGRKRE9OdzJmd2hxUWI5aEZRem5XS3FQTW9KZUhiRXhwT2ZWVjhp?= =?utf-8?B?bGNsVjFEUForejVJZE93aUtKTjgxV1EzTmZKZWhNK0JUS2JJNnpUbFdQWFVI?= =?utf-8?B?SkNOYnp4QTgzeW5WcGQyRENzMzJtZmF5NExNNHV3N1VPdHhYV0pLU1dDc2Iw?= =?utf-8?B?NVk5RzZvMmlEdTlZbVNCVkNFNjBHRW9YK1BSU0s3RVVqWjJCZExqMnp2eXlU?= =?utf-8?B?N2lLaUtENGxVN0RrOU4rd1hLZW5tdWY4UFc1QlE3V3BPKzJQL3I2bTE5OTJU?= =?utf-8?B?RWIxbDR3WVlDWVZjYVJjTjFHS3oxOWdWOFF1U1htWE9CWjdLQXplVzJFblEv?= =?utf-8?B?WnR2MDBnd0VSSXF1TWJoTmxqOHdZR3JLNkhScWFidkMzTEtuUWNiaTRaLzJP?= =?utf-8?B?YnhXdm4wVFMyYkZBbm5BOEx1d0RBbEhZSkZHSHkwOWNpeVNCcXJyZDJTempn?= =?utf-8?B?SkJJYTVhaXJzV0FORzZ0NStObmNTVDZEQjNxVGNBaGlYZjE1aEpQOHN3cjdh?= =?utf-8?B?RGQybXU3UDNESVBkR09qYmNpd2gwSm84UloyMmdJb0kzeGJmcVNUOFNYU1NF?= =?utf-8?Q?6FlReWY/dBT+5?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: ee1b6dff-ae9c-49a8-d82f-08d913d1ca14 X-MS-Exchange-CrossTenant-AuthSource: SN6PR12MB4623.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 May 2021 16:36:41.0761 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: gg2/cqAcVOkyV5by/Vh5xQyiFz1Po2KRvwU7imuk5QeMgJgAAx1f97+nyO00cH2m5lOTeGnD5ReSTQ3YbIwzeg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB4462 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: gregkh@linuxfoundation.org, Felix.Kuehling@amd.com, helgaas@kernel.org, Alexander.Deucher@amd.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Until now extracting a card either by physical extraction (e.g. eGPU with thunderbolt connection or by emulation through  sysfs -> /sys/bus/pci/devices/device_id/remove) would cause random crashes in user apps. The random crashes in apps were mostly due to the app having mapped a device backed BO into its address space and was still trying to access the BO while the backing device was gone. To answer this first problem Christian suggested fixing the handling of mapped memory in the clients when the device goes away by forcibly unmapping all buffers the user processes have by clearing their respective VMAs mapping the device BOs. Then when the VMAs try to fill in the page tables again we check in the fault handler if the device is removed and if so, return an error. This will generate a SIGBUS to the application which can then cleanly terminate. This indeed was done but this in turn created a problem of kernel OOPs where the OOPSes were due to the fact that while the app was terminating because of the SIGBUS it would trigger use after free in the driver by calling to access device structures that were already released from the pci remove sequence. This was handled by introducing a 'flush' sequence during device removal where we wait for drm file reference to drop to 0 meaning all user clients directly using this device terminated. v2: Based on discussions in the mailing list with Daniel and Pekka [1] and based on the document produced by Pekka from those discussions [2] the whole approach with returning SIGBUS and waiting for all user clients having CPU mapping of device BOs to die was dropped. Instead as per the document suggestion the device structures are kept alive until the last reference to the device is dropped by user client and in the meanwhile all existing and new CPU mappings of the BOs belonging to the device directly or by dma-buf import are rerouted to per user process dummy rw page.Also, I skipped the 'Requirements for KMS UAPI' section of [2] since i am trying to get the minimal set of requirements that still give useful solution to work and this is the'Requirements for Render and Cross-Device UAPI' section and so my test case is removing a secondary device, which is render only and is not involved in KMS. v3: More updates following comments from v2 such as removing loop to find DRM file when rerouting page faults to dummy page,getting rid of unnecessary sysfs handling refactoring and moving prevention of GPU recovery post device unplug from amdgpu to scheduler layer. On top of that added unplug support for the IOMMU enabled system. v4: Drop last sysfs hack and use sysfs default attribute. Guard against write accesses after device removal to avoid modifying released memory. Update dummy pages handling to on demand allocation and release through drm managed framework. Add return value to scheduler job TO handler (by Luben Tuikov) and use this in amdgpu for prevention of GPU recovery post device unplug Also rebase on top of drm-misc-mext instead of amd-staging-drm-next v5: The most significant in this series is the improved protection from kernel driver accessing MMIO ranges that were allocated for the device once the device is gone. To do this, first a patch 'drm/amdgpu: Unmap all MMIO mappings' is introduced. This patch unamps all MMIO mapped into the kernel address space in the form of BARs and kernel BOs with CPU visible VRAM mappings. This way it helped to discover multiple such access points because a page fault would be immediately generated on access. Most of them were solved by moving HW fini code into pci_remove stage (patch drm/amdgpu: Add early fini callback) and for some who were harder to unwind drm_dev_enter/exit scoping was used. In addition all the IOCTLs and all background work and timers are now protected with drm_dev_enter/exit at their root in an attempt that after drm_dev_unplug is finished none of them run anymore and the pci_remove thread is the only thread executing which might touch the HW. To prevent deadlocks in such case against threads stuck on various HW or SW fences patches 'drm/amdgpu: Finalise device fences on device remove' and drm/amdgpu: Add rw_sem to pushing job into sched queue' take care of force signaling all such existing fences and rejecting any newly added ones. v6: Drop using drm_dev_enter/exit in conjunction with signalling HW fences before setting drm_dev_unplug. We need to devise a more robust cros DRM approach to the problem of dma fence waits falling inside drm_dev_enter/exit scopes -> move to TODO. With these patches I am able to gracefully remove the secondary card using sysfs remove hook while glxgears is running off of secondary card (DRI_PRIME=1) without kernel oopses or hangs and keep working with the primary card or soft reset the device without hangs or oopses. Also as per Daniel's comment I added 3 tests to IGT [4] to core_hotunplug test suite - remove device while commands are submitted, exported BO and exported fence (not pushed yet). Also now it's possible to plug back the device after unplug Also some users now can successfully use those patches with eGPU boxes[3]. TODOs for followup work: Convert AMDGPU code to use devm (for hw stuff) and drmm (for sw stuff and allocations) (Daniel) Add support for 'Requirements for KMS UAPI' section of [2] - unplugging primary, display connected card. Annotate drm_dev_enter/exit against dma_fence_waits as first in deciding where to use drm_dev_enter/exit in code for device unplug. [1] - Discussions during v5 of the patchset https://lore.kernel.org/amd-gfx/20210428151207.1212258-1-andrey.grodzovsky@amd.com/ [2] - drm/doc: device hot-unplug for userspace https://www.spinics.net/lists/dri-devel/msg259755.html [3] - Related gitlab ticket https://gitlab.freedesktop.org/drm/amd/-/issues/1081 [4] - Related IGT tests https://gitlab.freedesktop.org/agrodzov/igt-gpu-tools/-/commits/master Andrey Grodzovsky (16): drm/ttm: Remap all page faults to per process dummy page. drm/ttm: Expose ttm_tt_unpopulate for driver use drm/amdgpu: Split amdgpu_device_fini into early and late drm/amdkfd: Split kfd suspend from devie exit drm/amdgpu: Add early fini callback drm/amdgpu: Handle IOMMU enabled case. drm/amdgpu: Remap all page faults to per process dummy page. PCI: Add support for dev_groups to struct pci_device_driver drm/amdgpu: Convert driver sysfs attributes to static attributes drm/amdgpu: Guard against write accesses after device removal drm/sched: Make timeout timer rearm conditional. drm/amdgpu: Prevent any job recoveries after device is unplugged. drm/amdgpu: Fix hang on device removal. drm/scheduler: Fix hang when sched_entity released drm/amd/display: Remove superflous drm_mode_config_cleanup drm/amdgpu: Verify DMA opearations from device are done drivers/gpu/drm/amd/amdgpu/amdgpu.h | 6 +- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_atombios.c | 17 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 98 +++++++++++++------ drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 26 ++++- drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c | 31 ++++-- drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c | 3 +- drivers/gpu/drm/amd/amdgpu/amdgpu_gart.h | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 9 ++ drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.c | 25 ++--- drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c | 17 +++- drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c | 35 +++++-- drivers/gpu/drm/amd/amdgpu/amdgpu_irq.h | 3 +- drivers/gpu/drm/amd/amdgpu/amdgpu_job.c | 19 +++- drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 12 ++- drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c | 63 +++++++----- drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h | 2 + drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c | 70 +++++++++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h | 52 ++-------- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 21 +++- drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c | 31 ++++-- drivers/gpu/drm/amd/amdgpu/amdgpu_vce.c | 11 ++- drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.c | 22 +++-- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 7 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 14 +-- drivers/gpu/drm/amd/amdgpu/cik_ih.c | 3 +- drivers/gpu/drm/amd/amdgpu/cz_ih.c | 3 +- drivers/gpu/drm/amd/amdgpu/iceland_ih.c | 3 +- drivers/gpu/drm/amd/amdgpu/navi10_ih.c | 5 +- drivers/gpu/drm/amd/amdgpu/psp_v11_0.c | 44 ++++----- drivers/gpu/drm/amd/amdgpu/psp_v12_0.c | 8 +- drivers/gpu/drm/amd/amdgpu/psp_v3_1.c | 8 +- drivers/gpu/drm/amd/amdgpu/si_ih.c | 3 +- drivers/gpu/drm/amd/amdgpu/tonga_ih.c | 3 +- drivers/gpu/drm/amd/amdgpu/vce_v4_0.c | 26 +++-- drivers/gpu/drm/amd/amdgpu/vcn_v3_0.c | 22 +++-- drivers/gpu/drm/amd/amdgpu/vega10_ih.c | 5 +- drivers/gpu/drm/amd/amdgpu/vega20_ih.c | 2 +- drivers/gpu/drm/amd/amdkfd/kfd_device.c | 3 +- .../gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 13 ++- drivers/gpu/drm/amd/include/amd_shared.h | 2 + .../drm/amd/pm/powerplay/smumgr/smu7_smumgr.c | 2 + drivers/gpu/drm/scheduler/sched_entity.c | 3 +- drivers/gpu/drm/scheduler/sched_main.c | 35 ++++++- drivers/gpu/drm/ttm/ttm_bo_vm.c | 57 ++++++++++- drivers/gpu/drm/ttm/ttm_tt.c | 1 + drivers/pci/pci-driver.c | 1 + include/drm/ttm/ttm_bo_api.h | 2 + include/linux/pci.h | 3 + 51 files changed, 585 insertions(+), 272 deletions(-) -- 2.25.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A58EC433B4 for ; Mon, 10 May 2021 16:36:49 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2C9CA61107 for ; Mon, 10 May 2021 16:36:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2C9CA61107 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=amd-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 7E5826E500; Mon, 10 May 2021 16:36:44 +0000 (UTC) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2055.outbound.protection.outlook.com [40.107.94.55]) by gabe.freedesktop.org (Postfix) with ESMTPS id AD2926E89A; Mon, 10 May 2021 16:36:42 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RLQ7lNFzHwMmiZdt8qAMYmQJBXCdFNft6y8/EOc1nAhs415oFtRqGXlrhEgL3Cd/TQ9aNOmElceD9yix0/EZ4kphj9MLWZZ+7Ggo12Rx+qLDLKgASpqG7arFqMBQB6RfIii8tC9tDdR6ErqolqgYw8pMcLQpqxGw0eTqe3x07PLxVlyb7O0HVfvYQe/IKaIwpWJlAKeBd/kbZlZO56fTQ6bWT0VLStws41OSqgy9PnIfyJaWjNwx+abgKxNAgi8h4ieRIc+krAmsA98+8rV/1fAwlhHjqHAkSPMMJAOOHndb4hvboZFU1PO7iQLRtnXHXkKFSrV8VGcBsI6lT+XPgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WH9mV/4G+pvjgm69H/sukJ+57rWHS6bSAKI6Wq2WwqI=; b=TRIjNJZXfjXKcKg0XR/Pon6Sne5iP+gglrTAclgwzMJNqTD8HGXou3qcV+3UZugJd8+Em4kXWX2nCxNfy1vTjnyHF7AJuXsJIblCQMZ+O/+St2FgTOzhw6Vbq0djSHqTa6Kp9DP1fzjEYXCOIPcsA9uLwGus00W96rgQ2mhm2SGkXEZbjwW8BlHIp3OIh4ggw3Rir2J4Srowzf5ocEP2tWPNn5YJvPEUe9Lx5TgGJUs9Me3DBHG1cV9SgFIqgNZvY/ppJVmkCrmkvrfNgI2Roxeg7XKzj3XbaX9Sx7eJXOqpJ4NKTjU/OVdcrq7QU4gHKaE5F6L9ZY+0GywbF57OBg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WH9mV/4G+pvjgm69H/sukJ+57rWHS6bSAKI6Wq2WwqI=; b=fLD2k7OtJ+7J+/1Y8V8ZjfYE7uM8dYtaUOLaBl2Rlft+ypuXMUmyEOOtQo5JxP5iyKhTGsVzvR9deVh11THQG7QfxNy3HN/gRz8mfwwN9g6AmMTQB70n57lI5z8p+zVhkmsI0lveMwt+hZCu9bl0WX4sUlMm/GY+sMlZoQqv4bI= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from SN6PR12MB4623.namprd12.prod.outlook.com (2603:10b6:805:e9::17) by SA0PR12MB4462.namprd12.prod.outlook.com (2603:10b6:806:95::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4108.25; Mon, 10 May 2021 16:36:41 +0000 Received: from SN6PR12MB4623.namprd12.prod.outlook.com ([fe80::ad51:8c49:b171:856c]) by SN6PR12MB4623.namprd12.prod.outlook.com ([fe80::ad51:8c49:b171:856c%7]) with mapi id 15.20.4108.031; Mon, 10 May 2021 16:36:41 +0000 From: Andrey Grodzovsky To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, linux-pci@vger.kernel.org, ckoenig.leichtzumerken@gmail.com, daniel.vetter@ffwll.ch, Harry.Wentland@amd.com Subject: [PATCH v6 00/16] RFC Support hot device unplug in amdgpu Date: Mon, 10 May 2021 12:36:09 -0400 Message-Id: <20210510163625.407105-1-andrey.grodzovsky@amd.com> X-Mailer: git-send-email 2.25.1 X-Originating-IP: [2607:fea8:3edf:49b0:6a5:47b8:e610:f6a3] X-ClientProxiedBy: YT2PR01CA0020.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:38::25) To SN6PR12MB4623.namprd12.prod.outlook.com (2603:10b6:805:e9::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from agrodzovsky-All-Series.hitronhub.home (2607:fea8:3edf:49b0:6a5:47b8:e610:f6a3) by YT2PR01CA0020.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:38::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4108.31 via Frontend Transport; Mon, 10 May 2021 16:36:40 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ee1b6dff-ae9c-49a8-d82f-08d913d1ca14 X-MS-TrafficTypeDiagnostic: SA0PR12MB4462: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: TIQxZkXXTman37N440jTguZDI5anXlTIleZ9e7Ci34SiXFUwgfcZHWnX3eT9Rxpvpgo9swllWMHBRdds6fveymthIt031gnGC0/NaxSve3YEXtHqTLRUlobpNEsxcw86dEq92ndkEm9DYuiw3ddsl4LnNCywmbFd0KiDKVzK62b9CJkRw0mygDazpIwC65X7Vxrzz/mPZEAhu6QsXHi9BJl2PPAJV6OzNjJ2u+lMSous6FhQ/5ZBWhuq7BEdl7QKMaFhyBHGjmFyBCW1NGWCi9pAnWc+hS8jCrCYczYIpHmBwtpYHQdzRRWuraIY7ThrS8h7+wmbwico2kCxuuq3kD6vbnSURm1JZWsoidVCtnRCku2RUK4UiqBIJ8w7JJvGMiE3+JuHXn2DxzjiP/6qNBuUvMA+fIfDoWSW5lRikLvyg5VsqIC2VJTftb+yHdZkSZFWufNx09kOWSFZfwdezqdbnYFxT+Clz4QExNZqPF+VLqhbvyYxJ+8vuN+FFarni31eRrtEIwRMjQHHGQBcuGBUz4u8Y4iX8SeVmO0GCjaLpr45am9Spq2k9uK0Vd37F82/a7k5D8+YvTUfbRDt15YD8EtwhnJl4PPjU0PsyH0hpY5eGES847FeoDA7/BNedA6fFAgik+VupxMqicLdjixl6AcmuclhJLrMdlGp7hk+wtz1LKJVECJxLkx6AFFgXjsYv2WZRi5MlpJatv0pCQ== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SN6PR12MB4623.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(136003)(39860400002)(396003)(376002)(346002)(86362001)(6506007)(38100700002)(52116002)(966005)(6666004)(44832011)(8676002)(36756003)(5660300002)(478600001)(8936002)(2906002)(6486002)(66946007)(2616005)(66476007)(66556008)(1076003)(316002)(6636002)(6512007)(16526019)(186003)(83380400001)(4326008); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?utf-8?B?YkR4NUJlaHBjQVNQMU9hQVFMMFFEUXdFdU5kbVB3eHhJeHZ3MHNtMFVCblRK?= =?utf-8?B?UmIwRUN4UzJ4RENhTk1vWWRNbWREeDhZN2JBU3ZLeEh4Ry83L0ZNeW9ocG5L?= =?utf-8?B?eWcxakUxL3dta2pFODVBbFQzdkU4bDlZYnU0MXhYanY0RW1WenZiK1ZnUmtp?= =?utf-8?B?NFhWcmZnbmdJNUNQa2RzY2wyRysyNVQ3SjE5eW95VS9FRzk4L3hhL01hb1d2?= =?utf-8?B?aEpBbnNXdkF6S3hQajkwUHB0a1Q5bWRNSmlJU3dHRTBwbFN4OFZCWFN1djFS?= =?utf-8?B?YzdOT1lvRnJ4cjI2eFNhUVVJbXZYYklSMCtVSXVudHV4OVJmZ0M2WE1pdEZJ?= =?utf-8?B?aXBhRU5EMndvbDltOEtyanFPQ0k3NXVybVVSRzdlVnJ3R2cvdXhHa1E3dkU4?= =?utf-8?B?YjNEOFNiOTZuK0V2MklKRmQyR1ZJRUI4RWQ2OEw2WTArcG43N2hFaGptUk9m?= =?utf-8?B?WlRTQ0YrYWhBN29scDFzWElPaG5oNndpbXFXNUl3WDNmRzJKUnZ5enZ5ajNB?= =?utf-8?B?Yk80SGYxWkQ5ZTdxcXJNSW1HTjhQczBLYUk0Zm1YTHE4Z1RRK21NaXd0dllD?= =?utf-8?B?YUZvazRpb1FsaGtGeUtPeXdHcVZxdUZkTW9obVY5VkV6ZTBCSWx6QnJNd2Q3?= =?utf-8?B?clVtTnFvSmFDK3dZNzBOZFRUUGZ3TStkQUY5QkQ1OXFIRHJPa1RuYUJ3QWh6?= =?utf-8?B?QWJpRmZSYmRLendYYmlSeE4yNlUwRHpScFVWMDU5RDU1U1pRRVM3OTUrYXJY?= =?utf-8?B?ZTRFZlZDSk9mcHlNbkpvQ2huOER4RStFZVRUOTV0djYzR3VJNWxwNXhFK1N0?= =?utf-8?B?dTByUXJoVUdHeFNoOFBCSkdVWFV5NW1kdlNNbnAybndqVFgzbjVHMnJSaDY0?= =?utf-8?B?a0c0dXl6SlhLcjQ0L2JPL1RGOGdDcE1PR0VBeWx5dU9aM3NBRjVjZDJMNWd4?= =?utf-8?B?S2o1clphTEpFRDdXTnRYdXFVa3pKRjJ2Y2s4cGg1bVN2Y1dyWXpXZVE0OXBx?= =?utf-8?B?NmRqekE0aHhjTWc1MWl3QXNPT0ZoSm1LUmQxUUdDNkFMQTc3L3BOTXFnczRu?= =?utf-8?B?MU03d0Z3UVdHK0JrSlAyYzZ3S1lKVDlHR2g1aVdXdTZPa1E3SGNzWlQ2MXgy?= =?utf-8?B?Ty9XNEpXMmxmWlhnL2YxTmxBZzgzTU5iNWRFN2hIY0xKaDJ1SmhmZlo2MjBx?= =?utf-8?B?SlNvYndnMkZvSWZGOXZMdmtXSHVPYUVSc09XS2NnalVFK0c0ejdlbFB2ZnVD?= =?utf-8?B?UnRUUmtydGRKRE9OdzJmd2hxUWI5aEZRem5XS3FQTW9KZUhiRXhwT2ZWVjhp?= =?utf-8?B?bGNsVjFEUForejVJZE93aUtKTjgxV1EzTmZKZWhNK0JUS2JJNnpUbFdQWFVI?= =?utf-8?B?SkNOYnp4QTgzeW5WcGQyRENzMzJtZmF5NExNNHV3N1VPdHhYV0pLU1dDc2Iw?= =?utf-8?B?NVk5RzZvMmlEdTlZbVNCVkNFNjBHRW9YK1BSU0s3RVVqWjJCZExqMnp2eXlU?= =?utf-8?B?N2lLaUtENGxVN0RrOU4rd1hLZW5tdWY4UFc1QlE3V3BPKzJQL3I2bTE5OTJU?= =?utf-8?B?RWIxbDR3WVlDWVZjYVJjTjFHS3oxOWdWOFF1U1htWE9CWjdLQXplVzJFblEv?= =?utf-8?B?WnR2MDBnd0VSSXF1TWJoTmxqOHdZR3JLNkhScWFidkMzTEtuUWNiaTRaLzJP?= =?utf-8?B?YnhXdm4wVFMyYkZBbm5BOEx1d0RBbEhZSkZHSHkwOWNpeVNCcXJyZDJTempn?= =?utf-8?B?SkJJYTVhaXJzV0FORzZ0NStObmNTVDZEQjNxVGNBaGlYZjE1aEpQOHN3cjdh?= =?utf-8?B?RGQybXU3UDNESVBkR09qYmNpd2gwSm84UloyMmdJb0kzeGJmcVNUOFNYU1NF?= =?utf-8?Q?6FlReWY/dBT+5?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: ee1b6dff-ae9c-49a8-d82f-08d913d1ca14 X-MS-Exchange-CrossTenant-AuthSource: SN6PR12MB4623.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 May 2021 16:36:41.0761 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: gg2/cqAcVOkyV5by/Vh5xQyiFz1Po2KRvwU7imuk5QeMgJgAAx1f97+nyO00cH2m5lOTeGnD5ReSTQ3YbIwzeg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB4462 X-BeenThere: amd-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion list for AMD gfx List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Andrey Grodzovsky , gregkh@linuxfoundation.org, Felix.Kuehling@amd.com, ppaalanen@gmail.com, helgaas@kernel.org, Alexander.Deucher@amd.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: amd-gfx-bounces@lists.freedesktop.org Sender: "amd-gfx" VW50aWwgbm93IGV4dHJhY3RpbmcgYSBjYXJkIGVpdGhlciBieSBwaHlzaWNhbCBleHRyYWN0aW9u IChlLmcuIGVHUFUgd2l0aCAKdGh1bmRlcmJvbHQgY29ubmVjdGlvbiBvciBieSBlbXVsYXRpb24g dGhyb3VnaCDCoHN5c2ZzIC0+IC9zeXMvYnVzL3BjaS9kZXZpY2VzL2RldmljZV9pZC9yZW1vdmUp IAp3b3VsZCBjYXVzZSByYW5kb20gY3Jhc2hlcyBpbiB1c2VyIGFwcHMuIFRoZSByYW5kb20gY3Jh c2hlcyBpbiBhcHBzIHdlcmUgCm1vc3RseSBkdWUgdG8gdGhlIGFwcCBoYXZpbmcgbWFwcGVkIGEg ZGV2aWNlIGJhY2tlZCBCTyBpbnRvIGl0cyBhZGRyZXNzIApzcGFjZSBhbmQgd2FzIHN0aWxsIHRy eWluZyB0byBhY2Nlc3MgdGhlIEJPIHdoaWxlIHRoZSBiYWNraW5nIGRldmljZSB3YXMgZ29uZS4K VG8gYW5zd2VyIHRoaXMgZmlyc3QgcHJvYmxlbSBDaHJpc3RpYW4gc3VnZ2VzdGVkIGZpeGluZyB0 aGUgaGFuZGxpbmcgb2YgbWFwcGVkIAptZW1vcnkgaW4gdGhlIGNsaWVudHMgd2hlbiB0aGUgZGV2 aWNlIGdvZXMgYXdheSBieSBmb3JjaWJseSB1bm1hcHBpbmcgYWxsIGJ1ZmZlcnMgdGhlIAp1c2Vy wqBwcm9jZXNzZXMgaGF2ZcKgYnkgY2xlYXJpbmcgdGhlaXIgcmVzcGVjdGl2ZSBWTUFzIG1hcHBp bmcgdGhlIGRldmljZSBCT3MuClRoZW4gd2hlbiB0aGUgVk1BcyB0cnkgdG8gZmlsbCBpbiB0aGUg cGFnZSB0YWJsZXMgYWdhaW4gd2UgY2hlY2sgaW4gdGhlIGZhdWx0IApoYW5kbGVyIGlmIHRoZSBk ZXZpY2UgaXMgcmVtb3ZlZCBhbmQgaWYgc28sIHJldHVybiBhbiBlcnJvci4gVGhpcyB3aWxsIGdl bmVyYXRlIGEgClNJR0JVUyB0byB0aGUgYXBwbGljYXRpb24gd2hpY2ggY2FuIHRoZW4gY2xlYW5s eSB0ZXJtaW5hdGUuIFRoaXMgaW5kZWVkIHdhcyBkb25lIApidXQgdGhpcyBpbiB0dXJuIGNyZWF0 ZWQgYSBwcm9ibGVtIG9mIGtlcm5lbCBPT1BzIHdoZXJlIHRoZSBPT1BTZXMgd2VyZSBkdWUgdG8g dGhlIApmYWN0IHRoYXQgd2hpbGUgdGhlIGFwcCB3YXMgdGVybWluYXRpbmcgYmVjYXVzZSBvZiB0 aGUgU0lHQlVTIGl0IHdvdWxkIHRyaWdnZXIgdXNlIAphZnRlciBmcmVlIGluIHRoZSBkcml2ZXIg YnkgY2FsbGluZyB0b8KgYWNjZXNzwqBkZXZpY2Ugc3RydWN0dXJlcyB0aGF0IHdlcmUgYWxyZWFk eQpyZWxlYXNlZCBmcm9tIHRoZSBwY2kgcmVtb3ZlIHNlcXVlbmNlLiBUaGlzIHdhcyBoYW5kbGVk IGJ5IGludHJvZHVjaW5nIGEgJ2ZsdXNoJyAKc2VxdWVuY2UgZHVyaW5nIGRldmljZSByZW1vdmFs IHdoZXJlIHdlIHdhaXQgZm9yIGRybSBmaWxlIHJlZmVyZW5jZSB0byBkcm9wIHRvIDAgCm1lYW5p bmcgYWxsIHVzZXIgY2xpZW50cyBkaXJlY3RseSB1c2luZyB0aGlzIGRldmljZSB0ZXJtaW5hdGVk LgoKdjI6CkJhc2VkIG9uIGRpc2N1c3Npb25zIGluIHRoZSBtYWlsaW5nIGxpc3Qgd2l0aCBEYW5p ZWwgYW5kIFBla2thIFsxXSBhbmQgYmFzZWQgb24gdGhlIGRvY3VtZW50IApwcm9kdWNlZCBieSBQ ZWtrYSBmcm9tIHRob3NlIGRpc2N1c3Npb25zIFsyXSB0aGUgd2hvbGUgYXBwcm9hY2ggd2l0aCBy ZXR1cm5pbmcgU0lHQlVTIGFuZCAKd2FpdGluZyBmb3IgYWxsIHVzZXIgY2xpZW50cyBoYXZpbmcg Q1BVIG1hcHBpbmcgb2YgZGV2aWNlIEJPcyB0byBkaWUgd2FzIGRyb3BwZWQuIApJbnN0ZWFkIGFz IHBlciB0aGUgZG9jdW1lbnQgc3VnZ2VzdGlvbiB0aGUgZGV2aWNlIHN0cnVjdHVyZXMgYXJlIGtl cHQgYWxpdmUgdW50aWwgCnRoZSBsYXN0IHJlZmVyZW5jZSB0byB0aGUgZGV2aWNlIGlzIGRyb3Bw ZWQgYnkgdXNlciBjbGllbnQgYW5kIGluIHRoZSBtZWFud2hpbGUgYWxsIGV4aXN0aW5nIGFuZCBu ZXcgQ1BVIG1hcHBpbmdzIG9mIHRoZSBCT3MgCmJlbG9uZ2luZyB0byB0aGUgZGV2aWNlIGRpcmVj dGx5IG9yIGJ5IGRtYS1idWYgaW1wb3J0IGFyZSByZXJvdXRlZCB0byBwZXIgdXNlciAKcHJvY2Vz cyBkdW1teSBydyBwYWdlLkFsc28sIEkgc2tpcHBlZCB0aGUgJ1JlcXVpcmVtZW50cyBmb3IgS01T IFVBUEknIHNlY3Rpb24gb2YgWzJdIApzaW5jZSBpIGFtIHRyeWluZyB0byBnZXQgdGhlIG1pbmlt YWwgc2V0IG9mIHJlcXVpcmVtZW50cyB0aGF0IHN0aWxsIGdpdmUgdXNlZnVsIHNvbHV0aW9uIAp0 byB3b3JrIGFuZCB0aGlzIGlzIHRoZSdSZXF1aXJlbWVudHMgZm9yIFJlbmRlciBhbmQgQ3Jvc3Mt RGV2aWNlIFVBUEknIHNlY3Rpb24gYW5kIHNvIG15IAp0ZXN0IGNhc2UgaXMgcmVtb3ZpbmcgYSBz ZWNvbmRhcnkgZGV2aWNlLCB3aGljaCBpcyByZW5kZXIgb25seSBhbmQgaXMgbm90IGludm9sdmVk IAppbiBLTVMuCgp2MzoKTW9yZSB1cGRhdGVzIGZvbGxvd2luZyBjb21tZW50cyBmcm9tIHYyIHN1 Y2ggYXMgcmVtb3ZpbmcgbG9vcCB0byBmaW5kIERSTSBmaWxlIHdoZW4gcmVyb3V0aW5nIApwYWdl IGZhdWx0cyB0byBkdW1teSBwYWdlLGdldHRpbmcgcmlkIG9mIHVubmVjZXNzYXJ5IHN5c2ZzIGhh bmRsaW5nIHJlZmFjdG9yaW5nIGFuZCBtb3ZpbmcgCnByZXZlbnRpb24gb2YgR1BVIHJlY292ZXJ5 IHBvc3QgZGV2aWNlIHVucGx1ZyBmcm9tIGFtZGdwdSB0byBzY2hlZHVsZXIgbGF5ZXIuIApPbiB0 b3Agb2YgdGhhdCBhZGRlZCB1bnBsdWcgc3VwcG9ydCBmb3IgdGhlIElPTU1VIGVuYWJsZWQgc3lz dGVtLgoKdjQ6CkRyb3AgbGFzdCBzeXNmcyBoYWNrIGFuZCB1c2Ugc3lzZnMgZGVmYXVsdCBhdHRy aWJ1dGUuCkd1YXJkIGFnYWluc3Qgd3JpdGUgYWNjZXNzZXMgYWZ0ZXIgZGV2aWNlIHJlbW92YWwg dG8gYXZvaWQgbW9kaWZ5aW5nIHJlbGVhc2VkIG1lbW9yeS4KVXBkYXRlIGR1bW15IHBhZ2VzIGhh bmRsaW5nIHRvIG9uIGRlbWFuZCBhbGxvY2F0aW9uIGFuZCByZWxlYXNlIHRocm91Z2ggZHJtIG1h bmFnZWQgZnJhbWV3b3JrLgpBZGQgcmV0dXJuIHZhbHVlIHRvIHNjaGVkdWxlciBqb2IgVE8gaGFu ZGxlciAoYnkgTHViZW4gVHVpa292KSBhbmQgdXNlIHRoaXMgaW4gYW1kZ3B1IGZvciBwcmV2ZW50 aW9uIApvZiBHUFUgcmVjb3ZlcnkgcG9zdCBkZXZpY2UgdW5wbHVnCkFsc28gcmViYXNlIG9uIHRv cCBvZiBkcm0tbWlzYy1tZXh0IGluc3RlYWQgb2YgYW1kLXN0YWdpbmctZHJtLW5leHQKCnY1OgpU aGUgbW9zdCBzaWduaWZpY2FudCBpbiB0aGlzIHNlcmllcyBpcyB0aGUgaW1wcm92ZWQgcHJvdGVj dGlvbiBmcm9tIGtlcm5lbCBkcml2ZXIgYWNjZXNzaW5nIE1NSU8gcmFuZ2VzIHRoYXQgd2VyZSBh bGxvY2F0ZWQKZm9yIHRoZSBkZXZpY2Ugb25jZSB0aGUgZGV2aWNlIGlzIGdvbmUuIFRvIGRvIHRo aXMsIGZpcnN0IGEgcGF0Y2ggJ2RybS9hbWRncHU6IFVubWFwIGFsbCBNTUlPIG1hcHBpbmdzJyBp cyBpbnRyb2R1Y2VkLgpUaGlzIHBhdGNoIHVuYW1wcyBhbGwgTU1JTyBtYXBwZWQgaW50byB0aGUg a2VybmVsIGFkZHJlc3Mgc3BhY2UgaW4gdGhlIGZvcm0gb2YgQkFScyBhbmQga2VybmVsIEJPcyB3 aXRoIENQVSB2aXNpYmxlIFZSQU0gbWFwcGluZ3MuClRoaXMgd2F5IGl0IGhlbHBlZCB0byBkaXNj b3ZlciBtdWx0aXBsZSBzdWNoIGFjY2VzcyBwb2ludHMgYmVjYXVzZSBhIHBhZ2UgZmF1bHQgd291 bGQgYmUgaW1tZWRpYXRlbHkgZ2VuZXJhdGVkIG9uIGFjY2Vzcy4gTW9zdCBvZiB0aGVtCndlcmUg c29sdmVkIGJ5IG1vdmluZyBIVyBmaW5pIGNvZGUgaW50byBwY2lfcmVtb3ZlIHN0YWdlIChwYXRj aCBkcm0vYW1kZ3B1OiBBZGQgZWFybHkgZmluaSBjYWxsYmFjaykgYW5kIGZvciBzb21lIHdobyAK d2VyZSBoYXJkZXIgdG8gdW53aW5kIGRybV9kZXZfZW50ZXIvZXhpdCBzY29waW5nIHdhcyB1c2Vk LiBJbiBhZGRpdGlvbiBhbGwgdGhlIElPQ1RMcyBhbmQgYWxsIGJhY2tncm91bmQgd29yayBhbmQg dGltZXJzIAphcmUgbm93IHByb3RlY3RlZCB3aXRoIGRybV9kZXZfZW50ZXIvZXhpdCBhdCB0aGVp ciByb290IGluIGFuIGF0dGVtcHQgdGhhdCBhZnRlciBkcm1fZGV2X3VucGx1ZyBpcyBmaW5pc2hl ZCBub25lIG9mIHRoZW0gCnJ1biBhbnltb3JlIGFuZCB0aGUgcGNpX3JlbW92ZSB0aHJlYWQgaXMg dGhlIG9ubHkgdGhyZWFkIGV4ZWN1dGluZyB3aGljaCBtaWdodCB0b3VjaCB0aGUgSFcuIFRvIHBy ZXZlbnQgZGVhZGxvY2tzIGluIHN1Y2ggCmNhc2UgYWdhaW5zdCB0aHJlYWRzIHN0dWNrIG9uIHZh cmlvdXMgSFcgb3IgU1cgZmVuY2VzIHBhdGNoZXMgJ2RybS9hbWRncHU6IEZpbmFsaXNlIGRldmlj ZSBmZW5jZXMgb24gZGV2aWNlIHJlbW92ZScgIAphbmQgZHJtL2FtZGdwdTogQWRkIHJ3X3NlbSB0 byBwdXNoaW5nIGpvYiBpbnRvIHNjaGVkIHF1ZXVlJyB0YWtlIGNhcmUgb2YgZm9yY2Ugc2lnbmFs aW5nIGFsbCBzdWNoIGV4aXN0aW5nIGZlbmNlcyAKYW5kIHJlamVjdGluZyBhbnkgbmV3bHkgYWRk ZWQgb25lcy4KCnY2OgpEcm9wIHVzaW5nIGRybV9kZXZfZW50ZXIvZXhpdCBpbiBjb25qdW5jdGlv biB3aXRoIHNpZ25hbGxpbmcgSFcgZmVuY2VzIGJlZm9yZSBzZXR0aW5nIGRybV9kZXZfdW5wbHVn LgpXZSBuZWVkIHRvIGRldmlzZSBhIG1vcmUgcm9idXN0IGNyb3MgRFJNIGFwcHJvYWNoIHRvIHRo ZSBwcm9ibGVtIG9mIGRtYSBmZW5jZSB3YWl0cyBmYWxsaW5nCmluc2lkZSBkcm1fZGV2X2VudGVy L2V4aXQgc2NvcGVzIC0+IG1vdmUgdG8gVE9ETy4KCldpdGggdGhlc2UgcGF0Y2hlcyBJIGFtIGFi bGUgdG8gZ3JhY2VmdWxseSByZW1vdmUgdGhlIHNlY29uZGFyeSBjYXJkIHVzaW5nIHN5c2ZzIHJl bW92ZSBob29rIHdoaWxlIGdseGdlYXJzIGlzIHJ1bm5pbmcgb2ZmIG9mIHNlY29uZGFyeSAKY2Fy ZCAoRFJJX1BSSU1FPTEpIHdpdGhvdXQga2VybmVsIG9vcHNlcyBvciBoYW5ncyBhbmQga2VlcCB3 b3JraW5nIHdpdGggdGhlIHByaW1hcnkgY2FyZCBvciBzb2Z0IHJlc2V0IHRoZSBkZXZpY2Ugd2l0 aG91dCBoYW5ncyBvciBvb3BzZXMuCkFsc28gYXMgcGVyIERhbmllbCdzIGNvbW1lbnQgSSBhZGRl ZCAzIHRlc3RzIHRvIElHVCBbNF0gdG8gY29yZV9ob3R1bnBsdWcgdGVzdCBzdWl0ZSAtIHJlbW92 ZSBkZXZpY2Ugd2hpbGUgY29tbWFuZHMgYXJlIHN1Ym1pdHRlZCwgCmV4cG9ydGVkIEJPIGFuZCBl eHBvcnRlZCBmZW5jZSAobm90IHB1c2hlZCB5ZXQpLgpBbHNvIG5vdyBpdCdzIHBvc3NpYmxlIHRv IHBsdWcgYmFjayB0aGUgZGV2aWNlIGFmdGVyIHVucGx1ZyAKQWxzbyBzb21lIHVzZXJzIG5vdyBj YW4gc3VjY2Vzc2Z1bGx5IHVzZSB0aG9zZSBwYXRjaGVzIHdpdGggZUdQVSBib3hlc1szXS4KClRP RE9zIGZvciBmb2xsb3d1cCB3b3JrOgpDb252ZXJ0IEFNREdQVSBjb2RlIHRvIHVzZSBkZXZtIChm b3IgaHcgc3R1ZmYpIGFuZCBkcm1tIChmb3Igc3cgc3R1ZmYgYW5kIGFsbG9jYXRpb25zKSAoRGFu aWVsKQpBZGQgc3VwcG9ydCBmb3IgJ1JlcXVpcmVtZW50cyBmb3IgS01TIFVBUEknIHNlY3Rpb24g b2YgWzJdIC0gdW5wbHVnZ2luZyBwcmltYXJ5LCBkaXNwbGF5IGNvbm5lY3RlZCBjYXJkLgpBbm5v dGF0ZSBkcm1fZGV2X2VudGVyL2V4aXQgYWdhaW5zdCBkbWFfZmVuY2Vfd2FpdHMgYXMgZmlyc3Qg aW4gZGVjaWRpbmcgd2hlcmUgdG8gdXNlIGRybV9kZXZfZW50ZXIvZXhpdAppbiBjb2RlIGZvciBk ZXZpY2UgdW5wbHVnLgoKWzFdIC0gRGlzY3Vzc2lvbnMgZHVyaW5nIHY1IG9mIHRoZSBwYXRjaHNl dCBodHRwczovL2xvcmUua2VybmVsLm9yZy9hbWQtZ2Z4LzIwMjEwNDI4MTUxMjA3LjEyMTIyNTgt MS1hbmRyZXkuZ3JvZHpvdnNreUBhbWQuY29tLwpbMl0gLSBkcm0vZG9jOiBkZXZpY2UgaG90LXVu cGx1ZyBmb3IgdXNlcnNwYWNlIGh0dHBzOi8vd3d3LnNwaW5pY3MubmV0L2xpc3RzL2RyaS1kZXZl bC9tc2cyNTk3NTUuaHRtbApbM10gLSBSZWxhdGVkIGdpdGxhYiB0aWNrZXQgaHR0cHM6Ly9naXRs YWIuZnJlZWRlc2t0b3Aub3JnL2RybS9hbWQvLS9pc3N1ZXMvMTA4MQpbNF0gLSBSZWxhdGVkIElH VCB0ZXN0cyBodHRwczovL2dpdGxhYi5mcmVlZGVza3RvcC5vcmcvYWdyb2R6b3YvaWd0LWdwdS10 b29scy8tL2NvbW1pdHMvbWFzdGVyCgpBbmRyZXkgR3JvZHpvdnNreSAoMTYpOgogIGRybS90dG06 IFJlbWFwIGFsbCBwYWdlIGZhdWx0cyB0byBwZXIgcHJvY2VzcyBkdW1teSBwYWdlLgogIGRybS90 dG06IEV4cG9zZSB0dG1fdHRfdW5wb3B1bGF0ZSBmb3IgZHJpdmVyIHVzZQogIGRybS9hbWRncHU6 IFNwbGl0IGFtZGdwdV9kZXZpY2VfZmluaSBpbnRvIGVhcmx5IGFuZCBsYXRlCiAgZHJtL2FtZGtm ZDogU3BsaXQga2ZkIHN1c3BlbmQgZnJvbSBkZXZpZSBleGl0CiAgZHJtL2FtZGdwdTogQWRkIGVh cmx5IGZpbmkgY2FsbGJhY2sKICBkcm0vYW1kZ3B1OiBIYW5kbGUgSU9NTVUgZW5hYmxlZCBjYXNl LgogIGRybS9hbWRncHU6IFJlbWFwIGFsbCBwYWdlIGZhdWx0cyB0byBwZXIgcHJvY2VzcyBkdW1t eSBwYWdlLgogIFBDSTogQWRkIHN1cHBvcnQgZm9yIGRldl9ncm91cHMgdG8gc3RydWN0IHBjaV9k ZXZpY2VfZHJpdmVyCiAgZHJtL2FtZGdwdTogQ29udmVydCBkcml2ZXIgc3lzZnMgYXR0cmlidXRl cyB0byBzdGF0aWMgYXR0cmlidXRlcwogIGRybS9hbWRncHU6IEd1YXJkIGFnYWluc3Qgd3JpdGUg YWNjZXNzZXMgYWZ0ZXIgZGV2aWNlIHJlbW92YWwKICBkcm0vc2NoZWQ6IE1ha2UgdGltZW91dCB0 aW1lciByZWFybSBjb25kaXRpb25hbC4KICBkcm0vYW1kZ3B1OiBQcmV2ZW50IGFueSBqb2IgcmVj b3ZlcmllcyBhZnRlciBkZXZpY2UgaXMgdW5wbHVnZ2VkLgogIGRybS9hbWRncHU6IEZpeCBoYW5n IG9uIGRldmljZSByZW1vdmFsLgogIGRybS9zY2hlZHVsZXI6IEZpeCBoYW5nIHdoZW4gc2NoZWRf ZW50aXR5IHJlbGVhc2VkCiAgZHJtL2FtZC9kaXNwbGF5OiBSZW1vdmUgc3VwZXJmbG91cyBkcm1f bW9kZV9jb25maWdfY2xlYW51cAogIGRybS9hbWRncHU6IFZlcmlmeSBETUEgb3BlYXJhdGlvbnMg ZnJvbSBkZXZpY2UgYXJlIGRvbmUKCiBkcml2ZXJzL2dwdS9kcm0vYW1kL2FtZGdwdS9hbWRncHUu aCAgICAgICAgICAgfCAgNiArLQogZHJpdmVycy9ncHUvZHJtL2FtZC9hbWRncHUvYW1kZ3B1X2Ft ZGtmZC5jICAgIHwgIDIgKy0KIGRyaXZlcnMvZ3B1L2RybS9hbWQvYW1kZ3B1L2FtZGdwdV9hbWRr ZmQuaCAgICB8ICAyICstCiBkcml2ZXJzL2dwdS9kcm0vYW1kL2FtZGdwdS9hbWRncHVfYXRvbWJp b3MuYyAgfCAxNyArKy0tCiBkcml2ZXJzL2dwdS9kcm0vYW1kL2FtZGdwdS9hbWRncHVfZGV2aWNl LmMgICAgfCA5OCArKysrKysrKysrKysrLS0tLS0tCiBkcml2ZXJzL2dwdS9kcm0vYW1kL2FtZGdw dS9hbWRncHVfZHJ2LmMgICAgICAgfCAyNiArKysrLQogZHJpdmVycy9ncHUvZHJtL2FtZC9hbWRn cHUvYW1kZ3B1X2ZlbmNlLmMgICAgIHwgMzEgKysrKy0tCiBkcml2ZXJzL2dwdS9kcm0vYW1kL2Ft ZGdwdS9hbWRncHVfZ2FydC5jICAgICAgfCAgMyArLQogZHJpdmVycy9ncHUvZHJtL2FtZC9hbWRn cHUvYW1kZ3B1X2dhcnQuaCAgICAgIHwgIDEgKwogZHJpdmVycy9ncHUvZHJtL2FtZC9hbWRncHUv YW1kZ3B1X2dtYy5jICAgICAgIHwgIDkgKysKIGRyaXZlcnMvZ3B1L2RybS9hbWQvYW1kZ3B1L2Ft ZGdwdV9ndHRfbWdyLmMgICB8IDI1ICsrLS0tCiBkcml2ZXJzL2dwdS9kcm0vYW1kL2FtZGdwdS9h bWRncHVfaWIuYyAgICAgICAgfCAxNyArKystCiBkcml2ZXJzL2dwdS9kcm0vYW1kL2FtZGdwdS9h bWRncHVfaXJxLmMgICAgICAgfCAzNSArKysrKy0tCiBkcml2ZXJzL2dwdS9kcm0vYW1kL2FtZGdw dS9hbWRncHVfaXJxLmggICAgICAgfCAgMyArLQogZHJpdmVycy9ncHUvZHJtL2FtZC9hbWRncHUv YW1kZ3B1X2pvYi5jICAgICAgIHwgMTkgKysrLQogZHJpdmVycy9ncHUvZHJtL2FtZC9hbWRncHUv YW1kZ3B1X2ttcy5jICAgICAgIHwgMTIgKystCiBkcml2ZXJzL2dwdS9kcm0vYW1kL2FtZGdwdS9h bWRncHVfcHNwLmMgICAgICAgfCA2MyArKysrKysrLS0tLS0KIGRyaXZlcnMvZ3B1L2RybS9hbWQv YW1kZ3B1L2FtZGdwdV9wc3AuaCAgICAgICB8ICAyICsKIGRyaXZlcnMvZ3B1L2RybS9hbWQvYW1k Z3B1L2FtZGdwdV9yYXMuYyAgICAgICB8ICAxICsKIGRyaXZlcnMvZ3B1L2RybS9hbWQvYW1kZ3B1 L2FtZGdwdV9yaW5nLmMgICAgICB8IDcwICsrKysrKysrKysrKysKIGRyaXZlcnMvZ3B1L2RybS9h bWQvYW1kZ3B1L2FtZGdwdV9yaW5nLmggICAgICB8IDUyICsrLS0tLS0tLS0KIGRyaXZlcnMvZ3B1 L2RybS9hbWQvYW1kZ3B1L2FtZGdwdV90dG0uYyAgICAgICB8IDIxICsrKy0KIGRyaXZlcnMvZ3B1 L2RybS9hbWQvYW1kZ3B1L2FtZGdwdV91dmQuYyAgICAgICB8IDMxICsrKystLQogZHJpdmVycy9n cHUvZHJtL2FtZC9hbWRncHUvYW1kZ3B1X3ZjZS5jICAgICAgIHwgMTEgKystCiBkcml2ZXJzL2dw dS9kcm0vYW1kL2FtZGdwdS9hbWRncHVfdmNuLmMgICAgICAgfCAyMiArKystLQogZHJpdmVycy9n cHUvZHJtL2FtZC9hbWRncHUvYW1kZ3B1X3ZtLmMgICAgICAgIHwgIDcgKy0KIGRyaXZlcnMvZ3B1 L2RybS9hbWQvYW1kZ3B1L2FtZGdwdV92cmFtX21nci5jICB8IDE0ICstLQogZHJpdmVycy9ncHUv ZHJtL2FtZC9hbWRncHUvY2lrX2loLmMgICAgICAgICAgIHwgIDMgKy0KIGRyaXZlcnMvZ3B1L2Ry bS9hbWQvYW1kZ3B1L2N6X2loLmMgICAgICAgICAgICB8ICAzICstCiBkcml2ZXJzL2dwdS9kcm0v YW1kL2FtZGdwdS9pY2VsYW5kX2loLmMgICAgICAgfCAgMyArLQogZHJpdmVycy9ncHUvZHJtL2Ft ZC9hbWRncHUvbmF2aTEwX2loLmMgICAgICAgIHwgIDUgKy0KIGRyaXZlcnMvZ3B1L2RybS9hbWQv YW1kZ3B1L3BzcF92MTFfMC5jICAgICAgICB8IDQ0ICsrKystLS0tLQogZHJpdmVycy9ncHUvZHJt L2FtZC9hbWRncHUvcHNwX3YxMl8wLmMgICAgICAgIHwgIDggKy0KIGRyaXZlcnMvZ3B1L2RybS9h bWQvYW1kZ3B1L3BzcF92M18xLmMgICAgICAgICB8ICA4ICstCiBkcml2ZXJzL2dwdS9kcm0vYW1k L2FtZGdwdS9zaV9paC5jICAgICAgICAgICAgfCAgMyArLQogZHJpdmVycy9ncHUvZHJtL2FtZC9h bWRncHUvdG9uZ2FfaWguYyAgICAgICAgIHwgIDMgKy0KIGRyaXZlcnMvZ3B1L2RybS9hbWQvYW1k Z3B1L3ZjZV92NF8wLmMgICAgICAgICB8IDI2ICsrKy0tCiBkcml2ZXJzL2dwdS9kcm0vYW1kL2Ft ZGdwdS92Y25fdjNfMC5jICAgICAgICAgfCAyMiArKystLQogZHJpdmVycy9ncHUvZHJtL2FtZC9h bWRncHUvdmVnYTEwX2loLmMgICAgICAgIHwgIDUgKy0KIGRyaXZlcnMvZ3B1L2RybS9hbWQvYW1k Z3B1L3ZlZ2EyMF9paC5jICAgICAgICB8ICAyICstCiBkcml2ZXJzL2dwdS9kcm0vYW1kL2FtZGtm ZC9rZmRfZGV2aWNlLmMgICAgICAgfCAgMyArLQogLi4uL2dwdS9kcm0vYW1kL2Rpc3BsYXkvYW1k Z3B1X2RtL2FtZGdwdV9kbS5jIHwgMTMgKystCiBkcml2ZXJzL2dwdS9kcm0vYW1kL2luY2x1ZGUv YW1kX3NoYXJlZC5oICAgICAgfCAgMiArCiAuLi4vZHJtL2FtZC9wbS9wb3dlcnBsYXkvc211bWdy L3NtdTdfc211bWdyLmMgfCAgMiArCiBkcml2ZXJzL2dwdS9kcm0vc2NoZWR1bGVyL3NjaGVkX2Vu dGl0eS5jICAgICAgfCAgMyArLQogZHJpdmVycy9ncHUvZHJtL3NjaGVkdWxlci9zY2hlZF9tYWlu LmMgICAgICAgIHwgMzUgKysrKysrLQogZHJpdmVycy9ncHUvZHJtL3R0bS90dG1fYm9fdm0uYyAg ICAgICAgICAgICAgIHwgNTcgKysrKysrKysrKy0KIGRyaXZlcnMvZ3B1L2RybS90dG0vdHRtX3R0 LmMgICAgICAgICAgICAgICAgICB8ICAxICsKIGRyaXZlcnMvcGNpL3BjaS1kcml2ZXIuYyAgICAg ICAgICAgICAgICAgICAgICB8ICAxICsKIGluY2x1ZGUvZHJtL3R0bS90dG1fYm9fYXBpLmggICAg ICAgICAgICAgICAgICB8ICAyICsKIGluY2x1ZGUvbGludXgvcGNpLmggICAgICAgICAgICAgICAg ICAgICAgICAgICB8ICAzICsKIDUxIGZpbGVzIGNoYW5nZWQsIDU4NSBpbnNlcnRpb25zKCspLCAy NzIgZGVsZXRpb25zKC0pCgotLSAKMi4yNS4xCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fXwphbWQtZ2Z4IG1haWxpbmcgbGlzdAphbWQtZ2Z4QGxpc3RzLmZy ZWVkZXNrdG9wLm9yZwpodHRwczovL2xpc3RzLmZyZWVkZXNrdG9wLm9yZy9tYWlsbWFuL2xpc3Rp bmZvL2FtZC1nZngK