From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752986AbcE2S1L (ORCPT ); Sun, 29 May 2016 14:27:11 -0400 Received: from mail-oi0-f66.google.com ([209.85.218.66]:34187 "EHLO mail-oi0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752784AbcE2S1J (ORCPT ); Sun, 29 May 2016 14:27:09 -0400 MIME-Version: 1.0 X-Originating-IP: [2a02:168:56b5:0:ac27:b86c:7764:9429] In-Reply-To: <1464311916-10065-1-git-send-email-mcgrof@kernel.org> References: <20160527004642.GQ11948@wotan.suse.de> <1464311916-10065-1-git-send-email-mcgrof@kernel.org> Date: Sun, 29 May 2016 20:27:07 +0200 X-Google-Sender-Auth: X3YVjIT25dUxetVx4KliDbIAxkU Message-ID: Subject: Re: [RFT v3] drm: use late_initcall() for amdkfd and radeon From: Daniel Vetter To: "Luis R. Rodriguez" Cc: vw@iommu.org, Oded Gabbay , Joerg Roedel , =?UTF-8?Q?Christian_K=C3=B6nig?= , "alexander.deucher@amd.com" , Dave Airlie , iommu@lists.linux-foundation.org, Linux Kernel Mailing List , dri-devel Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, May 27, 2016 at 3:18 AM, Luis R. Rodriguez wrote: > To get KFD support in radeon we need the following > initialization to happen in this order, their > respective driver file that has its init routine > listed next to it: > > 0. AMD IOMMUv1: arch/x86/kernel/pci-dma.c > 1. AMD IOMMUv2: drivers/iommu/amd_iommu_v2.c > 2. AMD KFD: drivers/gpu/drm/amd/amdkfd/kfd_module.c > 3. AMD Radeon: drivers/gpu/drm/radeon/radeon_drv.c > > Order is rather implicit, but these drivers can currently > only do so much given the amount of leg room available. > Below are the respective init routines and how they are > initialized: > > arch/x86/kernel/pci-dma.c rootfs_initcall(pci_iommu_init); > drivers/iommu/amd_iommu_v2.c module_init(amd_iommu_v2_init); > drivers/gpu/drm/amd/amdkfd/kfd_module.c module_init(kfd_module_init); > drivers/gpu/drm/radeon/radeon_drv.c module_init(radeon_init); > > When a driver is built-in module_init() folds to use > device_initcall(), and we have the following possible > orders: > > #define pure_initcall(fn) __define_initcall(fn, 0) > #define core_initcall(fn) __define_initcall(fn, 1) > #define postcore_initcall(fn)__define_initcall(fn, 2) > #define arch_initcall(fn) __define_initcall(fn, 3) > #define subsys_initcall(fn) __define_initcall(fn, 4) > #define fs_initcall(fn) __define_initcall(fn, 5) > --------------------------------------------------------- > #define rootfs_initcall(fn) __define_initcall(fn, rootfs) > #define device_initcall(fn) __define_initcall(fn, 6) > #define late_initcall(fn) __define_initcall(fn, 7) > > Since we start off from rootfs_initcall(), it gives us 3 more > levels of leg room to play with for order semantics, this isn't > enough to address all required levels of dependencies, this > is specially true given that AMD-KFD needs to be loaded before > the radeon driver -- -but this it not enforced by symbols. > If the AMD-KFD driver is not loaded prior to the radeon driver > because otherwise the radeon driver will not initialize the > AMD-KFD driver and you get no KFD functionality in userspace. > > Commit 1bacc894c227fad8a7 ("drivers: Move iommu/ before gpu/ in > Makefile") works around some of the possibe races between > the AMD IOMMU v2 and GPU drivers by changing the link order. > This is fragile, however its the bets we can do, given that > making the GPU drivers use late_initcall() would also implicate > a similar race between them. That possible race is fortunatley > addressed given that the drm Makefile currently has amdkfd > linked prior to radeon: > > drivers/gpu/drm/Makefile > ... > obj-$(CONFIG_HSA_AMD) += amd/amdkfd/ > obj-$(CONFIG_DRM_RADEON)+= radeon/ > ... > > Changing amdkfd and radeon to late_initcall() however is > still the right call in orde to annotate explicitly a > delayed dependency requirement between the GPU drivers > and the IOMMUs. > > We can't address the fragile nature of the link order > right now, but in the future that might be possible. > > Signed-off-by: Luis R. Rodriguez > --- > > Please note, the changes to drivers/Makefile are just > for the sake of forcing the possible race to occur, > if this works well the actual [PATCH] submission will > skip those changes as its pointless to remove those > work arounds as it stands, due to the limited nature > of the levels available for addressing requirements. > > Also, if you are aware of further dependency hell > things like these -- please do let me know as I am > interested in looking at addressing them. This should be fixed with -EDEFER_PROBE instead. Frobbing initcall levels should then just be done as an optimization to avoid too much reprobing. Thanks, Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch