From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.9 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 410FBC43603 for ; Thu, 19 Dec 2019 09:54:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B5ED9206D8 for ; Thu, 19 Dec 2019 09:54:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=ffwll.ch header.i=@ffwll.ch header.b="dLEl3UPj" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726908AbfLSJyo (ORCPT ); Thu, 19 Dec 2019 04:54:44 -0500 Received: from mail-ot1-f66.google.com ([209.85.210.66]:41925 "EHLO mail-ot1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726742AbfLSJyn (ORCPT ); Thu, 19 Dec 2019 04:54:43 -0500 Received: by mail-ot1-f66.google.com with SMTP id r27so6530168otc.8 for ; Thu, 19 Dec 2019 01:54:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=0WmVu4IgtlTUPIDTr6BuAgCj4OJ6/8JQ0XMLcherZ8w=; b=dLEl3UPj/egc82qia3GmcXdXUheNN1g7oyA70vrPbp7ndmfYzCQB+6RYiMKjTjJsSV GqXFiVh2m+XYYojPGs1qdmjVzRUHgW5KWKlUl0oMoSeazNFIJ7XhjFS2v5iBGMxD0Avl wiAF16AyHND/6ugwQCXiI8izcVbAOGUNF8EC4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=0WmVu4IgtlTUPIDTr6BuAgCj4OJ6/8JQ0XMLcherZ8w=; b=M3Lut2V+QmPYBB4C3IW9kRHggQGcTLDCw+61gliIYv61T8YZp+82CX2dYvGKqIWKyL e9eO+IekhG/HOSRwIp2IVlJvjKPKmL0AJePojTK4htyaDIG4xpUyi32Wp2IRR2wYgZ4t +OYffJeTLXdDXDclgNj8jqj4rTYhOkPzzoKYPCZk847476en6WThFWr8MeENhBK4fqQA TEeE+2Mxae07On/lnCEtzGXKiKSeSl4l4R4l2ZpTpsVIsnmFo4un/meI+e1JyRp0ds5Y eZLPuvzRmKi0OZuIGPFVLQxM7X5FTMyiFpcxLj8vzrb/VmfW6J66azb4EfXo7cuLtRMt K7xQ== X-Gm-Message-State: APjAAAUBiSuL1fbL4wST7JKKIiUC2fT4A+qJLHI2UCuTkL/0wEBfLa95 78x9aHID5hkzsRjVy3EoIhkK/RNdxB1PMpHKVaDlFA== X-Google-Smtp-Source: APXvYqwpF+JmqQuuWIu5qxZwkxKhECtZxRuDFYOTuRUJ1SYwpacyQ6Erxna5CodtXbxTmADy1SSXmN2K02DmVNgj5ko= X-Received: by 2002:a9d:7510:: with SMTP id r16mr7791826otk.303.1576749282374; Thu, 19 Dec 2019 01:54:42 -0800 (PST) MIME-Version: 1.0 References: <07899bd5-e9a5-cff0-395f-b4fb3f0f7f6c@huawei.com> In-Reply-To: From: Daniel Vetter Date: Thu, 19 Dec 2019 10:54:31 +0100 Message-ID: Subject: Re: Warnings in DRM code when removing/unbinding a driver To: John Garry Cc: Ezequiel Garcia , "kongxinwei (A)" , "Chenfeng (puck)" , "airlied@linux.ie" , Thomas Zimmermann , Linuxarm , "dri-devel@lists.freedesktop.org" , "linux-kernel@vger.kernel.org" , Gerd Hoffmann , dbueso@suse.de Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Dec 18, 2019 at 7:08 PM John Garry wrote: > > + > > So the v5.4 kernel does not have this issue. > > I have bisected the initial occurrence to: > > commit 37a48adfba6cf6e87df9ba8b75ab85d514ed86d8 > Author: Thomas Zimmermann > Date: Fri Sep 6 14:20:53 2019 +0200 > > drm/vram: Add kmap ref-counting to GEM VRAM objects > > The kmap and kunmap operations of GEM VRAM buffers can now be called > in interleaving pairs. The first call to drm_gem_vram_kmap() maps the > buffer's memory to kernel address space and the final call to > drm_gem_vram_kunmap() unmaps the memory. Intermediate calls to these > functions increment or decrement a reference counter. > > So this either exposes or creates the issue. Yeah that's just shooting the messenger. Like I said, for most drivers you can pretty much assume that their unload sequence has been broken since forever. It's not often tested, and especially the hotunbind from a device (as opposed to driver unload) stuff wasn't even possible to get right until just recently. -Daniel > > John > > >> On Mon, 2019-12-16 at 17:23 +0000, John Garry wrote: > >>> Hi all, > >>> > >>> Enabling CONFIG_DEBUG_TEST_DRIVER_REMOVE causes many warns on a system > >>> with the HIBMC hw: > >>> > >>> [ 27.788806] WARNING: CPU: 24 PID: 1 at > >>> drivers/gpu/drm/drm_gem_vram_helper.c:564 > >>> bo_driver_move_notify+0x8c/0x98 > >> > >> A total shot in the dark. This might make no sense, > >> but it's worth a try: > > > > Thanks for the suggestion, but still the same splat. > > > > I haven't had a chance to analyze the problem myself. But perhaps we > > should just change over the device-managed interface, as Daniel mentioned. > > > >> > >> diff --git a/drivers/gpu/drm/hisilicon/hibmc/hibmc_drm_drv.c > >> b/drivers/gpu/drm/hisilicon/hibmc/hibmc_drm_drv.c > >> index 2fd4ca91a62d..69bb0e29da88 100644 > >> --- a/drivers/gpu/drm/hisilicon/hibmc/hibmc_drm_drv.c > >> +++ b/drivers/gpu/drm/hisilicon/hibmc/hibmc_drm_drv.c > >> @@ -247,9 +247,8 @@ static int hibmc_unload(struct drm_device *dev) > >> { > >> struct hibmc_drm_private *priv = dev->dev_private; > >> - hibmc_fbdev_fini(priv); > >> - > >> drm_atomic_helper_shutdown(dev); > >> + hibmc_fbdev_fini(priv); > >> if (dev->irq_enabled) > >> drm_irq_uninstall(dev); > >> > >> Hope it helps, > >> Ezequiel > >> > > > > Thanks, > > John > > > > [EOM] > > > >>> [ 27.798969] Modules linked in: > >>> [ 27.802018] CPU: 24 PID: 1 Comm: swapper/0 Tainted: G B > >>> 5.5.0-rc1-dirty #565 > >>> [ 27.810358] Hardware name: Huawei D06 /D06, BIOS Hisilicon D06 UEFI > >>> RC0 - V1.16.01 03/15/2019 > >>> [ 27.818872] pstate: 20c00009 (nzCv daif +PAN +UAO) > >>> [ 27.823654] pc : bo_driver_move_notify+0x8c/0x98 > >>> [ 27.828262] lr : bo_driver_move_notify+0x40/0x98 > >>> [ 27.832868] sp : ffff00236f0677e0 > >>> [ 27.836173] x29: ffff00236f0677e0 x28: ffffa0001454e5e0 > >>> [ 27.841476] x27: ffff002366e52128 x26: ffffa000149e67b0 > >>> [ 27.846779] x25: ffff002366e523e0 x24: ffff002336936120 > >>> [ 27.852082] x23: ffff0023346f4010 x22: ffff002336936128 > >>> [ 27.857385] x21: ffffa000149c15c0 x20: ffff0023369361f8 > >>> [ 27.862687] x19: ffff002336936000 x18: 0000000000001258 > >>> [ 27.867989] x17: 0000000000001190 x16: 00000000000011d0 > >>> [ 27.873292] x15: 0000000000001348 x14: ffffa00012d68190 > >>> [ 27.878595] x13: 0000000000000006 x12: 1ffff40003241f91 > >>> [ 27.883897] x11: ffff940003241f91 x10: dfffa00000000000 > >>> [ 27.889200] x9 : ffff940003241f92 x8 : 0000000000000001 > >>> [ 27.894502] x7 : ffffa0001920fc88 x6 : ffff940003241f92 > >>> [ 27.899804] x5 : ffff940003241f92 x4 : ffff0023369363a0 > >>> [ 27.905107] x3 : ffffa00010c104b8 x2 : dfffa00000000000 > >>> [ 27.910409] x1 : 0000000000000003 x0 : 0000000000000001 > >>> [ 27.915712] Call trace: > >>> [ 27.918151] bo_driver_move_notify+0x8c/0x98 > >>> [ 27.922412] ttm_bo_cleanup_memtype_use+0x54/0x100 > >>> [ 27.927194] ttm_bo_put+0x3a0/0x5d0 > >>> [ 27.930673] drm_gem_vram_object_free+0xc/0x18 > >>> [ 27.935109] drm_gem_object_free+0x34/0xd0 > >>> [ 27.939196] drm_gem_object_put_unlocked+0xc8/0xf0 > >>> [ 27.943978] hibmc_user_framebuffer_destroy+0x20/0x40 > >>> [ 27.949020] drm_framebuffer_free+0x48/0x58 > >>> [ 27.953194] drm_mode_object_put.part.1+0x90/0xe8 > >>> [ 27.957889] drm_mode_object_put+0x28/0x38 > >>> [ 27.961976] hibmc_fbdev_fini+0x54/0x78 > >>> [ 27.965802] hibmc_unload+0x2c/0xd0 > >>> [ 27.969281] hibmc_pci_remove+0x2c/0x40 > >>> [ 27.973109] pci_device_remove+0x6c/0x140 > >>> [ 27.977110] really_probe+0x174/0x548 > >>> [ 27.980763] driver_probe_device+0x7c/0x148 > >>> [ 27.984936] device_driver_attach+0x94/0xa0 > >>> [ 27.989109] __driver_attach+0xa8/0x110 > >>> [ 27.992935] bus_for_each_dev+0xe8/0x158 > >>> [ 27.996849] driver_attach+0x30/0x40 > >>> [ 28.000415] bus_add_driver+0x234/0x2f0 > >>> [ 28.004241] driver_register+0xbc/0x1d0 > >>> [ 28.008067] __pci_register_driver+0xbc/0xd0 > >>> [ 28.012329] hibmc_pci_driver_init+0x20/0x28 > >>> [ 28.016590] do_one_initcall+0xb4/0x254 > >>> [ 28.020417] kernel_init_freeable+0x27c/0x328 > >>> [ 28.024765] kernel_init+0x10/0x118 > >>> [ 28.028245] ret_from_fork+0x10/0x18 > >>> [ 28.031813] ---[ end trace 35a83b71b657878d ]--- > >>> [ 28.036503] ------------[ cut here ]------------ > >>> [ 28.041115] WARNING: CPU: 24 PID: 1 at > >>> drivers/gpu/drm/drm_gem_vram_helper.c:40 > >>> ttm_buffer_object_destroy+0x4c/0x80 > >>> [ 28.051537] Modules linked in: > >>> [ 28.054585] CPU: 24 PID: 1 Comm: swapper/0 Tainted: G B W > >>> 5.5.0-rc1-dirty #565 > >>> [ 28.062924] Hardware name: Huawei D06 /D06, BIOS Hisilicon D06 UEFI > >>> RC0 - V1.16.01 03/15/2019 > >>> > >>> [snip] > >>> > >>> Indeed, simply unbinding the device from the driver causes the same sort > >>> of issue: > >>> > >>> root@(none)$ cd ./bus/pci/drivers/hibmc-drm/ > >>> root@(none)$ ls > >>> 0000:05:00.0 bind new_id remove_id uevent > >>> unbind > >>> root@(none)$ echo 0000\:05\:00.0 > unbind > >>> [ 116.074352] ------------[ cut here ]------------ > >>> [ 116.078978] WARNING: CPU: 17 PID: 1178 at > >>> drivers/gpu/drm/drm_gem_vram_helper.c:40 > >>> ttm_buffer_object_destroy+0x4c/0x80 > >>> [ 116.089661] Modules linked in: > >>> [ 116.092711] CPU: 17 PID: 1178 Comm: sh Tainted: G B W > >>> 5.5.0-rc1-dirty #565 > >>> [ 116.100704] Hardware name: Huawei D06 /D06, BIOS Hisilicon D06 UEFI > >>> RC0 - V1.16.01 03/15/2019 > >>> [ 116.109218] pstate: 20400009 (nzCv daif +PAN -UAO) > >>> [ 116.114001] pc : ttm_buffer_object_destroy+0x4c/0x80 > >>> [ 116.118956] lr : ttm_buffer_object_destroy+0x18/0x80 > >>> [ 116.123910] sp : ffff0022e6cef8e0 > >>> [ 116.127215] x29: ffff0022e6cef8e0 x28: ffff00231b1fb000 > >>> [ 116.132519] x27: 0000000000000000 x26: ffff00231b1fb000 > >>> [ 116.137821] x25: ffff0022e6cefdc0 x24: 0000000000002480 > >>> [ 116.143124] x23: ffff0023682b6ab0 x22: ffff0023682b6800 > >>> [ 116.148427] x21: ffff0023682b6800 x20: 0000000000000000 > >>> [ 116.153730] x19: ffff0023682b6800 x18: 0000000000000000 > >>> [ 116.159032] x17: 000000000000000000000000001 > >>> [ 116.185545] x7 : ffff0023682b6b07 x6 : ffff80046d056d61 > >>> [ 116.190848] x5 : ffff80046d056d61 x4 : ffff0023682b6ba0 > >>> [ 116.196151] x3 : ffffa00010197338 x2 : dfffa00000000000 > >>> [ 116.201453] x1 : 0000000000000003 x0 : 0000000000000001 > >>> [ 116.206756] Call trace: > >>> [ 116.209195] ttm_buffer_object_destroy+0x4c/0x80 > >>> [ 116.213803] ttm_bo_release_list+0x184/0x220 > >>> [ 116.218064] ttm_bo_put+0x410/0x5d0 > >>> [ 116.221544] drm_gem_vram_object_free+0xc/0x18 > >>> [ 116.225979] drm_gem_object_free+0x34/0xd0 > >>> [ 116.230066] drm_gem_object_put_unlocked+0xc8/0xf0 > >>> [ 116.234848] hibmc_user_framebuffer_destroy+0x20/0x40 > >>> [ 116.239890] drm_framebuffer_free+0x48/0x58 > >>> [ 116.244064] drm_mode_object_put.part.1+0x90/0xe8 > >>> [ 116.248759] drm_mode_object_put+0x28/0x38 > >>> [ 116.252846] hibmc_fbdev_fini+0x54/0x78 > >>> [ 116.256672] hibmc_unload+0x2c/0xd0 > >>> [ 116.260151] hibmc_pci_remove+0x2c/0x40 > >>> [ 116.263979] pci_device_remove+0x6c/0x140 > >>> [ 116.267980] device_release_driver_internal+0x134/0x250 > >>> [ 116.273196] device_driver_detach+0x28/0x38 > >>> [ 116.277369] unbind_store+0xfc/0x150 > >>> [ 116.280934] drv_attr_store+0x48/0x60 > >>> [ 116.284589] sysfs_kf_write+0x80/0xb0 > >>> [ 116.288241] kernfs_fop_write+0x1d4/0x320 > >>> [ 116.292243] __vfs_write+0x54/0x98 > >>> [ 116.295635] vfs_write+0xe8/0x270 > >>> [ 116.298940] ksys_write+0xc8/0x180 > >>> [ 116.302333] __arm64_sys_write+0x40/0x50 > >>> [ 116.306248] el0_svc_common.constprop.0+0xa4/0x1f8 > >>> [ 116.311029] el0_svc_handler+0x34/0xb0 > >>> [ 116.314770] el0_sync_handler+0x10c/0x1c8 > >>> [ 116.318769] el0_sync+0x140/0x180 > >>> [ 116.322074] ---[ end trace e60e43d0e316b5c8 ]--- > >>> [ 116.326868] ------------[ cut here ]------------ > >>> > >>> > >>> dmesg and .config is here: > >>> https://pastebin.com/4P5yaZBS > >>> > >>> I'm not sure if this is a HIBMC driver issue or issue with the > >>> framework. > >>> > >>> john > >>> > >>> > >>> _______________________________________________ > >>> dri-devel mailing list > >>> dri-devel@lists.freedesktop.org > >>> https://lists.freedesktop.org/mailman/listinfo/dri-devel > >> > >> > > > > _______________________________________________ > > Linuxarm mailing list > > Linuxarm@huawei.com > > http://hulk.huawei.com/mailman/listinfo/linuxarm > > . > -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.5 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2367AC2D0C0 for ; Thu, 19 Dec 2019 09:54:46 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E9F29206D8 for ; Thu, 19 Dec 2019 09:54:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=ffwll.ch header.i=@ffwll.ch header.b="dLEl3UPj" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E9F29206D8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 56B676EAF3; Thu, 19 Dec 2019 09:54:45 +0000 (UTC) Received: from mail-ot1-x341.google.com (mail-ot1-x341.google.com [IPv6:2607:f8b0:4864:20::341]) by gabe.freedesktop.org (Postfix) with ESMTPS id 604E16EAF3 for ; Thu, 19 Dec 2019 09:54:43 +0000 (UTC) Received: by mail-ot1-x341.google.com with SMTP id 66so6527644otd.9 for ; Thu, 19 Dec 2019 01:54:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=0WmVu4IgtlTUPIDTr6BuAgCj4OJ6/8JQ0XMLcherZ8w=; b=dLEl3UPj/egc82qia3GmcXdXUheNN1g7oyA70vrPbp7ndmfYzCQB+6RYiMKjTjJsSV GqXFiVh2m+XYYojPGs1qdmjVzRUHgW5KWKlUl0oMoSeazNFIJ7XhjFS2v5iBGMxD0Avl wiAF16AyHND/6ugwQCXiI8izcVbAOGUNF8EC4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=0WmVu4IgtlTUPIDTr6BuAgCj4OJ6/8JQ0XMLcherZ8w=; b=LIRGzO8MO6h6Vgu8ugc8joA68kkZnVmITnsWcBXD46tjpSv9koy3hzQgC77tdhzwzp 9MNxHd4Y3Nb9IbMY0fCQEuNaDSzZaCvkFIWYsnyqc7qGFxW95HnmpCULzccvah/4ozKb Ie2F9syMeuzF4eQ9V8HNKolfh0H2RJaGsVlcmWdzFpGOya9Hb+0quJZSDvUsD2gVeN5r YPuAudzQVyhtmfT5a6ga3/qnUzMggzvImW9UrnFB1rMGvOGP7obeLG1VDn/F9Rf3CU5s oJqiKXQdLsMwUpwNBzVVKlJiHIA3rlY0tqbByP24L/eH0U2znikt503jNDXK49wM1HIz PEPA== X-Gm-Message-State: APjAAAXdFPMwjRIWuonqWH6qtchyHcCViikaUXvDxJz2PVczRWMWUW1R RJKGY218nr4gURO/ARj2HQgDcreqNi6zNNrjPFtKUA== X-Google-Smtp-Source: APXvYqwpF+JmqQuuWIu5qxZwkxKhECtZxRuDFYOTuRUJ1SYwpacyQ6Erxna5CodtXbxTmADy1SSXmN2K02DmVNgj5ko= X-Received: by 2002:a9d:7510:: with SMTP id r16mr7791826otk.303.1576749282374; Thu, 19 Dec 2019 01:54:42 -0800 (PST) MIME-Version: 1.0 References: <07899bd5-e9a5-cff0-395f-b4fb3f0f7f6c@huawei.com> In-Reply-To: From: Daniel Vetter Date: Thu, 19 Dec 2019 10:54:31 +0100 Message-ID: Subject: Re: Warnings in DRM code when removing/unbinding a driver To: John Garry X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: dbueso@suse.de, "airlied@linux.ie" , "Chenfeng \(puck\)" , Linuxarm , "dri-devel@lists.freedesktop.org" , "linux-kernel@vger.kernel.org" , "kongxinwei \(A\)" , Gerd Hoffmann , Thomas Zimmermann , Ezequiel Garcia Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" On Wed, Dec 18, 2019 at 7:08 PM John Garry wrote: > > + > > So the v5.4 kernel does not have this issue. > > I have bisected the initial occurrence to: > > commit 37a48adfba6cf6e87df9ba8b75ab85d514ed86d8 > Author: Thomas Zimmermann > Date: Fri Sep 6 14:20:53 2019 +0200 > > drm/vram: Add kmap ref-counting to GEM VRAM objects > > The kmap and kunmap operations of GEM VRAM buffers can now be called > in interleaving pairs. The first call to drm_gem_vram_kmap() maps the > buffer's memory to kernel address space and the final call to > drm_gem_vram_kunmap() unmaps the memory. Intermediate calls to these > functions increment or decrement a reference counter. > > So this either exposes or creates the issue. Yeah that's just shooting the messenger. Like I said, for most drivers you can pretty much assume that their unload sequence has been broken since forever. It's not often tested, and especially the hotunbind from a device (as opposed to driver unload) stuff wasn't even possible to get right until just recently. -Daniel > > John > > >> On Mon, 2019-12-16 at 17:23 +0000, John Garry wrote: > >>> Hi all, > >>> > >>> Enabling CONFIG_DEBUG_TEST_DRIVER_REMOVE causes many warns on a system > >>> with the HIBMC hw: > >>> > >>> [ 27.788806] WARNING: CPU: 24 PID: 1 at > >>> drivers/gpu/drm/drm_gem_vram_helper.c:564 > >>> bo_driver_move_notify+0x8c/0x98 > >> > >> A total shot in the dark. This might make no sense, > >> but it's worth a try: > > > > Thanks for the suggestion, but still the same splat. > > > > I haven't had a chance to analyze the problem myself. But perhaps we > > should just change over the device-managed interface, as Daniel mentioned. > > > >> > >> diff --git a/drivers/gpu/drm/hisilicon/hibmc/hibmc_drm_drv.c > >> b/drivers/gpu/drm/hisilicon/hibmc/hibmc_drm_drv.c > >> index 2fd4ca91a62d..69bb0e29da88 100644 > >> --- a/drivers/gpu/drm/hisilicon/hibmc/hibmc_drm_drv.c > >> +++ b/drivers/gpu/drm/hisilicon/hibmc/hibmc_drm_drv.c > >> @@ -247,9 +247,8 @@ static int hibmc_unload(struct drm_device *dev) > >> { > >> struct hibmc_drm_private *priv = dev->dev_private; > >> - hibmc_fbdev_fini(priv); > >> - > >> drm_atomic_helper_shutdown(dev); > >> + hibmc_fbdev_fini(priv); > >> if (dev->irq_enabled) > >> drm_irq_uninstall(dev); > >> > >> Hope it helps, > >> Ezequiel > >> > > > > Thanks, > > John > > > > [EOM] > > > >>> [ 27.798969] Modules linked in: > >>> [ 27.802018] CPU: 24 PID: 1 Comm: swapper/0 Tainted: G B > >>> 5.5.0-rc1-dirty #565 > >>> [ 27.810358] Hardware name: Huawei D06 /D06, BIOS Hisilicon D06 UEFI > >>> RC0 - V1.16.01 03/15/2019 > >>> [ 27.818872] pstate: 20c00009 (nzCv daif +PAN +UAO) > >>> [ 27.823654] pc : bo_driver_move_notify+0x8c/0x98 > >>> [ 27.828262] lr : bo_driver_move_notify+0x40/0x98 > >>> [ 27.832868] sp : ffff00236f0677e0 > >>> [ 27.836173] x29: ffff00236f0677e0 x28: ffffa0001454e5e0 > >>> [ 27.841476] x27: ffff002366e52128 x26: ffffa000149e67b0 > >>> [ 27.846779] x25: ffff002366e523e0 x24: ffff002336936120 > >>> [ 27.852082] x23: ffff0023346f4010 x22: ffff002336936128 > >>> [ 27.857385] x21: ffffa000149c15c0 x20: ffff0023369361f8 > >>> [ 27.862687] x19: ffff002336936000 x18: 0000000000001258 > >>> [ 27.867989] x17: 0000000000001190 x16: 00000000000011d0 > >>> [ 27.873292] x15: 0000000000001348 x14: ffffa00012d68190 > >>> [ 27.878595] x13: 0000000000000006 x12: 1ffff40003241f91 > >>> [ 27.883897] x11: ffff940003241f91 x10: dfffa00000000000 > >>> [ 27.889200] x9 : ffff940003241f92 x8 : 0000000000000001 > >>> [ 27.894502] x7 : ffffa0001920fc88 x6 : ffff940003241f92 > >>> [ 27.899804] x5 : ffff940003241f92 x4 : ffff0023369363a0 > >>> [ 27.905107] x3 : ffffa00010c104b8 x2 : dfffa00000000000 > >>> [ 27.910409] x1 : 0000000000000003 x0 : 0000000000000001 > >>> [ 27.915712] Call trace: > >>> [ 27.918151] bo_driver_move_notify+0x8c/0x98 > >>> [ 27.922412] ttm_bo_cleanup_memtype_use+0x54/0x100 > >>> [ 27.927194] ttm_bo_put+0x3a0/0x5d0 > >>> [ 27.930673] drm_gem_vram_object_free+0xc/0x18 > >>> [ 27.935109] drm_gem_object_free+0x34/0xd0 > >>> [ 27.939196] drm_gem_object_put_unlocked+0xc8/0xf0 > >>> [ 27.943978] hibmc_user_framebuffer_destroy+0x20/0x40 > >>> [ 27.949020] drm_framebuffer_free+0x48/0x58 > >>> [ 27.953194] drm_mode_object_put.part.1+0x90/0xe8 > >>> [ 27.957889] drm_mode_object_put+0x28/0x38 > >>> [ 27.961976] hibmc_fbdev_fini+0x54/0x78 > >>> [ 27.965802] hibmc_unload+0x2c/0xd0 > >>> [ 27.969281] hibmc_pci_remove+0x2c/0x40 > >>> [ 27.973109] pci_device_remove+0x6c/0x140 > >>> [ 27.977110] really_probe+0x174/0x548 > >>> [ 27.980763] driver_probe_device+0x7c/0x148 > >>> [ 27.984936] device_driver_attach+0x94/0xa0 > >>> [ 27.989109] __driver_attach+0xa8/0x110 > >>> [ 27.992935] bus_for_each_dev+0xe8/0x158 > >>> [ 27.996849] driver_attach+0x30/0x40 > >>> [ 28.000415] bus_add_driver+0x234/0x2f0 > >>> [ 28.004241] driver_register+0xbc/0x1d0 > >>> [ 28.008067] __pci_register_driver+0xbc/0xd0 > >>> [ 28.012329] hibmc_pci_driver_init+0x20/0x28 > >>> [ 28.016590] do_one_initcall+0xb4/0x254 > >>> [ 28.020417] kernel_init_freeable+0x27c/0x328 > >>> [ 28.024765] kernel_init+0x10/0x118 > >>> [ 28.028245] ret_from_fork+0x10/0x18 > >>> [ 28.031813] ---[ end trace 35a83b71b657878d ]--- > >>> [ 28.036503] ------------[ cut here ]------------ > >>> [ 28.041115] WARNING: CPU: 24 PID: 1 at > >>> drivers/gpu/drm/drm_gem_vram_helper.c:40 > >>> ttm_buffer_object_destroy+0x4c/0x80 > >>> [ 28.051537] Modules linked in: > >>> [ 28.054585] CPU: 24 PID: 1 Comm: swapper/0 Tainted: G B W > >>> 5.5.0-rc1-dirty #565 > >>> [ 28.062924] Hardware name: Huawei D06 /D06, BIOS Hisilicon D06 UEFI > >>> RC0 - V1.16.01 03/15/2019 > >>> > >>> [snip] > >>> > >>> Indeed, simply unbinding the device from the driver causes the same sort > >>> of issue: > >>> > >>> root@(none)$ cd ./bus/pci/drivers/hibmc-drm/ > >>> root@(none)$ ls > >>> 0000:05:00.0 bind new_id remove_id uevent > >>> unbind > >>> root@(none)$ echo 0000\:05\:00.0 > unbind > >>> [ 116.074352] ------------[ cut here ]------------ > >>> [ 116.078978] WARNING: CPU: 17 PID: 1178 at > >>> drivers/gpu/drm/drm_gem_vram_helper.c:40 > >>> ttm_buffer_object_destroy+0x4c/0x80 > >>> [ 116.089661] Modules linked in: > >>> [ 116.092711] CPU: 17 PID: 1178 Comm: sh Tainted: G B W > >>> 5.5.0-rc1-dirty #565 > >>> [ 116.100704] Hardware name: Huawei D06 /D06, BIOS Hisilicon D06 UEFI > >>> RC0 - V1.16.01 03/15/2019 > >>> [ 116.109218] pstate: 20400009 (nzCv daif +PAN -UAO) > >>> [ 116.114001] pc : ttm_buffer_object_destroy+0x4c/0x80 > >>> [ 116.118956] lr : ttm_buffer_object_destroy+0x18/0x80 > >>> [ 116.123910] sp : ffff0022e6cef8e0 > >>> [ 116.127215] x29: ffff0022e6cef8e0 x28: ffff00231b1fb000 > >>> [ 116.132519] x27: 0000000000000000 x26: ffff00231b1fb000 > >>> [ 116.137821] x25: ffff0022e6cefdc0 x24: 0000000000002480 > >>> [ 116.143124] x23: ffff0023682b6ab0 x22: ffff0023682b6800 > >>> [ 116.148427] x21: ffff0023682b6800 x20: 0000000000000000 > >>> [ 116.153730] x19: ffff0023682b6800 x18: 0000000000000000 > >>> [ 116.159032] x17: 000000000000000000000000001 > >>> [ 116.185545] x7 : ffff0023682b6b07 x6 : ffff80046d056d61 > >>> [ 116.190848] x5 : ffff80046d056d61 x4 : ffff0023682b6ba0 > >>> [ 116.196151] x3 : ffffa00010197338 x2 : dfffa00000000000 > >>> [ 116.201453] x1 : 0000000000000003 x0 : 0000000000000001 > >>> [ 116.206756] Call trace: > >>> [ 116.209195] ttm_buffer_object_destroy+0x4c/0x80 > >>> [ 116.213803] ttm_bo_release_list+0x184/0x220 > >>> [ 116.218064] ttm_bo_put+0x410/0x5d0 > >>> [ 116.221544] drm_gem_vram_object_free+0xc/0x18 > >>> [ 116.225979] drm_gem_object_free+0x34/0xd0 > >>> [ 116.230066] drm_gem_object_put_unlocked+0xc8/0xf0 > >>> [ 116.234848] hibmc_user_framebuffer_destroy+0x20/0x40 > >>> [ 116.239890] drm_framebuffer_free+0x48/0x58 > >>> [ 116.244064] drm_mode_object_put.part.1+0x90/0xe8 > >>> [ 116.248759] drm_mode_object_put+0x28/0x38 > >>> [ 116.252846] hibmc_fbdev_fini+0x54/0x78 > >>> [ 116.256672] hibmc_unload+0x2c/0xd0 > >>> [ 116.260151] hibmc_pci_remove+0x2c/0x40 > >>> [ 116.263979] pci_device_remove+0x6c/0x140 > >>> [ 116.267980] device_release_driver_internal+0x134/0x250 > >>> [ 116.273196] device_driver_detach+0x28/0x38 > >>> [ 116.277369] unbind_store+0xfc/0x150 > >>> [ 116.280934] drv_attr_store+0x48/0x60 > >>> [ 116.284589] sysfs_kf_write+0x80/0xb0 > >>> [ 116.288241] kernfs_fop_write+0x1d4/0x320 > >>> [ 116.292243] __vfs_write+0x54/0x98 > >>> [ 116.295635] vfs_write+0xe8/0x270 > >>> [ 116.298940] ksys_write+0xc8/0x180 > >>> [ 116.302333] __arm64_sys_write+0x40/0x50 > >>> [ 116.306248] el0_svc_common.constprop.0+0xa4/0x1f8 > >>> [ 116.311029] el0_svc_handler+0x34/0xb0 > >>> [ 116.314770] el0_sync_handler+0x10c/0x1c8 > >>> [ 116.318769] el0_sync+0x140/0x180 > >>> [ 116.322074] ---[ end trace e60e43d0e316b5c8 ]--- > >>> [ 116.326868] ------------[ cut here ]------------ > >>> > >>> > >>> dmesg and .config is here: > >>> https://pastebin.com/4P5yaZBS > >>> > >>> I'm not sure if this is a HIBMC driver issue or issue with the > >>> framework. > >>> > >>> john > >>> > >>> > >>> _______________________________________________ > >>> dri-devel mailing list > >>> dri-devel@lists.freedesktop.org > >>> https://lists.freedesktop.org/mailman/listinfo/dri-devel > >> > >> > > > > _______________________________________________ > > Linuxarm mailing list > > Linuxarm@huawei.com > > http://hulk.huawei.com/mailman/listinfo/linuxarm > > . > -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel