From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.3 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1DACC388F2 for ; Thu, 22 Oct 2020 08:08:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8EF6F223BF for ; Thu, 22 Oct 2020 08:08:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="VXy3s7Qj" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2894371AbgJVIIH (ORCPT ); Thu, 22 Oct 2020 04:08:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58848 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2894374AbgJVIGr (ORCPT ); Thu, 22 Oct 2020 04:06:47 -0400 Received: from mail-pj1-x1043.google.com (mail-pj1-x1043.google.com [IPv6:2607:f8b0:4864:20::1043]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9085CC0613CF for ; Thu, 22 Oct 2020 01:06:47 -0700 (PDT) Received: by mail-pj1-x1043.google.com with SMTP id hk7so573110pjb.2 for ; Thu, 22 Oct 2020 01:06:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=TVfb7WnwNXKGUxz4iiR3sXsOPTJFuGFroopSW7eD6Gk=; b=VXy3s7QjvhLDl7KsBOg8MVmOhDTShSxGrDo6aWcBaOc02FD0F85qf0Ti+YtY62gqo0 fQvsKNmMSTtfrleV5GPKxLU+JcUEc2NSUgl+bZqbTTJxWGVdAW4Ks+6O+WpMyT0UYZQU Rk77fNgDCNEX1rBFx3tEw9PNvK5zoEOTcw4Exzt4s4Kw7B/lg/rPlU9NFNmT5+fHkPsJ L6jVrWsH6VfFT6UlRDL93//CvMxZ+gekuFnxD58I6yOGqSKCXsq41PVDX1T/ttn8Zw46 jyhR4qHH+pZ/cQmtByv3Eoc+7RyZzWcCjxuptfiSir3cOI6jaqjAtUWeYCqvGt5RwgoA 0OuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=TVfb7WnwNXKGUxz4iiR3sXsOPTJFuGFroopSW7eD6Gk=; b=Ek8iACJK+gui2wnjefvOM7QDPXbRMPr3VWFoEqtgzoUh9AVD6BBWKnovn7cTJNApLP yYkY3bZ6jFtV6l/HPRcFkIO3Htd8eUB3M+eqcLKC0WSRui3DMiGYgPNb0BXVhwl7aN/M jisQEpQW4F3iL6uwttL3v5ueERsz2l4GkOaG1j1fzu7ixnJaSEUDT5DnSjOGfAVPQS+z Y5pni2IP5VO77FrTu1VobMetDLZx3yANuwy+3GNU4306u5H9wcCZ1cJvAp9+D6TUZj+z O9IHoV10XE9FtVO5NRMTtIgMjTWdA0YT7L+ydDNNmgXza35r1HMtN8dq3PK0WUaeEtb9 RymQ== X-Gm-Message-State: AOAM532pZKhtsmehVM73s5bkPsEBWmMmACH2WjS/LyKKh/1TdOfxJMV4 PzniTXXx7UTg5DU62JdO0Lc/sw== X-Google-Smtp-Source: ABdhPJwVqL7oVk5t6FqFadGaytgpFZzZfoLCZKLY/TTEFKuYJ70F6UW4MwiJzoxuD6cE1PnzwUK9mQ== X-Received: by 2002:a17:902:7fcd:b029:d3:f037:d9dc with SMTP id t13-20020a1709027fcdb02900d3f037d9dcmr1305142plb.79.1603354007060; Thu, 22 Oct 2020 01:06:47 -0700 (PDT) Received: from localhost ([122.181.54.133]) by smtp.gmail.com with ESMTPSA id v13sm1292150pgl.6.2020.10.22.01.06.45 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 22 Oct 2020 01:06:46 -0700 (PDT) Date: Thu, 22 Oct 2020 13:36:44 +0530 From: Viresh Kumar To: Rob Clark Cc: Daniel Vetter , dri-devel , Rob Clark , Sean Paul , David Airlie , "open list:DRM DRIVER FOR MSM ADRENO GPU" , "open list:DRM DRIVER FOR MSM ADRENO GPU" , open list , "Menon, Nishanth" Subject: Re: [PATCH v2 07/22] drm/msm: Do rpm get sooner in the submit path Message-ID: <20201022080644.2ck4okrxygmkuatn@vireshk-i7> References: <20201012020958.229288-1-robdclark@gmail.com> <20201012020958.229288-8-robdclark@gmail.com> <20201012143555.GA438822@phenom.ffwll.local> <20201020090729.qgqish5kqamhvatj@vireshk-i7> <20201020112413.xbk2vow2kgjky3pb@vireshk-i7> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: NeoMutt/20180716-391-311a52 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org On 20-10-20, 07:13, Rob Clark wrote: > On Tue, Oct 20, 2020 at 4:24 AM Viresh Kumar wrote: > > > > On 20-10-20, 12:56, Daniel Vetter wrote: > > > Yeah that's bad practice. Generally you shouldn't need to hold locks > > > in setup/teardown code, since there's no other thread which can > > > possible hold a reference to anything your touching anymore. Ofc > > > excluding quickly grabbing/dropping a lock to insert/remove objects > > > into lists and stuff. > > > > > > The other reason is that especially with anything related to sysfs or > > > debugfs, the locking dependencies you're pulling in are enormous: vfs > > > locks pull in mm locks (due to mmap) and at that point there's pretty > > > much nothing left you're allowed to hold while acquiring such a lock. > > > For simple drivers this is no issue, but for fancy drivers (like gpu > > > drivers) which need to interact with core mm) this means your > > > subsystem is a major pain to use. > > > > > > Usually the correct fix is to only hold your subsystem locks in > > > setup/teardown when absolutely required, and fix any data > > > inconsistency issues by reordering your setup/teardown code: When you > > > register as the last step and unregister as the first step, there's no > > > need for any additional locking. And hence no need to call debugfs > > > functions while holding your subsystem locks. > > > > > > The catch phrase I use for this is "don't solve object lifetime issues > > > with locking". Instead use refcounting and careful ordering in > > > setup/teardown code. > > > > This is exactly what I have done in the OPP core, the locks were taken > > only when really necessary, though as we have seen now I have missed > > that at a single place and that should be fixed as well. Will do that, > > thanks. > > I do have an easy enough way to repro the issue, so if you have a > patch I can certainly test it. Does this fix it for you ? There is one more place still left where we are taking the opp_table_lock while adding stuff to debugfs and that's not that straight forward to fix. But I didn't see that path in your circular dependency trace, so who knows :) diff --git a/drivers/opp/core.c b/drivers/opp/core.c index 2483e765318a..4cc0fb716381 100644 --- a/drivers/opp/core.c +++ b/drivers/opp/core.c @@ -1181,6 +1181,10 @@ static void _opp_table_kref_release(struct kref *kref) struct opp_device *opp_dev, *temp; int i; + /* Drop the lock as soon as we can */ + list_del(&opp_table->node); + mutex_unlock(&opp_table_lock); + _of_clear_opp_table(opp_table); /* Release clk */ @@ -1208,10 +1212,7 @@ static void _opp_table_kref_release(struct kref *kref) mutex_destroy(&opp_table->genpd_virt_dev_lock); mutex_destroy(&opp_table->lock); - list_del(&opp_table->node); kfree(opp_table); - - mutex_unlock(&opp_table_lock); } void dev_pm_opp_put_opp_table(struct opp_table *opp_table) -- viresh From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 372F1C55178 for ; Fri, 23 Oct 2020 07:23:04 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CFE3720936 for ; Fri, 23 Oct 2020 07:23:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="VXy3s7Qj" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CFE3720936 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 4E3A96E4B1; Fri, 23 Oct 2020 07:22:49 +0000 (UTC) Received: from mail-pj1-x1044.google.com (mail-pj1-x1044.google.com [IPv6:2607:f8b0:4864:20::1044]) by gabe.freedesktop.org (Postfix) with ESMTPS id 7FB726F53B for ; Thu, 22 Oct 2020 08:06:47 +0000 (UTC) Received: by mail-pj1-x1044.google.com with SMTP id m3so570427pjf.4 for ; Thu, 22 Oct 2020 01:06:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=TVfb7WnwNXKGUxz4iiR3sXsOPTJFuGFroopSW7eD6Gk=; b=VXy3s7QjvhLDl7KsBOg8MVmOhDTShSxGrDo6aWcBaOc02FD0F85qf0Ti+YtY62gqo0 fQvsKNmMSTtfrleV5GPKxLU+JcUEc2NSUgl+bZqbTTJxWGVdAW4Ks+6O+WpMyT0UYZQU Rk77fNgDCNEX1rBFx3tEw9PNvK5zoEOTcw4Exzt4s4Kw7B/lg/rPlU9NFNmT5+fHkPsJ L6jVrWsH6VfFT6UlRDL93//CvMxZ+gekuFnxD58I6yOGqSKCXsq41PVDX1T/ttn8Zw46 jyhR4qHH+pZ/cQmtByv3Eoc+7RyZzWcCjxuptfiSir3cOI6jaqjAtUWeYCqvGt5RwgoA 0OuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=TVfb7WnwNXKGUxz4iiR3sXsOPTJFuGFroopSW7eD6Gk=; b=CIw4Q+I+Qw+0qJpB45/mxbTO8spLDN6i7jpJEeoHdP4ssIYXR/ojMQax3bRQnb40V9 s7ZUSUNo9kvyik1p/XdJmGUl2mvPKNLB/YUwSvLLyce0dbQdCPCC0DiwkmCRS+xprbHy K6AMg5OVmtiBEwCoHzVcUXTjJJcYMVWZ2YKHabGfoP81/S9JAIDQ8bqfwmvrUbeXuoJc cEoMtVgug6cVHa6OJ0MqdsSWi0Mi8lnilxDbdkIG/pYOkzddf8ns5+PGQjoGTG0O5dD/ WS/SDFBPdVyB420qUwzsCIM3MkEvC97bBS/KZIhbiQHna/Luf/lPBU5/LW+ctFCLNbOv QHgg== X-Gm-Message-State: AOAM532n9AOc0Q7DXJHMM1sgWgXgRz6/hI6FDoNta0SYEaSS2CXhkeQk thUHTwzPBTi4IzYnlFcHPKJBlQ== X-Google-Smtp-Source: ABdhPJwVqL7oVk5t6FqFadGaytgpFZzZfoLCZKLY/TTEFKuYJ70F6UW4MwiJzoxuD6cE1PnzwUK9mQ== X-Received: by 2002:a17:902:7fcd:b029:d3:f037:d9dc with SMTP id t13-20020a1709027fcdb02900d3f037d9dcmr1305142plb.79.1603354007060; Thu, 22 Oct 2020 01:06:47 -0700 (PDT) Received: from localhost ([122.181.54.133]) by smtp.gmail.com with ESMTPSA id v13sm1292150pgl.6.2020.10.22.01.06.45 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 22 Oct 2020 01:06:46 -0700 (PDT) Date: Thu, 22 Oct 2020 13:36:44 +0530 From: Viresh Kumar To: Rob Clark Subject: Re: [PATCH v2 07/22] drm/msm: Do rpm get sooner in the submit path Message-ID: <20201022080644.2ck4okrxygmkuatn@vireshk-i7> References: <20201012020958.229288-1-robdclark@gmail.com> <20201012020958.229288-8-robdclark@gmail.com> <20201012143555.GA438822@phenom.ffwll.local> <20201020090729.qgqish5kqamhvatj@vireshk-i7> <20201020112413.xbk2vow2kgjky3pb@vireshk-i7> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: NeoMutt/20180716-391-311a52 X-Mailman-Approved-At: Fri, 23 Oct 2020 07:22:29 +0000 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Rob Clark , "open list:DRM DRIVER FOR MSM ADRENO GPU" , "Menon, Nishanth" , David Airlie , "open list:DRM DRIVER FOR MSM ADRENO GPU" , open list , dri-devel , Sean Paul Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" On 20-10-20, 07:13, Rob Clark wrote: > On Tue, Oct 20, 2020 at 4:24 AM Viresh Kumar wrote: > > > > On 20-10-20, 12:56, Daniel Vetter wrote: > > > Yeah that's bad practice. Generally you shouldn't need to hold locks > > > in setup/teardown code, since there's no other thread which can > > > possible hold a reference to anything your touching anymore. Ofc > > > excluding quickly grabbing/dropping a lock to insert/remove objects > > > into lists and stuff. > > > > > > The other reason is that especially with anything related to sysfs or > > > debugfs, the locking dependencies you're pulling in are enormous: vfs > > > locks pull in mm locks (due to mmap) and at that point there's pretty > > > much nothing left you're allowed to hold while acquiring such a lock. > > > For simple drivers this is no issue, but for fancy drivers (like gpu > > > drivers) which need to interact with core mm) this means your > > > subsystem is a major pain to use. > > > > > > Usually the correct fix is to only hold your subsystem locks in > > > setup/teardown when absolutely required, and fix any data > > > inconsistency issues by reordering your setup/teardown code: When you > > > register as the last step and unregister as the first step, there's no > > > need for any additional locking. And hence no need to call debugfs > > > functions while holding your subsystem locks. > > > > > > The catch phrase I use for this is "don't solve object lifetime issues > > > with locking". Instead use refcounting and careful ordering in > > > setup/teardown code. > > > > This is exactly what I have done in the OPP core, the locks were taken > > only when really necessary, though as we have seen now I have missed > > that at a single place and that should be fixed as well. Will do that, > > thanks. > > I do have an easy enough way to repro the issue, so if you have a > patch I can certainly test it. Does this fix it for you ? There is one more place still left where we are taking the opp_table_lock while adding stuff to debugfs and that's not that straight forward to fix. But I didn't see that path in your circular dependency trace, so who knows :) diff --git a/drivers/opp/core.c b/drivers/opp/core.c index 2483e765318a..4cc0fb716381 100644 --- a/drivers/opp/core.c +++ b/drivers/opp/core.c @@ -1181,6 +1181,10 @@ static void _opp_table_kref_release(struct kref *kref) struct opp_device *opp_dev, *temp; int i; + /* Drop the lock as soon as we can */ + list_del(&opp_table->node); + mutex_unlock(&opp_table_lock); + _of_clear_opp_table(opp_table); /* Release clk */ @@ -1208,10 +1212,7 @@ static void _opp_table_kref_release(struct kref *kref) mutex_destroy(&opp_table->genpd_virt_dev_lock); mutex_destroy(&opp_table->lock); - list_del(&opp_table->node); kfree(opp_table); - - mutex_unlock(&opp_table_lock); } void dev_pm_opp_put_opp_table(struct opp_table *opp_table) -- viresh _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel