From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.2 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECBC1C433EF for ; Mon, 13 Sep 2021 06:16:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BCC2E60FC0 for ; Mon, 13 Sep 2021 06:16:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237193AbhIMGRO (ORCPT ); Mon, 13 Sep 2021 02:17:14 -0400 Received: from m43-7.mailgun.net ([69.72.43.7]:35944 "EHLO m43-7.mailgun.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234655AbhIMGRN (ORCPT ); Mon, 13 Sep 2021 02:17:13 -0400 DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=mg.codeaurora.org; q=dns/txt; s=smtp; t=1631513758; h=Content-Transfer-Encoding: Content-Type: In-Reply-To: MIME-Version: Date: Message-ID: From: References: Cc: To: Subject: Sender; bh=pREX3LGwYAggCgYEPmLhKYdIv1mP8IQUL0WMH1SUxdA=; b=wLMsUZ3UShA1TC05KHGw6gSEM1w+5NzGCxyEe60NY43Mg4DjQo8w3VI7tUVJjBH+tpnSBfh7 JpNC6f7v7QO6xULA9TBZyE3AYU/vniAO4t50bnPyeTpMdODRVoGmZeNCafqgn5ICWaFbW9A3 i6kRZUzVBec0bRbGUrFR8TpydUU= X-Mailgun-Sending-Ip: 69.72.43.7 X-Mailgun-Sid: WyI1MzIzYiIsICJsaW51eC1hcm0tbXNtQHZnZXIua2VybmVsLm9yZyIsICJiZTllNGEiXQ== Received: from smtp.codeaurora.org (ec2-35-166-182-171.us-west-2.compute.amazonaws.com [35.166.182.171]) by smtp-out-n07.prod.us-east-1.postgun.com with SMTP id 613eec9dd914b051822d1fd3 (version=TLS1.2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256); Mon, 13 Sep 2021 06:15:57 GMT Sender: akhilpo=codeaurora.org@mg.codeaurora.org Received: by smtp.codeaurora.org (Postfix, from userid 1001) id A0E48C43460; Mon, 13 Sep 2021 06:15:56 +0000 (UTC) Received: from [192.168.1.12] (unknown [59.89.228.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: akhilpo) by smtp.codeaurora.org (Postfix) with ESMTPSA id CEDFAC4338F; Mon, 13 Sep 2021 06:15:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 smtp.codeaurora.org CEDFAC4338F Authentication-Results: aws-us-west-2-caf-mail-1.web.codeaurora.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: aws-us-west-2-caf-mail-1.web.codeaurora.org; spf=fail smtp.mailfrom=codeaurora.org Subject: Re: [PATCH] drm/msm: Disable frequency clamping on a630 To: Caleb Connolly , Rob Clark , Bjorn Andersson Cc: dri-devel , freedreno , linux-arm-msm , Rob Clark , Sean Paul , David Airlie , Daniel Vetter , Jordan Crouse , Jonathan Marek , Sai Prakash Ranjan , Sharat Masetty , open list , Stephen Boyd References: <8aa590be-6a9f-9343-e897-18e86ea48202@linaro.org> <6eefedb2-9e59-56d2-7703-2faf6cb0ca3a@codeaurora.org> <83ecbe74-caf0-6c42-e6f5-4887b3b534c6@linaro.org> <53d3e5b7-9dc0-a806-70e9-b9b5ff877462@codeaurora.org> From: Akhil P Oommen Message-ID: Date: Mon, 13 Sep 2021 11:45:50 +0530 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org On 9/10/2021 11:04 PM, Caleb Connolly wrote: > > > On 10/09/2021 18:18, Rob Clark wrote: >> On Tue, Sep 7, 2021 at 7:20 PM Bjorn Andersson >> wrote: >>> >>> On Mon 09 Aug 10:26 PDT 2021, Akhil P Oommen wrote: >>> >>>> On 8/9/2021 9:48 PM, Caleb Connolly wrote: >>>>> >>>>> >>>>> On 09/08/2021 17:12, Rob Clark wrote: >>>>>> On Mon, Aug 9, 2021 at 7:52 AM Akhil P Oommen >>>>>> wrote: >>> [..] >>>>>>> I am a bit confused. We don't define a power domain for gpu in dt, >>>>>>> correct? Then what exactly set_opp do here? Do you think this >>>>>>> usleep is >>>>>>> what is helping here somehow to mask the issue? >>>>> The power domains (for cx and gx) are defined in the GMU DT, the >>>>> OPPs in >>>>> the GPU DT. For the sake of simplicity I'll refer to the lowest >>>>> frequency (257000000) and OPP level (RPMH_REGULATOR_LEVEL_LOW_SVS) as >>>>> the "min" state, and the highest frequency (710000000) and OPP level >>>>> (RPMH_REGULATOR_LEVEL_TURBO_L1) as the "max" state. These are >>>>> defined in >>>>> sdm845.dtsi under the gpu node. >>>>> >>>>> The new devfreq behaviour unmasks what I think is a driver bug, it >>>>> inadvertently puts much more strain on the GPU regulators than they >>>>> usually get. With the new behaviour the GPU jumps from it's min >>>>> state to >>>>> the max state and back again extremely rapidly under workloads as >>>>> small >>>>> as refreshing UI. Where previously the GPU would rarely if ever go >>>>> above >>>>> 342MHz when interacting with the device, it now jumps between min and >>>>> max many times per second. >>>>> >>>>> If my understanding is correct, the current implementation of the GMU >>>>> set freq is the following: >>>>>    - Get OPP for frequency to set >>>>>    - Push the frequency to the GMU - immediately updating the core >>>>> clock >>>>>    - Call dev_pm_opp_set_opp() which triggers a notify chain, this >>>>> winds >>>>> up somewhere in power management code and causes the gx regulator >>>>> level >>>>> to be updated >>>> >>>> Nope. dev_pm_opp_set_opp() sets the bandwidth for gpu and nothing >>>> else. We >>>> were using a different api earlier which got deprecated - >>>> dev_pm_opp_set_bw(). >>>> >>> >>> On the Lenovo Yoga C630 this is reproduced by starting alacritty and if >>> I'm lucky I managed to hit a few keys before it crashes, so I spent a >>> few hours looking into this as well... >>> >>> As you say, the dev_pm_opp_set_opp() will only cast a interconnect vote. >>> The opp-level is just there for show and isn't used by anything, at >>> least not on 845. >>> >>> Further more, I'm missing something in my tree, so the interconnect >>> doesn't hit sync_state, and as such we're not actually scaling the >>> buses. So the problem is not that Linux doesn't turn on the buses in >>> time. >>> >>> So I suspect that the "AHB bus error" isn't saying that we turned off >>> the bus, but rather that the GPU becomes unstable or something of that >>> sort. >>> >>> >>> Lastly, I reverted 9bc95570175a ("drm/msm: Devfreq tuning") and ran >>> Aquarium for 20 minutes without a problem. I then switched the gpu >>> devfreq governor to "userspace" and ran the following: >>> >>> while true; do >>>    echo 257000000 > /sys/class/devfreq/5000000.gpu/userspace/set_freq >>>    echo 710000000 > /sys/class/devfreq/5000000.gpu/userspace/set_freq >>> done >>> >>> It took 19 iterations of this loop to crash the GPU. >> >> I assume you still had aquarium running, to keep the gpu awake while >> you ran that loop? >> >> Fwiw, I modified this slightly to match sc7180's min/max gpu freq and >> could not trigger any issue.. interestingly sc7180 has a lower min >> freq (180) and higher max freq (800) so it was toggling over a wider >> freq range.  I also tried on a device that  had the higher 825MHz opp >> (since I noticed that was the only opp that used >> RPMH_REGULATOR_LEVEL_TURBO_L1 and wanted to rule that out), but could >> not reproduce. >> >> I guess a630 (sdm845) should have higher power draw (it is 2x # of >> shader cores and 2x GMEM size, but lower max freq).. the question is, >> is this the reason we see this on sdm845 and not sc7180?  Or is there >> some other difference.  On the gpu side of this, they are both closely >> related (ie. the same "sub-generation" of a6xx, same gmu fw, etc).. >> I'm less sure about the other parts (icc, rpmh, etc) > > My guess would be power draw, nobody has mentioned this yet but I've > realised that the vdd_gfx rail is powered by a buck converter, which > could explain a lot of the symptoms. > > Buck converters depend on high frequency switching and inductors to > work, this inherently leads to some lag time when changing voltages, and > also means that the behaviour of the regulator is defined in part by how > much current is being drawn. Wikipedia has a pretty good explanation: > https://en.wikipedia.org/wiki/Buck_converter > > At the best of times these regulators have a known voltage ripple, when > under load and when rapidly switching voltages this will get a lot worse. > > Someone with an oscilloscope and schematics could probe the rail and > probably see exactly what's going on when the GPU crashes. Because of > the lag time in the regulator changing voltage, it might be > undershooting whilst the GPU is trying to clock up and draw more current > - causing instability and crashes. Both of you are correct. The GPU is very similar including the GMU (we have same fw for both), except the GBIF block. As far as I am aware, the non-gpu blocks within SoC should be similar except the configs. And yes, for these sort of issues where we suspect a power issue, gx rail should be probed for droops using a very high resolution oscilloscopes (these droops might last less than 1us). I am aware of only Dragonboard that is still alive from QC perspective. Can someone report this issue to DB support team as it is fairly easy to reproduce? -Akhil. >> >> BR, >> -R >> >>> So the problem doesn't seem to be Rob's change, it's just that prior to >>> it the chance to hitting it is way lower. Question is still what it is >>> that we're triggering. >>> >>> Regards, >>> Bjorn >