From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.3 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 330CFC433DB for ; Sun, 21 Feb 2021 13:45:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 061C964EDB for ; Sun, 21 Feb 2021 13:45:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230024AbhBUNpY (ORCPT ); Sun, 21 Feb 2021 08:45:24 -0500 Received: from mx3.molgen.mpg.de ([141.14.17.11]:57133 "EHLO mx1.molgen.mpg.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S229802AbhBUNpU (ORCPT ); Sun, 21 Feb 2021 08:45:20 -0500 Received: from [192.168.1.176] (dynamic-078-055-128-098.78.55.pool.telefonica.de [78.55.128.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: pmenzel) by mx.molgen.mpg.de (Postfix) with ESMTPSA id 5723A2064792E; Sun, 21 Feb 2021 14:44:36 +0100 (CET) Subject: Re: [PATCH] iommu/amd: Fix event counter availability check From: Paul Menzel To: Alexander Monakov , linux-kernel@vger.kernel.org Cc: iommu@lists.linux-foundation.org, Shuah Khan , =?UTF-8?B?SsO2cmcgUsO2ZGVs?= , Suravee Suthikulpanit References: <20200529200738.1923-1-amonakov@ispras.ru> <1b2652e9-903e-c02f-8393-2775ae60f0b4@molgen.mpg.de> Message-ID: <90b556ae-abdc-ffe7-e473-15ff1488f572@molgen.mpg.de> Date: Sun, 21 Feb 2021 14:44:36 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: <1b2652e9-903e-c02f-8393-2775ae60f0b4@molgen.mpg.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Dear Alexander, Am 01.06.20 um 04:48 schrieb Paul Menzel: […] > Am 31.05.20 um 09:22 schrieb Alexander Monakov: > >> Adding Shuah Khan to Cc: I've noticed you've seen this issue on Ryzen 2400GE; >> can you have a look at the patch? Would be nice to know if it fixes the >> problem for you too. > >> On Fri, 29 May 2020, Alexander Monakov wrote: >> >>> The driver performs an extra check if the IOMMU's capabilities advertise >>> presence of performance counters: it verifies that counters are writable >>> by writing a hard-coded value to a counter and testing that reading that >>> counter gives back the same value. >>> >>> Unfortunately it does so quite early, even before pci_enable_device is >>> called for the IOMMU, i.e. when accessing its MMIO space is not >>> guaranteed to work. On Ryzen 4500U CPU, this actually breaks the test: >>> the driver assumes the counters are not writable, and disables the >>> functionality. >>> >>> Moving init_iommu_perf_ctr just after iommu_flush_all_caches resolves >>> the issue. This is the earliest point in amd_iommu_init_pci where the >>> call succeeds on my laptop. >>> >>> Signed-off-by: Alexander Monakov >>> Cc: Joerg Roedel >>> Cc: Suravee Suthikulpanit >>> Cc: iommu@lists.linux-foundation.org >>> --- >>> >>> PS. I'm seeing another hiccup with IOMMU probing on my system: >>> pci 0000:00:00.2: can't derive routing for PCI INT A >>> pci 0000:00:00.2: PCI INT A: not connected >>> >>> Hopefully I can figure it out, but I'd appreciate hints. > > I guess it’s a firmware bug, but I contacted the linux-pci folks [1]. Unfortunately, it’s still present in Linux 5.11. >>>   drivers/iommu/amd_iommu_init.c | 6 +++--- >>>   1 file changed, 3 insertions(+), 3 deletions(-) >>> >>> diff --git a/drivers/iommu/amd_iommu_init.c >>> b/drivers/iommu/amd_iommu_init.c >>> index 5b81fd16f5fa..1b7ec6b6a282 100644 >>> --- a/drivers/iommu/amd_iommu_init.c >>> +++ b/drivers/iommu/amd_iommu_init.c >>> @@ -1788,8 +1788,6 @@ static int __init iommu_init_pci(struct >>> amd_iommu *iommu) >>>       if (iommu->cap & (1UL << IOMMU_CAP_NPCACHE)) >>>           amd_iommu_np_cache = true; >>> -    init_iommu_perf_ctr(iommu); >>> - >>>       if (is_rd890_iommu(iommu->dev)) { >>>           int i, j; >>> @@ -1891,8 +1889,10 @@ static int __init amd_iommu_init_pci(void) >>>       init_device_table_dma(); >>> -    for_each_iommu(iommu) >>> +    for_each_iommu(iommu) { >>>           iommu_flush_all_caches(iommu); >>> +        init_iommu_perf_ctr(iommu); >>> +    } >>>       if (!ret) >>>           print_iommu_info(); >>> >>> base-commit: 75caf310d16cc5e2f851c048cd597f5437013368 > > Thank you very much for fixing this issue, which is almost two years old > for me. > > Tested-by: Paul Menzel > MSI MSI MS-7A37/B350M MORTAR with AMD Ryzen 3 2200G > Link: https://lore.kernel.org/linux-iommu/20180727102710.GA6738@8bytes.org/ Just a small note, that I am applying your patch, but it looks like there is still some timing issue. At least today, I noticed it during one boot with Linux 5.11. (Before I never noticed it again in the several years, but I am not always paying attention and do not save the logs.) Kind regards, Paul > [1]: https://lore.kernel.org/linux-pci/8579bd14-e369-1141-917b-204d20cff528@molgen.mpg.de/ From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.3 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17DAFC433DB for ; Sun, 21 Feb 2021 13:44:52 +0000 (UTC) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5C31E64D99 for ; Sun, 21 Feb 2021 13:44:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5C31E64D99 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=molgen.mpg.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 0FC866ED68 for ; Sun, 21 Feb 2021 13:44:51 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YL1-ANFPQ2T7 for ; Sun, 21 Feb 2021 13:44:49 +0000 (UTC) Received: by smtp3.osuosl.org (Postfix, from userid 1001) id 3E0146F47A; Sun, 21 Feb 2021 13:44:47 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTP id 04E80605C9; Sun, 21 Feb 2021 13:44:43 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id DBB44C000B; Sun, 21 Feb 2021 13:44:43 +0000 (UTC) Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 5037BC0001 for ; Sun, 21 Feb 2021 13:44:41 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 3F2DE87094 for ; Sun, 21 Feb 2021 13:44:41 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id L1eLxxFlZ8MC for ; Sun, 21 Feb 2021 13:44:39 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from mx1.molgen.mpg.de (mx3.molgen.mpg.de [141.14.17.11]) by hemlock.osuosl.org (Postfix) with ESMTPS id 7488187092 for ; Sun, 21 Feb 2021 13:44:39 +0000 (UTC) Received: from [192.168.1.176] (dynamic-078-055-128-098.78.55.pool.telefonica.de [78.55.128.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: pmenzel) by mx.molgen.mpg.de (Postfix) with ESMTPSA id 5723A2064792E; Sun, 21 Feb 2021 14:44:36 +0100 (CET) Subject: Re: [PATCH] iommu/amd: Fix event counter availability check From: Paul Menzel To: Alexander Monakov , linux-kernel@vger.kernel.org References: <20200529200738.1923-1-amonakov@ispras.ru> <1b2652e9-903e-c02f-8393-2775ae60f0b4@molgen.mpg.de> Message-ID: <90b556ae-abdc-ffe7-e473-15ff1488f572@molgen.mpg.de> Date: Sun, 21 Feb 2021 14:44:36 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: <1b2652e9-903e-c02f-8393-2775ae60f0b4@molgen.mpg.de> Content-Language: en-US Cc: iommu@lists.linux-foundation.org, Shuah Khan X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" RGVhciBBbGV4YW5kZXIsCgoKQW0gMDEuMDYuMjAgdW0gMDQ6NDggc2NocmllYiBQYXVsIE1lbnpl bDoKClvigKZdCgo+IEFtIDMxLjA1LjIwIHVtIDA5OjIyIHNjaHJpZWIgQWxleGFuZGVyIE1vbmFr b3Y6Cj4gCj4+IEFkZGluZyBTaHVhaCBLaGFuIHRvIENjOiBJJ3ZlIG5vdGljZWQgeW91J3ZlIHNl ZW4gdGhpcyBpc3N1ZSBvbiBSeXplbiAyNDAwR0U7Cj4+IGNhbiB5b3UgaGF2ZSBhIGxvb2sgYXQg dGhlIHBhdGNoPyBXb3VsZCBiZSBuaWNlIHRvIGtub3cgaWYgaXQgZml4ZXMgdGhlCj4+IHByb2Js ZW0gZm9yIHlvdSB0b28uCj4gCj4+IE9uIEZyaSwgMjkgTWF5IDIwMjAsIEFsZXhhbmRlciBNb25h a292IHdyb3RlOgo+Pgo+Pj4gVGhlIGRyaXZlciBwZXJmb3JtcyBhbiBleHRyYSBjaGVjayBpZiB0 aGUgSU9NTVUncyBjYXBhYmlsaXRpZXMgYWR2ZXJ0aXNlCj4+PiBwcmVzZW5jZSBvZiBwZXJmb3Jt YW5jZSBjb3VudGVyczogaXQgdmVyaWZpZXMgdGhhdCBjb3VudGVycyBhcmUgd3JpdGFibGUKPj4+ IGJ5IHdyaXRpbmcgYSBoYXJkLWNvZGVkIHZhbHVlIHRvIGEgY291bnRlciBhbmQgdGVzdGluZyB0 aGF0IHJlYWRpbmcgdGhhdAo+Pj4gY291bnRlciBnaXZlcyBiYWNrIHRoZSBzYW1lIHZhbHVlLgo+ Pj4KPj4+IFVuZm9ydHVuYXRlbHkgaXQgZG9lcyBzbyBxdWl0ZSBlYXJseSwgZXZlbiBiZWZvcmUg cGNpX2VuYWJsZV9kZXZpY2UgaXMKPj4+IGNhbGxlZCBmb3IgdGhlIElPTU1VLCBpLmUuIHdoZW4g YWNjZXNzaW5nIGl0cyBNTUlPIHNwYWNlIGlzIG5vdAo+Pj4gZ3VhcmFudGVlZCB0byB3b3JrLiBP biBSeXplbiA0NTAwVSBDUFUsIHRoaXMgYWN0dWFsbHkgYnJlYWtzIHRoZSB0ZXN0Ogo+Pj4gdGhl IGRyaXZlciBhc3N1bWVzIHRoZSBjb3VudGVycyBhcmUgbm90IHdyaXRhYmxlLCBhbmQgZGlzYWJs ZXMgdGhlCj4+PiBmdW5jdGlvbmFsaXR5Lgo+Pj4KPj4+IE1vdmluZyBpbml0X2lvbW11X3BlcmZf Y3RyIGp1c3QgYWZ0ZXIgaW9tbXVfZmx1c2hfYWxsX2NhY2hlcyByZXNvbHZlcwo+Pj4gdGhlIGlz c3VlLiBUaGlzIGlzIHRoZSBlYXJsaWVzdCBwb2ludCBpbiBhbWRfaW9tbXVfaW5pdF9wY2kgd2hl cmUgdGhlCj4+PiBjYWxsIHN1Y2NlZWRzIG9uIG15IGxhcHRvcC4KPj4+Cj4+PiBTaWduZWQtb2Zm LWJ5OiBBbGV4YW5kZXIgTW9uYWtvdiA8YW1vbmFrb3ZAaXNwcmFzLnJ1Pgo+Pj4gQ2M6IEpvZXJn IFJvZWRlbCA8am9yb0A4Ynl0ZXMub3JnPgo+Pj4gQ2M6IFN1cmF2ZWUgU3V0aGlrdWxwYW5pdCA8 c3VyYXZlZS5zdXRoaWt1bHBhbml0QGFtZC5jb20+Cj4+PiBDYzogaW9tbXVAbGlzdHMubGludXgt Zm91bmRhdGlvbi5vcmcKPj4+IC0tLQo+Pj4KPj4+IFBTLiBJJ20gc2VlaW5nIGFub3RoZXIgaGlj Y3VwIHdpdGggSU9NTVUgcHJvYmluZyBvbiBteSBzeXN0ZW06Cj4+PiBwY2kgMDAwMDowMDowMC4y OiBjYW4ndCBkZXJpdmUgcm91dGluZyBmb3IgUENJIElOVCBBCj4+PiBwY2kgMDAwMDowMDowMC4y OiBQQ0kgSU5UIEE6IG5vdCBjb25uZWN0ZWQKPj4+Cj4+PiBIb3BlZnVsbHkgSSBjYW4gZmlndXJl IGl0IG91dCwgYnV0IEknZCBhcHByZWNpYXRlIGhpbnRzLgo+IAo+IEkgZ3Vlc3MgaXTigJlzIGEg ZmlybXdhcmUgYnVnLCBidXQgSSBjb250YWN0ZWQgdGhlIGxpbnV4LXBjaSBmb2xrcyBbMV0uCgpV bmZvcnR1bmF0ZWx5LCBpdOKAmXMgc3RpbGwgcHJlc2VudCBpbiBMaW51eCA1LjExLgoKPj4+IMKg IGRyaXZlcnMvaW9tbXUvYW1kX2lvbW11X2luaXQuYyB8IDYgKysrLS0tCj4+PiDCoCAxIGZpbGUg Y2hhbmdlZCwgMyBpbnNlcnRpb25zKCspLCAzIGRlbGV0aW9ucygtKQo+Pj4KPj4+IGRpZmYgLS1n aXQgYS9kcml2ZXJzL2lvbW11L2FtZF9pb21tdV9pbml0LmMgCj4+PiBiL2RyaXZlcnMvaW9tbXUv YW1kX2lvbW11X2luaXQuYwo+Pj4gaW5kZXggNWI4MWZkMTZmNWZhLi4xYjdlYzZiNmEyODIgMTAw NjQ0Cj4+PiAtLS0gYS9kcml2ZXJzL2lvbW11L2FtZF9pb21tdV9pbml0LmMKPj4+ICsrKyBiL2Ry aXZlcnMvaW9tbXUvYW1kX2lvbW11X2luaXQuYwo+Pj4gQEAgLTE3ODgsOCArMTc4OCw2IEBAIHN0 YXRpYyBpbnQgX19pbml0IGlvbW11X2luaXRfcGNpKHN0cnVjdCAKPj4+IGFtZF9pb21tdSAqaW9t bXUpCj4+PiDCoMKgwqDCoMKgIGlmIChpb21tdS0+Y2FwICYgKDFVTCA8PCBJT01NVV9DQVBfTlBD QUNIRSkpCj4+PiDCoMKgwqDCoMKgwqDCoMKgwqAgYW1kX2lvbW11X25wX2NhY2hlID0gdHJ1ZTsK Pj4+IC3CoMKgwqAgaW5pdF9pb21tdV9wZXJmX2N0cihpb21tdSk7Cj4+PiAtCj4+PiDCoMKgwqDC oMKgIGlmIChpc19yZDg5MF9pb21tdShpb21tdS0+ZGV2KSkgewo+Pj4gwqDCoMKgwqDCoMKgwqDC oMKgIGludCBpLCBqOwo+Pj4gQEAgLTE4OTEsOCArMTg4OSwxMCBAQCBzdGF0aWMgaW50IF9faW5p dCBhbWRfaW9tbXVfaW5pdF9wY2kodm9pZCkKPj4+IMKgwqDCoMKgwqAgaW5pdF9kZXZpY2VfdGFi bGVfZG1hKCk7Cj4+PiAtwqDCoMKgIGZvcl9lYWNoX2lvbW11KGlvbW11KQo+Pj4gK8KgwqDCoCBm b3JfZWFjaF9pb21tdShpb21tdSkgewo+Pj4gwqDCoMKgwqDCoMKgwqDCoMKgIGlvbW11X2ZsdXNo X2FsbF9jYWNoZXMoaW9tbXUpOwo+Pj4gK8KgwqDCoMKgwqDCoMKgIGluaXRfaW9tbXVfcGVyZl9j dHIoaW9tbXUpOwo+Pj4gK8KgwqDCoCB9Cj4+PiDCoMKgwqDCoMKgIGlmICghcmV0KQo+Pj4gwqDC oMKgwqDCoMKgwqDCoMKgIHByaW50X2lvbW11X2luZm8oKTsKPj4+Cj4+PiBiYXNlLWNvbW1pdDog NzVjYWYzMTBkMTZjYzVlMmY4NTFjMDQ4Y2Q1OTdmNTQzNzAxMzM2OAo+IAo+IFRoYW5rIHlvdSB2 ZXJ5IG11Y2ggZm9yIGZpeGluZyB0aGlzIGlzc3VlLCB3aGljaCBpcyBhbG1vc3QgdHdvIHllYXJz IG9sZCAKPiBmb3IgbWUuCj4gCj4gVGVzdGVkLWJ5OiBQYXVsIE1lbnplbCA8cG1lbnplbEBtb2xn ZW4ubXBnLmRlPgo+IE1TSSBNU0kgTVMtN0EzNy9CMzUwTSBNT1JUQVIgd2l0aCBBTUQgUnl6ZW4g MyAyMjAwRwo+IExpbms6IGh0dHBzOi8vbG9yZS5rZXJuZWwub3JnL2xpbnV4LWlvbW11LzIwMTgw NzI3MTAyNzEwLkdBNjczOEA4Ynl0ZXMub3JnLwoKSnVzdCBhIHNtYWxsIG5vdGUsIHRoYXQgSSBh bSBhcHBseWluZyB5b3VyIHBhdGNoLCBidXQgaXQgbG9va3MgbGlrZSAKdGhlcmUgaXMgc3RpbGwg c29tZSB0aW1pbmcgaXNzdWUuIEF0IGxlYXN0IHRvZGF5LCBJIG5vdGljZWQgaXQgZHVyaW5nIApv bmUgYm9vdCB3aXRoIExpbnV4IDUuMTEuIChCZWZvcmUgSSBuZXZlciBub3RpY2VkIGl0IGFnYWlu IGluIHRoZSAKc2V2ZXJhbCB5ZWFycywgYnV0IEkgYW0gbm90IGFsd2F5cyBwYXlpbmcgYXR0ZW50 aW9uIGFuZCBkbyBub3Qgc2F2ZSB0aGUgCmxvZ3MuKQoKCktpbmQgcmVnYXJkcywKClBhdWwKCgo+ IFsxXTogaHR0cHM6Ly9sb3JlLmtlcm5lbC5vcmcvbGludXgtcGNpLzg1NzliZDE0LWUzNjktMTE0 MS05MTdiLTIwNGQyMGNmZjUyOEBtb2xnZW4ubXBnLmRlLwpfX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fXwppb21tdSBtYWlsaW5nIGxpc3QKaW9tbXVAbGlzdHMu bGludXgtZm91bmRhdGlvbi5vcmcKaHR0cHM6Ly9saXN0cy5saW51eGZvdW5kYXRpb24ub3JnL21h aWxtYW4vbGlzdGluZm8vaW9tbXU=