[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

Neil (ledufff@hotmail.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |ledufff@hotmail.com

--- Comment #1 from Neil (ledufff@hotmail.com) ---
Similar finding:
[    1.144340] pci 0000:00:00.2: AMD-Vi: Unable to read/write to IOMMU perf
counter.
[    1.144456] pci 0000:00:00.2: can't derive routing for PCI INT A
[    1.144457] pci 0000:00:00.2: PCI INT A: not connected
[    1.144521] pci 0000:00:01.0: Adding to iommu group 0
[    1.144774] pci 0000:00:01.2: Adding to iommu group 1
[    1.145009] pci 0000:00:01.3: Adding to iommu group 2
[    1.145245] pci 0000:00:01.6: Adding to iommu group 3
[    1.145468] pci 0000:00:08.0: Adding to iommu group 4
[    1.145677] pci 0000:00:08.1: Adding to iommu group 5
[    1.145899] pci 0000:00:08.2: Adding to iommu group 4
[    1.145924] pci 0000:00:14.0: Adding to iommu group 6
[    1.146173] pci 0000:00:14.3: Adding to iommu group 6
[    1.146224] pci 0000:00:18.0: Adding to iommu group 7
[    1.146438] pci 0000:00:18.1: Adding to iommu group 7
[    1.146464] pci 0000:00:18.2: Adding to iommu group 7
[    1.146481] pci 0000:00:18.3: Adding to iommu group 7
[    1.146498] pci 0000:00:18.4: Adding to iommu group 7
[    1.146515] pci 0000:00:18.5: Adding to iommu group 7
[    1.146542] pci 0000:00:18.6: Adding to iommu group 7
[    1.146560] pci 0000:00:18.7: Adding to iommu group 7
[    1.146598] pci 0000:01:00.0: Adding to iommu group 8
[    1.146851] pci 0000:02:00.0: Adding to iommu group 9
[    1.147143] pci 0000:03:00.0: Adding to iommu group 10
[    1.147467] pci 0000:04:00.0: Adding to iommu group 11
[    1.147606] pci 0000:04:00.0: Using iommu direct mapping
[    1.147648] pci 0000:04:00.1: Adding to iommu group 12
[    1.147875] pci 0000:04:00.2: Adding to iommu group 12
[    1.147912] pci 0000:04:00.3: Adding to iommu group 12
[    1.147941] pci 0000:04:00.4: Adding to iommu group 12
[    1.147979] pci 0000:04:00.5: Adding to iommu group 12
[    1.148015] pci 0000:04:00.6: Adding to iommu group 12
[    1.148032] pci 0000:05:00.0: Adding to iommu group 4
[    1.148310] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.148311] pci 0000:00:00.2: AMD-Vi: Extended features (0x4f77ef22294ada):
[    1.148312]  PPR NX GT IA GA PC GA_vAPIC

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #2 from Neil (ledufff@hotmail.com) ---
Created attachment 288095
  --> https://bugzilla.kernel.org/attachment.cgi?id=288095&action=edit
dmesg e495

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

Suravee Suthikulpanit (suravee.suthikulpanit@amd.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |suravee.suthikulpanit@amd.c
                   |                            |om

--- Comment #3 from Suravee Suthikulpanit (suravee.suthikulpanit@amd.com) ---
Created attachment 295035
  --> https://bugzilla.kernel.org/attachment.cgi?id=295035&action=edit
Proposed workaround for the IOMMU PMC issue

Could you please try the attached patch to see if this help fix the issue on
your platform?

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #4 from Suravee Suthikulpanit (suravee.suthikulpanit@amd.com) ---
Created attachment 295089
  --> https://bugzilla.kernel.org/attachment.cgi?id=295089&action=edit
(RFCv2) Proposed workaround for the IOMMU PMC issue

RFCv2 patch adds the logic to retry after 10msec wait for each retry loop. I
have founded that certain platform takes about 10msec for the power gating to
disable.

Please give this a try to see if this works better on your platform.

Thanks,
Suravee

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #5 from Suravee Suthikulpanit (suravee.suthikulpanit@amd.com) ---
Created attachment 295091
  --> https://bugzilla.kernel.org/attachment.cgi?id=295091&action=edit
(RFCv3) Proposed workaround for the IOMMU PMC issue

RFCv3 patch adds the logic to retry checking after 20msec wait for each retry
loop since I have founded that certain platform takes about 10msec for the
power gating to disable.

Please give this a try to see if this works better on your platform.

Thanks,
Suravee

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #6 from Neil (ledufff@hotmail.com) ---
Hello, Suravee,  I tried your patch in my Thinkpad E495. With RFCv1 and had no
success, I'm sorry I forgot to save a dmesg for that, but I am sure it was like
no patch was used, it was the same dmesg as before.

I had success with RFCv2,  and got the following:

[    0.606952] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.607117] pci 0000:00:00.2: can't derive routing for PCI INT A
[    0.607118] pci 0000:00:00.2: PCI INT A: not connected
[    0.607169] pci 0000:00:01.0: Adding to iommu group 0
[    0.607180] pci 0000:00:01.1: Adding to iommu group 1
[    0.607191] pci 0000:00:01.2: Adding to iommu group 2
[    0.607204] pci 0000:00:01.3: Adding to iommu group 3
[    0.607215] pci 0000:00:01.6: Adding to iommu group 4
[    0.607238] pci 0000:00:08.0: Adding to iommu group 5
[    0.607249] pci 0000:00:08.1: Adding to iommu group 6
[    0.607257] pci 0000:00:08.2: Adding to iommu group 5
[    0.607271] pci 0000:00:14.0: Adding to iommu group 7
[    0.607279] pci 0000:00:14.3: Adding to iommu group 7
[    0.607307] pci 0000:00:18.0: Adding to iommu group 8
[    0.607314] pci 0000:00:18.1: Adding to iommu group 8
[    0.607321] pci 0000:00:18.2: Adding to iommu group 8
[    0.607328] pci 0000:00:18.3: Adding to iommu group 8
[    0.607336] pci 0000:00:18.4: Adding to iommu group 8
[    0.607343] pci 0000:00:18.5: Adding to iommu group 8
[    0.607350] pci 0000:00:18.6: Adding to iommu group 8
[    0.607357] pci 0000:00:18.7: Adding to iommu group 8
[    0.607368] pci 0000:01:00.0: Adding to iommu group 9
[    0.607380] pci 0000:02:00.0: Adding to iommu group 10
[    0.607392] pci 0000:03:00.0: Adding to iommu group 11
[    0.607403] pci 0000:04:00.0: Adding to iommu group 12
[    0.607434] pci 0000:05:00.0: Adding to iommu group 13
[    0.607474] pci 0000:05:00.1: Adding to iommu group 14
[    0.607493] pci 0000:05:00.2: Adding to iommu group 14
[    0.607511] pci 0000:05:00.3: Adding to iommu group 14
[    0.607530] pci 0000:05:00.4: Adding to iommu group 14
[    0.607548] pci 0000:05:00.5: Adding to iommu group 14
[    0.607566] pci 0000:05:00.6: Adding to iommu group 14
[    0.607571] pci 0000:06:00.0: Adding to iommu group 5
[    0.609950] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.609951] pci 0000:00:00.2: AMD-Vi: Extended features (0x4f77ef22294ada):
[    0.609952]  PPR NX GT IA GA PC GA_vAPIC
[    0.609954] AMD-Vi: Interrupt remapping enabled
[    0.609954] AMD-Vi: Virtual APIC enabled
[    0.610822] AMD-Vi: Lazy IO/TLB flushing enabled
[    0.611644] amd_uncore: AMD NB counters detected
[    0.611647] amd_uncore: AMD LLC counters detected
[    0.611963] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4
counters/bank).

I will attach the full dmesg later.
Should I try the v3?  
Is there anything else apart from booting and checking dmesg that I can do to
test your patch?

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #7 from Neil (ledufff@hotmail.com) ---
Created attachment 295199
  --> https://bugzilla.kernel.org/attachment.cgi?id=295199&action=edit
Thinkpad E495 with RFCv2 patch.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #8 from Suravee Suthikulpanit (suravee.suthikulpanit@amd.com) ---
RFC v2 and v3 should be similar. I have modified the retry loop a bit in V3 to
be more efficient.

Patch has been submitted here (https://lkml.org/lkml/2021/2/8/486)

Thanks,
Suravee

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #9 from Paul Menzel (pmenzel+bugzilla.kernel.org@molgen.mpg.de) ---
Created attachment 295423
  --> https://bugzilla.kernel.org/attachment.cgi?id=295423&action=edit
Linux 5.12+ messages

Some days ago, Linus added the commit to this main branch [1].

Unfortunately, I am still seeing this error.

    [    0.000000] Linux version 5.11.0-10281-g19b4f3edd5c9 (root@a2ab663d937e)
(gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian)
2.35.1) #138 SMP Wed Feb 24 11:28:17 UTC 2021
    […]
    [    0.106422] smpboot: CPU0: AMD Ryzen 3 2200G with Radeon Vega Graphics
(family: 0x17, model: 0x11, stepping: 0x0)
    […]
    [    0.291257] pci 0000:00:00.2: AMD-Vi: Unable to read/write to IOMMU perf
counter.
    […]

@00oo00, is the error gone for you now (AMD Ryzen 5 2400G with Radeon Vega
Graphics (family: 0x17, model: 0x11, stepping: 0x0))?


[1]:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6778ff5b21bd8e78c8bd547fd66437cf2657fd9b

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

David Coe (david.coe@live.co.uk) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |david.coe@live.co.uk

--- Comment #10 from David Coe (david.coe@live.co.uk) ---
Hi Suravee! 

This bug has been bothering Ryzen users for some while now and work-arounds
adding various linux boot parameters have had patchy success. It's good to see
an informed and more forensic approach. Bravo!

I've been exercising your RFCv3 patch over the last few days (AMD Ryzen 5
2400G, Asus Prime B350-Plus, Ubuntu 20.10, Linux 5.8.0-43). With the original
20 msec delay I still get:

[    0.703538] Trying to unpack rootfs image as initramfs...
[    0.859160] Freeing initrd memory: 83748K
[    0.859256] pci 0000:00:00.2: AMD-Vi: Unable to read/write to IOMMU perf
counter.
[    0.859265] fbcon: Taking over console
[    0.859367] pci 0000:00:00.2: can't derive routing for PCI INT A
[    0.859368] pci 0000:00:00.2: PCI INT A: not connected
[    0.859413] pci 0000:00:01.0: Adding to iommu group 0
                                                      ...
                                                      ...
[    0.859959] pci 0000:0b:00.0: Adding to iommu group 12
[    0.863140] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.863142] pci 0000:00:00.2: AMD-Vi: Extended features (0x4f77ef22294ada):
[    0.863143]  PPR NX GT IA GA PC GA_vAPIC
[    0.863144] AMD-Vi: Interrupt remapping enabled
[    0.863145] AMD-Vi: Virtual APIC enabled
[    0.863375] AMD-Vi: Lazy IO/TLB flushing enabled
[    0.864261] amd_uncore: AMD NB counters detected
[    0.864273] amd_uncore: AMD LLC counters detected
[    0.864696] check: Scanning for low memory corruption every 60 seconds

Increasing the delay to msleep(40) gives:

[    0.212761] Trying to unpack rootfs image as initramfs...
[    0.366407] Freeing initrd memory: 83304K
[    0.509105] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.509199] pci 0000:00:00.2: can't derive routing for PCI INT A
[    0.509200] pci 0000:00:00.2: PCI INT A: not connected
[    0.509244] pci 0000:00:01.0: Adding to iommu group 0
                                                      ...
                                                      ...
[    0.509802] pci 0000:0b:00.0: Adding to iommu group 12
[    0.512997] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.512999] pci 0000:00:00.2: AMD-Vi: Extended features (0x4f77ef22294ada):
[    0.512999]  PPR NX GT IA GA PC GA_vAPIC
[    0.513001] AMD-Vi: Interrupt remapping enabled
[    0.513001] AMD-Vi: Virtual APIC enabled
[    0.513181] AMD-Vi: Lazy IO/TLB flushing enabled
[    0.514116] amd_uncore: AMD NB counters detected
[    0.514128] amd_uncore: AMD LLC counters detected
[    0.514482] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4
counters/bank).
[    0.514534] check: Scanning for low memory corruption every 60 seconds

If Linus is taking up the patch (hurrah), is 40 msec too large over the Ryzen
range? I would quite like the good Ubuntu people to merge your patch into their
current and upcoming distributions (I think SuSE are already doing so) but
don't want to jump-the-gun or tread-on-any-toes :-).

Over the next few days, I'll try my 2400G on delays between 20 and 40 msec and
maybe even test the patch on the still rather unstable Ubuntu 21.04. Any
guidance
much appreciated and, once again, many thanks.

David

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #11 from David Coe (david.coe@live.co.uk) ---
Good morning again!

I've rebuilt the Ubuntu 20.10 kernel 5.8.0-44 overnight with increased delays.
Grep'ing dmesg for IOMMU gives:

20 msec delay:

[    0.506947] pci 0000:00:00.2: AMD-Vi: Unable to read/write to IOMMU perf
counter.
[    0.510822] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.764160] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>

30 & 40 msec delay:

[    1.017676] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    1.021591] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.023053] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4
counters/bank).
[    1.345196] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>

At least on my 2400G platform, that suggests 20 msec is still too short. Of
course, YMMV :-).

Machines are grinding through the same trial on Ubuntu 21.04 alpha's 5.10.11
kernel and I'll post results in due course.

Best regards

David

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #12 from Paul Menzel (pmenzel+bugzilla.kernel.org@molgen.mpg.de) ---
@David, please increase the number of retries instead of increasing the delay.
Please also print out the number of retries.

Looking at the patch, delaying the boot by up to 100 ms, and no proper logging,
it leaves a lot to be desired.

@David, you can also test Alex’ patch *iommu/amd: Fix event counter
availability check* [1].


[1]:
https://lore.kernel.org/linux-iommu/20200529200738.1923-1-amonakov@ispras.ru/

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #14 from David Coe (david.coe@live.co.uk) ---
Good evening, all!

Two extra results from the Ryzen 2400G:

1. 5 x 25 msec loop on Ubuntu 21.04, kernel 5.10.11 and Suravee's original
patch.  Works fine!

2. 10x 20 msec loop on Ubuntu 20.10, kernel 5.8.18 with both Suravee and Alex's
patches. Also fine!

   As expected,
      AMD-Vi: IOMMU performance counters supported
   moves to after
      Adding to iommu group ...

This means that, for this CPU, the power-gating needs between 100 - 125 msec to
disable. A logging printf would have got me there faster (not overly fond of
kernel rebuilds) but the missing information is the maximum settling-time
needed across the full Ryzen range. Paul's suggestion of increasing the loop
count to 10 gives 200 msec maximum headroom. Surely must be enough?

Now, all we need is a work-around for the PCI INT A: not connected diagnostic
:-).

Many thanks good people. Keep safe.

David

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #17 from David Coe (david.coe@live.co.uk) ---
Again, good day!

Two re-builds using Alex's (one-line) patch, for kernels 5.8.18 and 5.10.11.
Both give on the Ryzen 2400G:

[    0.374138] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.374141] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.375457] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4
counters/bank).
[    0.626269] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>

I don't know how significant an extra 125 msec on boot time is but, on grounds
of simplicity and elegance, I'd vote for Alex :-). Are there any downsides?

Much appreciated all.

David

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #18 from Paul Menzel (pmenzel+bugzilla.kernel.org@molgen.mpg.de) ---
Even with ten tries in the loop, it still fails with the AMD Ryzen 3 2200G with
Radeon Vega Graphics (family: 0x17, model: 0x11, stepping: 0x0):

    [    0.401152] pci 0000:00:00.2: AMD-Vi: Unable to read/write to IOMMU perf
counter. (retry = 0)

```
diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c
index 9126efcbaf2c7..70c00ee5ff354 100644
--- a/drivers/iommu/amd/init.c
+++ b/drivers/iommu/amd/init.c
@@ -1746,7 +1746,7 @@ static void __init init_iommu_perf_ctr(struct amd_iommu
*iommu)

        /* Check if the performance counters can be written to */
        val = 0xabcd;
-       for (retry = 5; retry; retry--) {
+       for (retry = 10; retry; retry--) {
                if (iommu_pc_get_set_reg(iommu, 0, 0, 0, &val, true) ||
                    iommu_pc_get_set_reg(iommu, 0, 0, 0, &val2, false) ||
                    val2)
@@ -1764,7 +1764,7 @@ static void __init init_iommu_perf_ctr(struct amd_iommu
*iommu)
        if (val != val2)
                goto pc_false;

-       pci_info(pdev, "IOMMU performance counters supported\n");
+       pci_info(pdev, "IOMMU performance counters supported (retry = %i)\n",
retry);

        val = readl(iommu->mmio_base + MMIO_CNTR_CONF_OFFSET);
        iommu->max_banks = (u8) ((val >> 12) & 0x3f);
@@ -1773,7 +1773,7 @@ static void __init init_iommu_perf_ctr(struct amd_iommu
*iommu)
        return;

 pc_false:
-       pci_err(pdev, "Unable to read/write to IOMMU perf counter.\n");
+       pci_err(pdev, "Unable to read/write to IOMMU perf counter. (retry =
%i)\n", retry);
        amd_iommu_pc_present = false;
        return;
 }
```

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #19 from David Coe (david.coe@live.co.uk) ---
Hi all!

It's a knotty problem: the Ryzen has IOMMU performance counters, but can't tell
you they're working without a (120 msec 2400g, > 200 msec 2200G) delay (not
popular with Paul or Alex). It can assume they are working pro-tem but check
later in the boot sequence (Alex's patch but Suravee has some products that
don't like this). It can assume they aren't working (although they are) and
leave the user-space application to handle the loss of facility (not at all
keen on this myself).

System OEM's will want clean boot interfaces and road-warriors with high-end
laptops hate unnecessary delay when they lift their lids. However, day-to-day
users of virtualisation software like (for example) KDM will notice if IOMMU/Vi
hardware acceleration is present but crippled by the OS (how do the Windows
drivers handle the same issue?). The usual solution for this sort of dilemma, I
think, is to put in a BIOS switch or have a boot command-line option.

The current default is to turn slow-reporting IOMMU performance counters off.
This gives problems later with:

# perf stat -a -e amd_iommu/mem_trans_total/ test
event syntax error: 'amd_iommu/mem_trans_total/'
                     \___ Cannot find PMU `amd_iommu'. Missing kernel support?

After either Suravee or Alex's patch:

# perf stat -a -e amd_iommu/mem_trans_total/ test
Performance counter stats for 'system wide':
               897      amd_iommu/mem_trans_total/                              
       0.000990106 seconds time elapsed

Installation checks for KVM like systool -m kvm_amd -v and kvm-ok then give
expected results about modules and hardware acceleration. In principle, I'm
sorted for the 2400G but repeatedly recompiling the kernel is not a lovely
solution. Very, very appreciative of all attempts to get a generally-acceptable
patch into the distributions!

Best regards

David

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #20 from David Coe (david.coe@live.co.uk) ---
More results and some progress!!

A. Alex Hung (Ubuntu) has cherry-picked the kernel-commit [1} of Suravee's
version 3 patch and placed test-builds on [2]. It works for his Ryzen 2500U. 
Bravo for a quick take-up!

I've checked it out and (alas, as expected with the original 5 x 20 msec delay)
it is isn't enough for my Ryzen 2400G which needs ~120 msecs to respond :-(.

B. I've done a couple more rebuilds for the current kernels on Ubuntu 20.10 and
21.04 using Suravee's patch updated both with Pauls' retry-logging and with
increasing the maximum delay to 25 x 20 msec.

Grepp'ing dmesg for IOMMU (on my 2400g) gives:

[    1.018329] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
(retry = 19)
[    1.022185] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.023669] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4
counters/bank).
[    1.283218] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>

The retrys count backwards so that confirms the Ryzen 2500G still needs 120
msecs to respond.

My (very strong) proposal is that we increase Suravee's retry-count to
something large enough to cover the numerous entry-level Ryzen's out there.
Only the absolutely necessary boot-up delay is introduced (and checked, if
required, by Paul's logging statement). Most purchasers of Ryzen CPUs are then
covered! Only the dyed-in-the-wool speed-merchants have to make a choice
between boot-up time and IOMMU capability - at least until the underlying
problem can be corrected in firmware or silicon.

Thanks again, good people.

David

[1].
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6778ff5b21bd8e78c8bd547fd66437cf2657fd9b
[2]. https://people.canonical.com/~alexhung/LP1917203/

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #21 from David Coe (david.coe@live.co.uk) ---
Good morning, Suravee!

One (final?!) test of your patch (with increased retry count and logging) on
the latest git kernel (5.10.0-rc2) all on Ryzen 2400G. In summary:

 5.8.0-44   (Ubuntu 20.10) 6 trys 20 msecs
 5.10.0-14  (Ubuntu 21.04) 5 trys 20 msecs
 5.12.0-rc2 (raw git)      6 trys 20 msecs

AFAIK you are AMD's go-to guru for open-source people and an IOMMU expert. It's
your patch that is now both in the kernel commit and (I hope) in Ubuntu's to-do
list. Could I ask if you could patch-the-patch (if there is no esoteric
technical downside) to increase the number of retrys? Maybe if Paul would give
it another shot with his Ryzen 2200G with a larger maximum count and his
logging :-).

Best regards and our thanks

David

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

BubuXP (bubuxp@gmail.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bubuxp@gmail.com

--- Comment #22 from BubuXP (bubuxp@gmail.com) ---
Just a note, maybe useless.
I own a Huawei laptop, model KPL-W00D with Ryzen 2500U and it suffer this same
bug.

I cannot compile a kernel so, after installing Devuan testing, I installed also
the generic kernel located here:
https://kernel.ubuntu.com/~kernel-ppa/mainline/daily/2021-03-15/amd64/
that is a 5.12.0 RC3 and it includes the upstream patch for this bug.

The patch works but only on cold boots, after a simple reboot the bug turns
back again.

Maybe it's only a hardware problem, maybe a distro fault or misconfiguration, I
don't know, but I felt compelled to report this.

If needed I can give more detailed informations.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #23 from David Coe (david.coe@live.co.uk) ---
I would guess that the difference between warm and cold boot is due to your
Ryzen 2500U (like my 2440G) hovering between needing 5 and 6 tries of 20 msecs.
Without adding Paul's logging you can't tell!

If it would help anyone (and you trust a non-official kernel recompile) I have
placed 4 variants on my Box account. Both Suravee's original patch with 25 x 20
msec tries maximum and logged and Alex's very simple patch are applied to
Ubuntu's 5.8.0-44 (20.10) and 5.10.0-14 (21.04) kernels.

The version numbers are identical so that Ubuntu's own DEB's for linux-tools,
linux-headers and linux-libc-dev are compatible. My linux-image replaces their
own linux-image and linux-modules so take care that you can still boot from a
'known' kernel.

They all work for me with quite heavy use on KVM but YMMV :-). Mail me if you
need access.

Best regards to all and my thanks to Suravee, Alex and Paul!

David

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #24 from David Coe (david.coe@live.co.uk) ---
Just confirmed BubuXP's Ryzen 2500U result for Suravee's (logged) patch on my
2400G with the latest Ubuntu 5.8.0-45 kernel. Cold boot is marginally faster,
needing 5 x 20 msec, rather than 6 x 20 msecs for warm boot.

David

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

kitchm@tutanota.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |kitchm@tutanota.com

--- Comment #25 from kitchm@tutanota.com ---
I do not know if this will help your testing, but I am using MSI 450-A Pro and
AMD Ryzen 3 2200G with Radeon Vega Graphics.  When booting into Debian 10 I
have always seen the error message that states:
"Unable to write to  IOMMU perf counter"
It flashes at the top of the screen as the first item then goes away only to be
rewritten at the top of the boot process steps.  It only last a moment and then
everything boots normally.

However, I just updated to the latest BIOS version and it hung at that point
every time.  The BIOS version was Beta, so re-flashing to the older version
solved the problem.  It is now back to its normal behavior.

I'm not into manual kernel changes or configuring from source, so I'm sorry if
I can't help more.

Hope this info helps your research, and thanks for it.  Keep up the great work.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

Felice Tufo (info@felicetufo.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |info@felicetufo.com

--- Comment #27 from Felice Tufo (info@felicetufo.com) ---
Good morning,
I'm using the latest 5.11.7 kernel on an Aura15 Tuxedo notebook (based on Amd
Ryzen 4700U). As far as I can tell, the patch was merged on 5.11.7, but at
startup (cold boot) I read:

[    1.888186] pci 0000:00:00.2: AMD-Vi: Unable to read/write to IOMMU perf
counter.
[    1.888454] pci 0000:00:00.2: can't derive routing for PCI INT A
[    1.888457] pci 0000:00:00.2: PCI INT A: not connected
[    1.888510] pci 0000:00:01.0: Adding to iommu group 0
[    1.888537] pci 0000:00:01.3: Adding to iommu group 1
[    1.888565] pci 0000:00:02.0: Adding to iommu group 2
[    1.888583] pci 0000:00:02.1: Adding to iommu group 3
[    1.888600] pci 0000:00:02.2: Adding to iommu group 4
[    1.888616] pci 0000:00:02.4: Adding to iommu group 5
[    1.888644] pci 0000:00:08.0: Adding to iommu group 6
[    1.888663] pci 0000:00:08.1: Adding to iommu group 6
[    1.888689] pci 0000:00:14.0: Adding to iommu group 7
[    1.888707] pci 0000:00:14.3: Adding to iommu group 7
[    1.888764] pci 0000:00:18.0: Adding to iommu group 8
[    1.888779] pci 0000:00:18.1: Adding to iommu group 8
[    1.888795] pci 0000:00:18.2: Adding to iommu group 8
[    1.888810] pci 0000:00:18.3: Adding to iommu group 8
[    1.888825] pci 0000:00:18.4: Adding to iommu group 8
[    1.888840] pci 0000:00:18.5: Adding to iommu group 8
[    1.888855] pci 0000:00:18.6: Adding to iommu group 8
[    1.888869] pci 0000:00:18.7: Adding to iommu group 8
[    1.888886] pci 0000:01:00.0: Adding to iommu group 9
[    1.888906] pci 0000:02:00.0: Adding to iommu group 10
[    1.888923] pci 0000:03:00.0: Adding to iommu group 11
[    1.888940] pci 0000:04:00.0: Adding to iommu group 12
[    1.888959] pci 0000:05:00.0: Adding to iommu group 6
[    1.888968] pci 0000:05:00.1: Adding to iommu group 6
[    1.888977] pci 0000:05:00.2: Adding to iommu group 6
[    1.888985] pci 0000:05:00.3: Adding to iommu group 6
[    1.888993] pci 0000:05:00.4: Adding to iommu group 6
[    1.889000] pci 0000:05:00.5: Adding to iommu group 6
[    1.889008] pci 0000:05:00.6: Adding to iommu group 6
[    1.891539] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.891545] pci 0000:00:00.2: AMD-Vi: Extended features
(0x206d73ef22254ade):
[    1.891547]  PPR X2APIC NX GT IA GA PC GA_vAPIC
[    1.891552] AMD-Vi: Interrupt remapping enabled
[    1.891552] AMD-Vi: Virtual APIC enabled
[    1.891553] AMD-Vi: X2APIC enabled
[    1.891781] AMD-Vi: Lazy IO/TLB flushing enabled

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #28 from kitchm@tutanota.com ---
Thanks, Paul.  Of course you are correct.  A sloppy mistake on my part.

Linux version 4.19.0-14-amd64 (debian-kernel@lists.debian.org) (gcc version
8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.171-2 (2021-01-30)

[    0.000000] DMI: Micro-Star International Co., Ltd MS-7B86/B450-A PRO
(MS-7B86), BIOS A.D0 06/11/2020
[    0.132574] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
[    5.445162] input: HD-Audio Generic HDMI/DP,pcm=3 as
/devices/pci0000:00/0000:00:08.1/0000:27:00.1/sound/card0/input16
[    5.445197] input: HD-Audio Generic HDMI/DP,pcm=7 as
/devices/pci0000:00/0000:00:08.1/0000:27:00.1/sound/card0/input17

I tried to report it to MSI, but they have a terrible communication system. 
Aw, well....

Thanks again.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

Yigit (yigitates52h@outlook.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |yigitates52h@outlook.com

--- Comment #29 from Yigit (yigitates52h@outlook.com) ---
Not fixed for me either.

I'm using a Huawei MateBook D 14 AMD Ryzen 3500u. 5.11.7 kernel on Void Linux.
Same on cold boot.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #30 from Felice Tufo (info@felicetufo.com) ---
Hello, a little update to my previous message: same machine, same kernel, but
this time the patch works. So it seems that also on the same HW the timings are
not so predictable...

[    1.844903] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    1.845157] pci 0000:00:00.2: can't derive routing for PCI INT A
[    1.845160] pci 0000:00:00.2: PCI INT A: not connected
[    1.845213] pci 0000:00:01.0: Adding to iommu group 0
[    1.845240] pci 0000:00:01.3: Adding to iommu group 1
[    1.845267] pci 0000:00:02.0: Adding to iommu group 2
[    1.845284] pci 0000:00:02.1: Adding to iommu group 3
[    1.845301] pci 0000:00:02.2: Adding to iommu group 4
[    1.845318] pci 0000:00:02.4: Adding to iommu group 5
[    1.845349] pci 0000:00:08.0: Adding to iommu group 6
[    1.845368] pci 0000:00:08.1: Adding to iommu group 6
[    1.845394] pci 0000:00:14.0: Adding to iommu group 7
[    1.845411] pci 0000:00:14.3: Adding to iommu group 7
[    1.845469] pci 0000:00:18.0: Adding to iommu group 8
[    1.845484] pci 0000:00:18.1: Adding to iommu group 8
[    1.845500] pci 0000:00:18.2: Adding to iommu group 8
[    1.845515] pci 0000:00:18.3: Adding to iommu group 8
[    1.845530] pci 0000:00:18.4: Adding to iommu group 8
[    1.845545] pci 0000:00:18.5: Adding to iommu group 8
[    1.845559] pci 0000:00:18.6: Adding to iommu group 8
[    1.845574] pci 0000:00:18.7: Adding to iommu group 8
[    1.845592] pci 0000:01:00.0: Adding to iommu group 9
[    1.845611] pci 0000:02:00.0: Adding to iommu group 10
[    1.845629] pci 0000:03:00.0: Adding to iommu group 11
[    1.845646] pci 0000:04:00.0: Adding to iommu group 12
[    1.845665] pci 0000:05:00.0: Adding to iommu group 6
[    1.845674] pci 0000:05:00.1: Adding to iommu group 6
[    1.845683] pci 0000:05:00.2: Adding to iommu group 6
[    1.845691] pci 0000:05:00.3: Adding to iommu group 6
[    1.845699] pci 0000:05:00.4: Adding to iommu group 6
[    1.845706] pci 0000:05:00.5: Adding to iommu group 6
[    1.845714] pci 0000:05:00.6: Adding to iommu group 6
[    1.848251] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.848257] pci 0000:00:00.2: AMD-Vi: Extended features
(0x206d73ef22254ade):
[    1.848258]  PPR X2APIC NX GT IA GA PC GA_vAPIC
[    1.848263] AMD-Vi: Interrupt remapping enabled
[    1.848264] AMD-Vi: Virtual APIC enabled
[    1.848264] AMD-Vi: X2APIC enabled
[    1.848490] AMD-Vi: Lazy IO/TLB flushing enabled

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #31 from Neil (ledufff@hotmail.com) ---
Hah.  It's true,  cold booted and IOMMU was implemented, but after the second
reboot, I got:

[    0.857880] pci 0000:00:00.2: AMD-Vi: Unable to read/write to IOMMU perf
counter.

my system: linux 5.10.23-200.fc33.x86_64,  thinkpad e495 / AMD Ryzen 3500u

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #32 from David Coe (david.coe@live.co.uk) ---
I'm pretty sure this is because IOMMU (for some Ryzens) unlatches after 5 tries
on cold boot and 6 tries on warm. Suravee, alas, has set his patch at 5 x 20
msecs just on the 'dither' point!

Some time mid-next week I'll lay my sticky hands on a 4700U laptop. I have
images re-compiled for Ubuntu's kernel 5.8.0-45 (20.10) and 5.11.0-11 (21.04)
patched at 25 x 20 msecs and logged. Ditto with Alex's oh-so-simple patch.

If all goes well(!), I'll draw up a nice, comparative table of mine and
everybody else's data so we can draw some conclusions. If you scan the
Debian/Ubuntu kernel changelog, you'll find a long list of 'cherry-picked'
patches for both AMD and Intel for IOMMU. It is very much a work-in-progress
still!

Best regards and many thanks all for input.

David

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #33 from David Coe (david.coe@live.co.uk) ---
Good evening all!

Herewith a summary of our various results, including my latest for the Ryzen 7
4700U (Acer Swift 3).

20 msec wait with logged retrys

Ryzen     Kernel            Cold    Warm

4700U   5.11.0-11            6      1
3500U   5.11.7               5      6
2500U   5.8.0-45                    5
        5.12.0 RC3           5    > 5
2400G   5.11.0-11            6      6
        5.8.0-45             5      6
2200G       ?                     >10

Two points are clear:

1. there are differences between cold and warm boot, mostly marginal but marked
and very consistent with my 4700U.

2. the choice of 5 as the maximum retry number is (sorry, Suravee) unfortunate.
Mostly, it guarantees that all our Ryzens just fail the IOMMU write test!

Paul's experiences with the 2200G are the exception and I can appreciate his
unease about just increasing the number of retrys. I'm sure he, Suravee, Alex
and the AMD engineers will find an elegant solution to the problem :-).

In the meantime, could you formally patch-the-patch, Suravee? I'll pass on the
above data to Alex Hung, our "cherry-picker" at Ubuntu.

Best regards

David

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #35 from Paul Menzel (pmenzel+bugzilla.kernel.org@molgen.mpg.de) ---
Commit 994d6608 (iommu/amd: Remove performance counter pre-initialization test)
[1] fixes this, and was added to Linux 5.13.

[1]:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=994d6608efe4a4c8834bdc5014c86f4bc6aceea6

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #36 from Paul Menzel (pmenzel+bugzilla.kernel.org@molgen.mpg.de) ---
@00oo00, can you please close the issue?

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201753] AMD-Vi: Unable to write to IOMMU perf counter

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201753

--- Comment #37 from Neil (ledufff@hotmail.com) ---
Whoops,  I forgot to report as well that this is fixed for me.
Thinkpad E495 / Ryzen 3500u / debian 11 linux 5.10.46-5


[    0.366749] iommu: Default domain type: Translated 
[    0.805580] pci 0000:00:01.0: Adding to iommu group 0
[    0.805589] pci 0000:00:01.1: Adding to iommu group 1
[    0.805600] pci 0000:00:01.2: Adding to iommu group 2
[    0.805608] pci 0000:00:01.3: Adding to iommu group 3
[    0.805616] pci 0000:00:01.6: Adding to iommu group 4
[    0.805635] pci 0000:00:08.0: Adding to iommu group 5
[    0.805644] pci 0000:00:08.1: Adding to iommu group 6
[    0.805651] pci 0000:00:08.2: Adding to iommu group 5
[    0.805663] pci 0000:00:14.0: Adding to iommu group 7
[    0.805670] pci 0000:00:14.3: Adding to iommu group 7
[    0.805701] pci 0000:00:18.0: Adding to iommu group 8
[    0.805709] pci 0000:00:18.1: Adding to iommu group 8
[    0.805716] pci 0000:00:18.2: Adding to iommu group 8
[    0.805723] pci 0000:00:18.3: Adding to iommu group 8
[    0.805730] pci 0000:00:18.4: Adding to iommu group 8
[    0.805736] pci 0000:00:18.5: Adding to iommu group 8
[    0.805743] pci 0000:00:18.6: Adding to iommu group 8
[    0.805749] pci 0000:00:18.7: Adding to iommu group 8
[    0.805758] pci 0000:01:00.0: Adding to iommu group 9
[    0.805767] pci 0000:02:00.0: Adding to iommu group 10
[    0.805775] pci 0000:03:00.0: Adding to iommu group 11
[    0.805783] pci 0000:04:00.0: Adding to iommu group 12
[    0.805809] pci 0000:05:00.0: Adding to iommu group 13
[    0.805835] pci 0000:05:00.1: Adding to iommu group 14
[    0.805846] pci 0000:05:00.2: Adding to iommu group 14
[    0.805857] pci 0000:05:00.3: Adding to iommu group 14
[    0.805868] pci 0000:05:00.4: Adding to iommu group 14
[    0.805879] pci 0000:05:00.5: Adding to iommu group 14
[    0.805890] pci 0000:05:00.6: Adding to iommu group 14
[    0.805895] pci 0000:06:00.0: Adding to iommu group 5
[    0.810116] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4
counters/bank).

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.