From: Maxim Levitsky <mlevitsk@redhat.com> To: linux-kernel@vger.kernel.org Cc: David Woodhouse <dwmw@amazon.co.uk>, Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>, Joerg Roedel <joro@8bytes.org>, Thomas Gleixner <tglx@linutronix.de>, "open list:AMD IOMMU (AMD-VI)" <iommu@lists.linux-foundation.org>, Will Deacon <will@kernel.org> Subject: Re: [PATCH 0/5] iommu/amd: fixes for suspend/resume Date: Fri, 10 Dec 2021 10:00:26 +0200 [thread overview] Message-ID: <5d7d591abdbfa8a5cb327a59b94ff7a6c1cfdee6.camel@redhat.com> (raw) In-Reply-To: <44f3f76cb14bafee2d81a615f258a4fb4be93fe1.camel@redhat.com> On Thu, 2021-12-02 at 01:08 +0200, Maxim Levitsky wrote: > On Tue, 2021-11-23 at 18:10 +0200, Maxim Levitsky wrote: > > As I sadly found out, a s3 cycle makes the AMD's iommu stop sending interrupts > > until the system is rebooted. > > > > I only noticed it now because otherwise the IOMMU works, and these interrupts > > are only used for errors and for GA log which I tend not to use by > > making my VMs do mwait/pause/etc in guest (cpu-pm=on). > > > > There are two issues here that prevent interrupts from being generated after > > s3 cycle: > > > > 1. GA log base address was not restored after resume, and was all zeroed > > after resume (by BIOS or such). > > > > In theory if BIOS writes some junk to it, that can even cause a memory corruption. > > Patch 2 fixes that. > > > > 2. INTX (aka x2apic mode) settings were not restored after resume. > > That mode is used regardless if the host uses/supports x2apic, but rather when > > the IOMMU supports it, and mine does. > > Patches 3-4 fix that. > > > > Note that there is still one slight (userspace) bug remaining: > > During suspend all but the boot CPU are offlined and then after resume > > are onlined again. > > > > The offlining moves all non-affinity managed interrupts to CPU0, and > > later when all other CPUs are onlined, there is nothing in the kernel > > to spread back the interrupts over the cores. > > > > The userspace 'irqbalance' daemon does fix this but it seems to ignore > > the IOMMU interrupts in INTX mode since they are not attached to any > > PCI device, and thus they remain on CPU0 after a s3 cycle, > > which is suboptimal when the system has multiple IOMMUs > > (mine has 4 of them). > > > > Setting the IRQ affinity manually via /proc/irq/ does work. > > > > This was tested on my 3970X with both INTX and regular MSI mode (later was enabled > > by patching out INTX detection), by running a guest with AVIC enabled and with > > a PCI assigned device (network card), and observing interrupts from > > IOMMU while guest is mostly idle. > > > > This was also tested on my AMD laptop with 4650U (which has the same issue) > > (I tested only INTX mode) > > > > Patch 1 is a small refactoring to remove an unused struct field. > > > > Best regards, > > Maxim Levitsky > > > > Maxim Levitsky (5): > > iommu/amd: restore GA log/tail pointer on host resume > > iommu/amd: x2apic mode: re-enable after resume > > iommu/amd: x2apic mode: setup the INTX registers on mask/unmask > > iommu/amd: x2apic mode: mask/unmask interrupts on suspend/resume > > iommu/amd: remove useless irq affinity notifier > > > > drivers/iommu/amd/amd_iommu_types.h | 2 - > > drivers/iommu/amd/init.c | 107 +++++++++++++++------------- > > 2 files changed, 58 insertions(+), 51 deletions(-) > > > > -- > > 2.26.3 > > > > > > Polite ping on these patches. Another very polite ping on these patches :) Best regards, Maxim Levitsky > Best regards, > Maxim Levitsky
WARNING: multiple messages have this Message-ID (diff)
From: Maxim Levitsky <mlevitsk@redhat.com> To: linux-kernel@vger.kernel.org Cc: "open list:AMD IOMMU \(AMD-VI\)" <iommu@lists.linux-foundation.org>, Thomas Gleixner <tglx@linutronix.de>, Will Deacon <will@kernel.org>, David Woodhouse <dwmw@amazon.co.uk> Subject: Re: [PATCH 0/5] iommu/amd: fixes for suspend/resume Date: Fri, 10 Dec 2021 10:00:26 +0200 [thread overview] Message-ID: <5d7d591abdbfa8a5cb327a59b94ff7a6c1cfdee6.camel@redhat.com> (raw) In-Reply-To: <44f3f76cb14bafee2d81a615f258a4fb4be93fe1.camel@redhat.com> On Thu, 2021-12-02 at 01:08 +0200, Maxim Levitsky wrote: > On Tue, 2021-11-23 at 18:10 +0200, Maxim Levitsky wrote: > > As I sadly found out, a s3 cycle makes the AMD's iommu stop sending interrupts > > until the system is rebooted. > > > > I only noticed it now because otherwise the IOMMU works, and these interrupts > > are only used for errors and for GA log which I tend not to use by > > making my VMs do mwait/pause/etc in guest (cpu-pm=on). > > > > There are two issues here that prevent interrupts from being generated after > > s3 cycle: > > > > 1. GA log base address was not restored after resume, and was all zeroed > > after resume (by BIOS or such). > > > > In theory if BIOS writes some junk to it, that can even cause a memory corruption. > > Patch 2 fixes that. > > > > 2. INTX (aka x2apic mode) settings were not restored after resume. > > That mode is used regardless if the host uses/supports x2apic, but rather when > > the IOMMU supports it, and mine does. > > Patches 3-4 fix that. > > > > Note that there is still one slight (userspace) bug remaining: > > During suspend all but the boot CPU are offlined and then after resume > > are onlined again. > > > > The offlining moves all non-affinity managed interrupts to CPU0, and > > later when all other CPUs are onlined, there is nothing in the kernel > > to spread back the interrupts over the cores. > > > > The userspace 'irqbalance' daemon does fix this but it seems to ignore > > the IOMMU interrupts in INTX mode since they are not attached to any > > PCI device, and thus they remain on CPU0 after a s3 cycle, > > which is suboptimal when the system has multiple IOMMUs > > (mine has 4 of them). > > > > Setting the IRQ affinity manually via /proc/irq/ does work. > > > > This was tested on my 3970X with both INTX and regular MSI mode (later was enabled > > by patching out INTX detection), by running a guest with AVIC enabled and with > > a PCI assigned device (network card), and observing interrupts from > > IOMMU while guest is mostly idle. > > > > This was also tested on my AMD laptop with 4650U (which has the same issue) > > (I tested only INTX mode) > > > > Patch 1 is a small refactoring to remove an unused struct field. > > > > Best regards, > > Maxim Levitsky > > > > Maxim Levitsky (5): > > iommu/amd: restore GA log/tail pointer on host resume > > iommu/amd: x2apic mode: re-enable after resume > > iommu/amd: x2apic mode: setup the INTX registers on mask/unmask > > iommu/amd: x2apic mode: mask/unmask interrupts on suspend/resume > > iommu/amd: remove useless irq affinity notifier > > > > drivers/iommu/amd/amd_iommu_types.h | 2 - > > drivers/iommu/amd/init.c | 107 +++++++++++++++------------- > > 2 files changed, 58 insertions(+), 51 deletions(-) > > > > -- > > 2.26.3 > > > > > > Polite ping on these patches. Another very polite ping on these patches :) Best regards, Maxim Levitsky > Best regards, > Maxim Levitsky _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
next prev parent reply other threads:[~2021-12-10 8:00 UTC|newest] Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-11-23 16:10 [PATCH 0/5] iommu/amd: fixes for suspend/resume Maxim Levitsky 2021-11-23 16:10 ` Maxim Levitsky 2021-11-23 16:10 ` [PATCH 1/5] iommu/amd: restore GA log/tail pointer on host resume Maxim Levitsky 2021-11-23 16:10 ` Maxim Levitsky 2021-11-23 16:10 ` [PATCH 2/5] iommu/amd: x2apic mode: re-enable after resume Maxim Levitsky 2021-11-23 16:10 ` Maxim Levitsky 2021-11-23 16:10 ` [PATCH 3/5] iommu/amd: x2apic mode: setup the INTX registers on mask/unmask Maxim Levitsky 2021-11-23 16:10 ` Maxim Levitsky 2021-11-23 16:10 ` [PATCH 4/5] iommu/amd: x2apic mode: mask/unmask interrupts on suspend/resume Maxim Levitsky 2021-11-23 16:10 ` Maxim Levitsky 2021-11-23 16:10 ` [PATCH 5/5] iommu/amd: remove useless irq affinity notifier Maxim Levitsky 2021-11-23 16:10 ` Maxim Levitsky 2021-12-01 23:08 ` [PATCH 0/5] iommu/amd: fixes for suspend/resume Maxim Levitsky 2021-12-01 23:08 ` Maxim Levitsky 2021-12-10 8:00 ` Maxim Levitsky [this message] 2021-12-10 8:00 ` Maxim Levitsky 2021-12-06 14:01 ` Joerg Roedel 2021-12-06 14:01 ` Joerg Roedel 2021-12-17 8:31 ` Joerg Roedel 2021-12-17 8:31 ` Joerg Roedel 2022-01-25 15:08 ` Mike Lothian 2022-01-25 15:08 ` Mike Lothian 2022-01-25 19:26 ` Maxim Levitsky 2022-01-25 19:26 ` Maxim Levitsky 2022-01-25 23:25 ` Mike Lothian 2022-01-25 23:25 ` Mike Lothian 2022-01-26 7:34 ` Maxim Levitsky 2022-01-26 7:34 ` Maxim Levitsky 2022-01-26 9:54 ` Mike Lothian 2022-01-26 9:54 ` Mike Lothian 2022-01-26 10:12 ` Maxim Levitsky 2022-01-26 10:12 ` Maxim Levitsky 2022-01-27 0:39 ` Mike Lothian 2022-01-27 0:39 ` Mike Lothian 2022-01-27 10:22 ` Maxim Levitsky 2022-01-27 10:22 ` Maxim Levitsky 2022-01-27 10:50 ` Mike Lothian 2022-01-27 10:50 ` Mike Lothian
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=5d7d591abdbfa8a5cb327a59b94ff7a6c1cfdee6.camel@redhat.com \ --to=mlevitsk@redhat.com \ --cc=dwmw@amazon.co.uk \ --cc=iommu@lists.linux-foundation.org \ --cc=joro@8bytes.org \ --cc=linux-kernel@vger.kernel.org \ --cc=suravee.suthikulpanit@amd.com \ --cc=tglx@linutronix.de \ --cc=will@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.