From: Takao Indoh <indou.takao@jp.fujitsu.com> To: ddutile@redhat.com Cc: kexec@lists.infradead.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, bhelgaas@google.com, vgoyal@redhat.com, hbabu@us.ibm.com, ishii.hironobu@jp.fujitsu.com, martin.wilck@ts.fujitsu.com Subject: Re: [RFC][PATCH] Reset PCIe devices to address DMA problem on kdump with iommu Date: Mon, 06 Aug 2012 13:45:02 +0900 [thread overview] Message-ID: <501F4BCE.4060702@jp.fujitsu.com> (raw) In-Reply-To: <501F4387.3010404@redhat.com> Hi Don, (2012/08/06 13:09), Don Dutile wrote: > On 08/03/2012 07:24 AM, Takao Indoh wrote: >> Hi all, >> >> This patch adds kernel parameter "reset_pcie_devices" which resets PCIe >> devices at boot time to address DMA problem on kdump with iommu. When >> this parameter is specified, a hot reset is triggered on each PCIe root >> port and downstream port to reset its downstream endpoint. >> >> Background: >> A kdump problem about DMA has been discussed for a long time. That is, >> when a kernel is switched to the kdump kernel DMA derived from first >> kernel affects second kernel. Recently this problem surfaces when iommu >> is used for PCI passthrough on KVM guest. In the case of the machine I >> use, when intel_iommu=on is specified, DMAR error is detected in kdump >> kernel and PCI SERR is also detected. Finally kdump fails because some >> devices does not work correctly. >> >> The root cause is that ongoing DMA from first kernel causes DMAR fault >> because page table of DMAR is initialized while kdump kernel is booting >> up. Therefore to address this problem DMA needs to be stopped before DMAR >> is initialized at kdump kernel boot time. By this patch, PCIe devices >> are reset by hot reset and its DMA is stopped when reset_pcie_devices is >> specified. One problem of this solution is that VGA is reset and the >> monitor blacks out when the link between the port and VGA controller was >> reset. So this patch does not reset the port whose child endpoint is VGA >> device. >> >> Any comments would be appreciated. >> >> Signed-off-by: Takao Indoh<indou.takao@jp.fujitsu.com> >> --- > Have you considered something less disruptive such as clearing the > Master Enable in each endpoint's PCI Command Register ? > That should prevent DMA transactions from endpoints during the kdump and > kexec, and when the driver for the device gets reconfigured, > Master Enable will be set back on, but after the driver has had the > opportunity to do a device-specific reset. Yes, that is what I tried first. First of all I found this thread. [PATCH] Disable Bus Master on PCI device shutdown https://lkml.org/lkml/2012/6/6/278 original patch: http://marc.info/?l=kexec&m=133546519231295&w=2 And I thought ongoing DMA could be stopped by disabling it on kdump kernel boot, but DMAR error is still found after adding quirk below. In this thread, AER link reset was also suggested, so I got the idea to reset root port and downstream port. Thanks, Takao Indoh diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index 5155317..0e8b41a 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -31,6 +31,22 @@ #include <asm/dma.h> /* isa_dma_bridge_buggy */ #include "pci.h" +static void test_pci_device_reset(struct pci_dev *dev) +{ + u16 pci_command; + + if (!reset_devices) + return; + + pci_read_config_word(dev, PCI_COMMAND, &pci_command); + if (pci_command & PCI_COMMAND_MASTER) { + dev_info(&dev->dev, "DEBUG: Disable Bus Master bit\n"); + pci_command &= ~PCI_COMMAND_MASTER; + pci_write_config_word(dev, PCI_COMMAND, pci_command); + } +} +DECLARE_PCI_FIXUP_EARLY(PCI_ANY_ID, PCI_ANY_ID, test_pci_device_reset); + /* * Decoding should be disabled for a PCI device during BAR sizing to avoid * conflict. But doing so may cause problems on host bridge and perhaps other
WARNING: multiple messages have this Message-ID (diff)
From: Takao Indoh <indou.takao@jp.fujitsu.com> To: ddutile@redhat.com Cc: martin.wilck@ts.fujitsu.com, linux-pci@vger.kernel.org, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, hbabu@us.ibm.com, ishii.hironobu@jp.fujitsu.com, bhelgaas@google.com, vgoyal@redhat.com Subject: Re: [RFC][PATCH] Reset PCIe devices to address DMA problem on kdump with iommu Date: Mon, 06 Aug 2012 13:45:02 +0900 [thread overview] Message-ID: <501F4BCE.4060702@jp.fujitsu.com> (raw) In-Reply-To: <501F4387.3010404@redhat.com> Hi Don, (2012/08/06 13:09), Don Dutile wrote: > On 08/03/2012 07:24 AM, Takao Indoh wrote: >> Hi all, >> >> This patch adds kernel parameter "reset_pcie_devices" which resets PCIe >> devices at boot time to address DMA problem on kdump with iommu. When >> this parameter is specified, a hot reset is triggered on each PCIe root >> port and downstream port to reset its downstream endpoint. >> >> Background: >> A kdump problem about DMA has been discussed for a long time. That is, >> when a kernel is switched to the kdump kernel DMA derived from first >> kernel affects second kernel. Recently this problem surfaces when iommu >> is used for PCI passthrough on KVM guest. In the case of the machine I >> use, when intel_iommu=on is specified, DMAR error is detected in kdump >> kernel and PCI SERR is also detected. Finally kdump fails because some >> devices does not work correctly. >> >> The root cause is that ongoing DMA from first kernel causes DMAR fault >> because page table of DMAR is initialized while kdump kernel is booting >> up. Therefore to address this problem DMA needs to be stopped before DMAR >> is initialized at kdump kernel boot time. By this patch, PCIe devices >> are reset by hot reset and its DMA is stopped when reset_pcie_devices is >> specified. One problem of this solution is that VGA is reset and the >> monitor blacks out when the link between the port and VGA controller was >> reset. So this patch does not reset the port whose child endpoint is VGA >> device. >> >> Any comments would be appreciated. >> >> Signed-off-by: Takao Indoh<indou.takao@jp.fujitsu.com> >> --- > Have you considered something less disruptive such as clearing the > Master Enable in each endpoint's PCI Command Register ? > That should prevent DMA transactions from endpoints during the kdump and > kexec, and when the driver for the device gets reconfigured, > Master Enable will be set back on, but after the driver has had the > opportunity to do a device-specific reset. Yes, that is what I tried first. First of all I found this thread. [PATCH] Disable Bus Master on PCI device shutdown https://lkml.org/lkml/2012/6/6/278 original patch: http://marc.info/?l=kexec&m=133546519231295&w=2 And I thought ongoing DMA could be stopped by disabling it on kdump kernel boot, but DMAR error is still found after adding quirk below. In this thread, AER link reset was also suggested, so I got the idea to reset root port and downstream port. Thanks, Takao Indoh diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index 5155317..0e8b41a 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -31,6 +31,22 @@ #include <asm/dma.h> /* isa_dma_bridge_buggy */ #include "pci.h" +static void test_pci_device_reset(struct pci_dev *dev) +{ + u16 pci_command; + + if (!reset_devices) + return; + + pci_read_config_word(dev, PCI_COMMAND, &pci_command); + if (pci_command & PCI_COMMAND_MASTER) { + dev_info(&dev->dev, "DEBUG: Disable Bus Master bit\n"); + pci_command &= ~PCI_COMMAND_MASTER; + pci_write_config_word(dev, PCI_COMMAND, pci_command); + } +} +DECLARE_PCI_FIXUP_EARLY(PCI_ANY_ID, PCI_ANY_ID, test_pci_device_reset); + /* * Decoding should be disabled for a PCI device during BAR sizing to avoid * conflict. But doing so may cause problems on host bridge and perhaps other _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec
next prev parent reply other threads:[~2012-08-06 4:49 UTC|newest] Thread overview: 42+ messages / expand[flat|nested] mbox.gz Atom feed top 2012-08-03 11:24 [RFC][PATCH] Reset PCIe devices to address DMA problem on kdump with iommu Takao Indoh 2012-08-03 11:24 ` Takao Indoh 2012-08-03 11:46 ` Vivek Goyal 2012-08-03 11:46 ` Vivek Goyal 2012-08-06 4:30 ` Takao Indoh 2012-08-06 4:30 ` Takao Indoh 2012-08-06 20:39 ` Vivek Goyal 2012-08-06 20:39 ` Vivek Goyal 2012-08-07 9:02 ` Takao Indoh 2012-08-07 9:02 ` Takao Indoh 2012-09-05 11:09 ` Takao Indoh 2012-09-05 11:09 ` Takao Indoh 2012-09-10 2:34 ` Kaneshige, Kenji 2012-09-10 2:34 ` Kaneshige, Kenji 2012-09-10 6:35 ` Takao Indoh 2012-09-10 6:35 ` Takao Indoh 2012-09-11 11:52 ` Kaneshige, Kenji 2012-09-11 11:52 ` Kaneshige, Kenji 2012-09-10 14:36 ` Vivek Goyal 2012-09-10 14:36 ` Vivek Goyal 2012-09-11 10:32 ` Takao Indoh 2012-09-11 10:32 ` Takao Indoh 2012-09-11 14:43 ` Vivek Goyal 2012-09-11 14:43 ` Vivek Goyal 2012-09-12 9:00 ` Takao Indoh 2012-09-12 9:00 ` Takao Indoh 2012-09-14 15:48 ` Vivek Goyal 2012-09-14 15:48 ` Vivek Goyal 2012-09-24 11:22 ` Takao Indoh 2012-09-24 11:22 ` Takao Indoh 2012-09-14 20:03 ` Konrad Rzeszutek Wilk 2012-09-14 20:03 ` Konrad Rzeszutek Wilk 2012-09-19 1:52 ` Takao Indoh 2012-09-19 1:52 ` Takao Indoh 2012-09-21 17:57 ` Don Dutile 2012-09-21 17:57 ` Don Dutile 2012-09-24 11:12 ` Takao Indoh 2012-09-24 11:12 ` Takao Indoh 2012-08-06 4:09 ` Don Dutile 2012-08-06 4:09 ` Don Dutile 2012-08-06 4:45 ` Takao Indoh [this message] 2012-08-06 4:45 ` Takao Indoh
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=501F4BCE.4060702@jp.fujitsu.com \ --to=indou.takao@jp.fujitsu.com \ --cc=bhelgaas@google.com \ --cc=ddutile@redhat.com \ --cc=hbabu@us.ibm.com \ --cc=ishii.hironobu@jp.fujitsu.com \ --cc=kexec@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-pci@vger.kernel.org \ --cc=martin.wilck@ts.fujitsu.com \ --cc=vgoyal@redhat.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.