From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754838AbcESRQh (ORCPT ); Thu, 19 May 2016 13:16:37 -0400 Received: from mga14.intel.com ([192.55.52.115]:4377 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754331AbcESRQf (ORCPT ); Thu, 19 May 2016 13:16:35 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.26,334,1459839600"; d="scan'208";a="106992018" Date: Thu, 19 May 2016 22:52:44 +0530 From: Vinod Koul To: "Jiang, Dave" Cc: Gavin Guo , "dmaengine@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "Williams, Dan J" Subject: Re: ioatdma(Intel(R) I/OAT DMA Engine init failed) Message-ID: <20160519172244.GE2735@localhost> References: <20160517100656.GT23734@localhost> <1463590132.5797.47.camel@intel.com> <112A412BB11A1242B37129D931BCE5347249E6B4@fmsmsx116.amr.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <112A412BB11A1242B37129D931BCE5347249E6B4@fmsmsx116.amr.corp.intel.com> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, May 19, 2016 at 08:19:30PM +0530, Jiang, Dave wrote: > > -----Original Message----- > > From: Gavin Guo [mailto:gavin.guo@canonical.com] > > Sent: Wednesday, May 18, 2016 8:19 PM > > To: Jiang, Dave > > Cc: Koul, Vinod ; dmaengine@vger.kernel.org; linux-kernel@vger.kernel.org; Williams, Dan J > > > > Subject: Re: ioatdma(Intel(R) I/OAT DMA Engine init failed) > > > > On Thu, May 19, 2016 at 12:49 AM, Jiang, Dave wrote: > > > On Wed, 2016-05-18 at 13:27 +0000, Gavin Guo wrote: > > >> On Tue, May 17, 2016 at 6:06 PM, Vinod Koul > > >> wrote: > > >> > > > >> > On Mon, May 16, 2016 at 06:08:20PM +0800, Gavin Guo wrote: > > >> > > > > >> > > The following error messages can be observed on the Intel > > >> > > Haswell-E > > >> > > chipset with v3.13 kernel. After the analysis, I found there is > > >> > > no > > >> > > difference in the logic of these error messages in the current > > >> > > upstream kernel. I also searched the git log and can't find any > > >> > > commit > > >> > > which is fix to the error(correct me if I am wrong). The > > >> > > following is > > >> > > the detail, and I'll really appreciate if there is any comment. > > >> > > :) > > >> > 3.13 is ancient, can you check this on latest kernel > > >> Thank you for the comment. It's running on the production system. > > >> However, > > >> I'll try to figure out if it's possible to test the latest kernel. > > > > > > I wonder if you don't have the extended PCI config space access enabled > > > in your kernel config. > > > > Really thanks for your advice. :) > > > > I searched the internet about the extended PCI config space and found > > the link: > > > > [Patch v2] Make PCI extended config space (MMCONFIG) a driver opt-in > > http://lwn.net/Articles/263288/ > > Can you try calling pci_enable_ext_config() in the PCI probe for your kernel? I just haven't seen this issue in the latest kernel. Do we need that to be called explicitly by driver, should that not be enabled by default? > > > > > And I checked the config and found the CONFIG_PCI_MMCONFIG=y. The > > following string also can be observed in the dmesg: > > > > [ 1.419853] PCI: MMCONFIG for domain 0000 [bus 00-ff] at > > [mem0x80000000-0x8fffffff] (base 0x80000000) > > [ 1.419855] PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820 > > > > It seems the extended PCI config space is enabled. If there is > > anything missed? > > > > > > > >> > > >> > > > >> > > > >> > > > > >> > > > > >> > > ioatdma 0000:00:04.0: channel error register unreachable > > >> > > ioatdma 0000:00:04.0: channel enumeration error > > >> > > ioatdma 0000:00:04.0: Intel(R) I/OAT DMA Engine init failed > > >> > > ioatdma 0000:00:04.1: channel error register unreachable > > >> > > ioatdma 0000:00:04.1: channel enumeration error > > >> > > ioatdma 0000:00:04.1: Intel(R) I/OAT DMA Engine init failed > > >> > > ... > > >> > > ioatdma 0000:00:04.7: channel error register unreachable > > >> > > ioatdma 0000:00:04.7: channel enumeration error > > >> > > ioatdma 0000:00:04.7: Intel(R) I/OAT DMA Engine init failed > > >> > > mei_me 0000:00:16.0: initialization failed. > > >> > > > > >> > > There are 8 I/OAT DMA controllers on the Haswell-E chipset: > > >> > > 8086:2f20 ~ 8086:2f27 > > >> > > 80:04.0 System peripheral: Intel Corporation Haswell-E DMA > > >> > > Channel 0 (rev 02) > > >> > > 80:04.1 System peripheral: Intel Corporation Haswell-E DMA > > >> > > Channel 1 (rev 02) > > >> > > 80:04.2 System peripheral: Intel Corporation Haswell-E DMA > > >> > > Channel 2 (rev 02) > > >> > > 80:04.3 System peripheral: Intel Corporation Haswell-E DMA > > >> > > Channel 3 (rev 02) > > >> > > 80:04.4 System peripheral: Intel Corporation Haswell-E DMA > > >> > > Channel 4 (rev 02) > > >> > > 80:04.5 System peripheral: Intel Corporation Haswell-E DMA > > >> > > Channel 5 (rev 02) > > >> > > 80:04.6 System peripheral: Intel Corporation Haswell-E DMA > > >> > > Channel 6 (rev 02) > > >> > > 80:04.7 System peripheral: Intel Corporation Haswell-E DMA > > >> > > Channel 7 (rev 02) > > >> > > > > >> > > Analysis: > > >> > > The bug happens when the driver is resetting DMA controller, this > > >> > > is > > >> > > the sequence: The function, ioat_pci_probe, is called when the > > >> > > DMA > > >> > > controller is detected by the PCI bus. Then, > > >> > > ioat3_dma_probe -> ioat_probe -> ioat2_enumerate_channels -> > > >> > > ioat3_reset_hw. The following code can be found in the > > >> > > ioat3_reset_hw: > > >> > > > > >> > > drivers/dma/ioat/dma_v3.c: > > >> > > chanerr = readl(chan->reg_base + IOAT_CHANERR_OFFSET); > > >> > > writel(chanerr, chan->reg_base + IOAT_CHANERR_OFFSET); > > >> > > ... > > >> > > err = pci_read_config_dword(pdev, > > >> > > IOAT_PCI_CHANERR_INT_OFFSET, &chanerr); > > >> > > if (err) { > > >> > > dev_err(&pdev->dev, > > >> > > "channel error register unreachable\n"); > > >> > > return err; > > >> > > } > > >> > > > > >> > > Obviously, there are something wrong in the channel error > > >> > > register > > >> > > reset process. Then all the way back to ioat_probe(). Because the > > >> > > error happens, the dma->chancnt will be set to 0: > > >> > > > > >> > > drivers/dma/ioat/dma.c: > > >> > > if (!dma->chancnt) { > > >> > > dev_err(dev, "channel enumeration error\n"); > > >> > > goto err_setup_interrupts; > > >> > > } > > >> > > > > >> > > Finally back to ioat_pci_probe: > > >> > > > > >> > > drivers/dma/ioat/pci.c: > > >> > > err = ioat3_dma_probe(device, ioat_dca_enabled); > > >> > > else > > >> > > return -ENODEV; > > >> > > > > >> > > if (err) { > > >> > > dev_err(dev, "Intel(R) I/OAT DMA Engine init > > >> > > failed\n"); > > >> > > return -ENODEV; > > >> > -- > > >> > ~Vinod -- ~Vinod