From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0691C31E5B for ; Tue, 18 Jun 2019 15:15:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C0F922147A for ; Tue, 18 Jun 2019 15:15:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729038AbfFRPP5 (ORCPT ); Tue, 18 Jun 2019 11:15:57 -0400 Received: from mx2.suse.de ([195.135.220.15]:46344 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728572AbfFRPP4 (ORCPT ); Tue, 18 Jun 2019 11:15:56 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 8E7DBAE5A; Tue, 18 Jun 2019 15:15:53 +0000 (UTC) Subject: Re: [PATCH] PCI: Add Intel remapped NVMe device support To: Daniel Drake Cc: Jens Axboe , Sagi Grimberg , Linux PCI , Linux Upstreaming Team , Keith Busch , linux-ide@vger.kernel.org, linux-nvme , Keith Busch , Bjorn Helgaas , Christoph Hellwig References: <20190610074456.2761-1-drake@endlessm.com> <20190613085402.GC13442@lst.de> <06c38b3e-603b-5bae-4959-9965ab40db62@suse.de> From: Hannes Reinecke Openpgp: preference=signencrypt Autocrypt: addr=hare@suse.de; prefer-encrypt=mutual; keydata= mQINBE6KyREBEACwRN6XKClPtxPiABx5GW+Yr1snfhjzExxkTYaINHsWHlsLg13kiemsS6o7 qrc+XP8FmhcnCOts9e2jxZxtmpB652lxRB9jZE40mcSLvYLM7S6aH0WXKn8bOqpqOGJiY2bc 6qz6rJuqkOx3YNuUgiAxjuoYauEl8dg4bzex3KGkGRuxzRlC8APjHlwmsr+ETxOLBfUoRNuE b4nUtaseMPkNDwM4L9+n9cxpGbdwX0XwKFhlQMbG3rWA3YqQYWj1erKIPpgpfM64hwsdk9zZ QO1krgfULH4poPQFpl2+yVeEMXtsSou915jn/51rBelXeLq+cjuK5+B/JZUXPnNDoxOG3j3V VSZxkxLJ8RO1YamqZZbVP6jhDQ/bLcAI3EfjVbxhw9KWrh8MxTcmyJPn3QMMEp3wpVX9nSOQ tzG72Up/Py67VQe0x8fqmu7R4MmddSbyqgHrab/Nu+ak6g2RRn3QHXAQ7PQUq55BDtj85hd9 W2iBiROhkZ/R+Q14cJkWhzaThN1sZ1zsfBNW0Im8OVn/J8bQUaS0a/NhpXJWv6J1ttkX3S0c QUratRfX4D1viAwNgoS0Joq7xIQD+CfJTax7pPn9rT////hSqJYUoMXkEz5IcO+hptCH1HF3 qz77aA5njEBQrDRlslUBkCZ5P+QvZgJDy0C3xRGdg6ZVXEXJOQARAQABtCpIYW5uZXMgUmVp bmVja2UgKFN1U0UgTGFicykgPGhhcmVAc3VzZS5kZT6JAkEEEwECACsCGwMFCRLMAwAGCwkI BwMCBhUIAgkKCwQWAgMBAh4BAheABQJOisquAhkBAAoJEGz4yi9OyKjPOHoQAJLeLvr6JNHx GPcHXaJLHQiinz2QP0/wtsT8+hE26dLzxb7hgxLafj9XlAXOG3FhGd+ySlQ5wSbbjdxNjgsq FIjqQ88/Lk1NfnqG5aUTPmhEF+PzkPogEV7Pm5Q17ap22VK623MPaltEba+ly6/pGOODbKBH ak3gqa7Gro5YCQzNU0QVtMpWyeGF7xQK76DY/atvAtuVPBJHER+RPIF7iv5J3/GFIfdrM+wS BubFVDOibgM7UBnpa7aohZ9RgPkzJpzECsbmbttxYaiv8+EOwark4VjvOne8dRaj50qeyJH6 HLpBXZDJH5ZcYJPMgunghSqghgfuUsd5fHmjFr3hDb5EoqAfgiRMSDom7wLZ9TGtT6viDldv hfWaIOD5UhpNYxfNgH6Y102gtMmN4o2P6g3UbZK1diH13s9DA5vI2mO2krGz2c5BOBmcctE5 iS+JWiCizOqia5Op+B/tUNye/YIXSC4oMR++Fgt30OEafB8twxydMAE3HmY+foawCpGq06yM vAguLzvm7f6wAPesDAO9vxRNC5y7JeN4Kytl561ciTICmBR80Pdgs/Obj2DwM6dvHquQbQrU Op4XtD3eGUW4qgD99DrMXqCcSXX/uay9kOG+fQBfK39jkPKZEuEV2QdpE4Pry36SUGfohSNq xXW+bMc6P+irTT39VWFUJMcSuQINBE6KyREBEACvEJggkGC42huFAqJcOcLqnjK83t4TVwEn JRisbY/VdeZIHTGtcGLqsALDzk+bEAcZapguzfp7cySzvuR6Hyq7hKEjEHAZmI/3IDc9nbdh EgdCiFatah0XZ/p4vp7KAelYqbv8YF/ORLylAdLh9rzLR6yHFqVaR4WL4pl4kEWwFhNSHLxe 55G56/dxBuoj4RrFoX3ynerXfbp4dH2KArPc0NfoamqebuGNfEQmDbtnCGE5zKcR0zvmXsRp qU7+caufueZyLwjTU+y5p34U4PlOO2Q7/bdaPEdXfpgvSpWk1o3H36LvkPV/PGGDCLzaNn04 BdiiiPEHwoIjCXOAcR+4+eqM4TSwVpTn6SNgbHLjAhCwCDyggK+3qEGJph+WNtNU7uFfscSP k4jqlxc8P+hn9IqaMWaeX9nBEaiKffR7OKjMdtFFnBRSXiW/kOKuuRdeDjL5gWJjY+IpdafP KhjvUFtfSwGdrDUh3SvB5knSixE3qbxbhbNxmqDVzyzMwunFANujyyVizS31DnWC6tKzANkC k15CyeFC6sFFu+WpRxvC6fzQTLI5CRGAB6FAxz8Hu5rpNNZHsbYs9Vfr/BJuSUfRI/12eOCL IvxRPpmMOlcI4WDW3EDkzqNAXn5Onx/b0rFGFpM4GmSPriEJdBb4M4pSD6fN6Y/Jrng/Bdwk SQARAQABiQIlBBgBAgAPBQJOiskRAhsMBQkSzAMAAAoJEGz4yi9OyKjPgEwQAIP/gy/Xqc1q OpzfFScswk3CEoZWSqHxn/fZasa4IzkwhTUmukuIvRew+BzwvrTxhHcz9qQ8hX7iDPTZBcUt ovWPxz+3XfbGqE+q0JunlIsP4N+K/I10nyoGdoFpMFMfDnAiMUiUatHRf9Wsif/nT6oRiPNJ T0EbbeSyIYe+ZOMFfZBVGPqBCbe8YMI+JiZeez8L9JtegxQ6O3EMQ//1eoPJ5mv5lWXLFQfx f4rAcKseM8DE6xs1+1AIsSIG6H+EE3tVm+GdCkBaVAZo2VMVapx9k8RMSlW7vlGEQsHtI0FT c1XNOCGjaP4ITYUiOpfkh+N0nUZVRTxWnJqVPGZ2Nt7xCk7eoJWTSMWmodFlsKSgfblXVfdM 9qoNScM3u0b9iYYuw/ijZ7VtYXFuQdh0XMM/V6zFrLnnhNmg0pnK6hO1LUgZlrxHwLZk5X8F uD/0MCbPmsYUMHPuJd5dSLUFTlejVXIbKTSAMd0tDSP5Ms8Ds84z5eHreiy1ijatqRFWFJRp ZtWlhGRERnDH17PUXDglsOA08HCls0PHx8itYsjYCAyETlxlLApXWdVl9YVwbQpQ+i693t/Y PGu8jotn0++P19d3JwXW8t6TVvBIQ1dRZHx1IxGLMn+CkDJMOmHAUMWTAXX2rf5tUjas8/v2 azzYF4VRJsdl+d0MCaSy8mUh Message-ID: <1f56c881-9005-f8ad-1557-5efd6e0ef535@suse.de> Date: Tue, 18 Jun 2019 17:15:52 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org On 6/18/19 10:06 AM, Daniel Drake wrote: > On Tue, Jun 18, 2019 at 3:46 PM Hannes Reinecke wrote: >> On 6/14/19 4:26 AM, Daniel Drake wrote: >>> On Thu, Jun 13, 2019 at 4:54 PM Christoph Hellwig wrote: >>>> So until we get very clear and good documentation from Intel on that >>>> I don't think any form of upstream support will fly. And given that >>>> Dan who submitted the original patch can't even talk about this thing >>>> any more and apparently got a gag order doesn't really give me confidence >>>> any of this will ever work. >>> >>> I realise the architecture here seems badly thought out, and the lack >>> of a decent spec makes the situation worse, but I'd encourage you to >>> reconsider this from the perspectives of: >>> - Are the patches really more ugly than the underlying architecture? >>> - We strive to make Linux work well on common platforms and sometimes >>> have to accept that hardware vendors do questionable things & do not >>> fully cooperate >>> - It works out of the box on Windows >>> >> Actually, there _is_ a register description: >> >> https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/300-series-chipset-pch-datasheet-vol-2.pdf >> >> Look for section 15: Intel RST for PCIe Storage. >> >> That gives you a reasonable description of the various registers etc. > > Thanks for your email! I also spotted it for the first time earlier today. > > Section 15 there (D24:F0) describes a special/hidden PCI device which > I can't figure out how to access from Linux (I believe it would be at > D18:F0 in the cases where the 300 PCH is integrated into the SoC, as > it is on the Whiskey Lake platform I have here). That's probably not > important because if even if we had access all the values are probably > read-only, as the BIOS will lock them all down during early boot, as > is documented. But the docs give some interesting insights into the > design. > > Section 15.2 is potentially more relevant, as it describes registers > within the AHCI BAR and we do have access to that. Some of these > registers are already used by the current code to determine the > presence of remapped devices. It might be nice to use Device Memory > BAR Length (DMBL_1) but I can't figure out what is meant by "A 1 in > the bit location indicates the corresponding lower memory BAR bit for > the PCIe SSD device is a Read/Write (RW) bit." The value is 0x3fff on > the platform I have here. > > We can probably also use these registers for MSI support. I started to > experiment, doesn't quite work but I'll keep poking. The doc suggests > there is a single MSI-X vector for the AHCI SATA device, and AHCI > MSI-X Starting Vector (AMXV) has value 0x140 on this platform. No idea > how to interpret that value. From experimentation, the AHCI SATA disk > generates interrupts on vector 0. > The 0x140 is probably the offset into the PCI config space where the AHCI MSI-X vector table can be found ... > Then there are multiple vectors for the remapped NVMe devices. Device > MSI-X Configuration (DMXC_L_1) is set up to assign vectors 1 to 19 to > NVMe on this platform. But it says "This field is only valid when > DMXC.ID indicates interrupt delivery using MSI-X" but what/where is > DMXC.ID? So far I can get NVMe-related interrupts on vector 1 but > apparently not enough of them, the driver hangs during probe. > I _think_ the problem here is the automatic interrupt affinity we're doing; we probably have to exclude the AHCI vector when setting up interrupt affinity. > I've nearly finished refreshing & extending Dan Williams' patches and > will send them for more discussion soon. > THX. Cheers, Hannes -- Dr. Hannes Reinecke Teamlead Storage & Networking hare@suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)