From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1162627AbeCAWb2 (ORCPT ); Thu, 1 Mar 2018 17:31:28 -0500 Received: from mail-it0-f68.google.com ([209.85.214.68]:33837 "EHLO mail-it0-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1161204AbeCAWbZ (ORCPT ); Thu, 1 Mar 2018 17:31:25 -0500 X-Google-Smtp-Source: AG47ELtj7q60Aae3BKD5/ZpqaxXiENnABmFCOWQfyBbyU+n7IqjCO/Z/muaCBB706d0TArT5ZQHOud1DYxd4ZF0JhkQ= MIME-Version: 1.0 In-Reply-To: <1519942012.4592.31.camel@au1.ibm.com> References: <20180228234006.21093-1-logang@deltatee.com> <1519876489.4592.3.camel@kernel.crashing.org> <1519876569.4592.4.camel@au1.ibm.com> <1519936477.4592.23.camel@au1.ibm.com> <1519936815.4592.25.camel@au1.ibm.com> <20180301205315.GJ19007@ziepe.ca> <1519942012.4592.31.camel@au1.ibm.com> From: Linus Torvalds Date: Thu, 1 Mar 2018 14:31:23 -0800 X-Google-Sender-Auth: MxlSlVin8F1pB7dkwfxOTY6hfgg Message-ID: Subject: Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory To: Benjamin Herrenschmidt Cc: Jason Gunthorpe , Dan Williams , Logan Gunthorpe , Linux Kernel Mailing List , linux-pci@vger.kernel.org, linux-nvme , linux-rdma , linux-nvdimm , linux-block , Stephen Bates , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Max Gurtovoy , =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Alex Williamson , Oliver OHalloran Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Mar 1, 2018 at 2:06 PM, Benjamin Herrenschmidt wrote: > > Could be that x86 has the smarts to do the right thing, still trying to > untangle the code :-) Afaik, x86 will not cache PCI unless the system is misconfigured, and even then it's more likely to just raise a machine check exception than cache things. The last-level cache is going to do fills and spills directly to the memory controller, not to the PCIe side of things. (I guess you *can* do things differently, and I wouldn't be surprised if some people inside Intel did try to do things differently with trying nvram over PCIe, but in general I think the above is true) You won't find it in the kernel code either. It's in hardware with firmware configuration of what addresses are mapped to the memory controllers (and _how_ they are mapped) and which are not. You _might_ find it in the BIOS, assuming you understood the tables and had the BIOS writer's guide to unravel the magic registers. But you might not even find it there. Some of the memory unit timing programming is done very early, and by code that Intel doesn't even release to the BIOS writers except as a magic encrypted blob, afaik. Some of the magic might even be in microcode. The page table settings for cacheability are more like a hint, and only _part_ of the whole picture. The memory type range registers are another part. And magic low-level uarch, northbridge and memory unit specific magic is yet another part. So you can disable caching for memory, but I'm pretty sure you can't enable caching for PCIe at least in the common case. At best you can affect how the store buffer works for PCIe. Linus