From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E23B8C11D3D for ; Thu, 27 Feb 2020 17:55:17 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A41072469B for ; Thu, 27 Feb 2020 17:55:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=intel-com.20150623.gappssmtp.com header.i=@intel-com.20150623.gappssmtp.com header.b="1wq4hmg+" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A41072469B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 549CE6B0006; Thu, 27 Feb 2020 12:55:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4A9C46B0007; Thu, 27 Feb 2020 12:55:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 371EB6B0008; Thu, 27 Feb 2020 12:55:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0110.hostedemail.com [216.40.44.110]) by kanga.kvack.org (Postfix) with ESMTP id 1B1A26B0006 for ; Thu, 27 Feb 2020 12:55:17 -0500 (EST) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id F1D3D8245578 for ; Thu, 27 Feb 2020 17:55:16 +0000 (UTC) X-FDA: 76536658632.24.bat70_10f398a8de50c X-HE-Tag: bat70_10f398a8de50c X-Filterd-Recvd-Size: 5780 Received: from mail-ot1-f66.google.com (mail-ot1-f66.google.com [209.85.210.66]) by imf28.hostedemail.com (Postfix) with ESMTP for ; Thu, 27 Feb 2020 17:55:16 +0000 (UTC) Received: by mail-ot1-f66.google.com with SMTP id j16so27372otl.1 for ; Thu, 27 Feb 2020 09:55:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=DMctyA7z97eyN2JvDNB9L3xQr/YlQGTFBf3sd5NKHWU=; b=1wq4hmg+uAsjEl/wgmRZB+WY61SnT3Q8t2Ct2r2UbddVJTiMvdR+jHFO1VPZwWNFln 3rf7Tc5G5oXeyVoTint/38zy/H44ulX2ua0wHcyBIAHpC8mnjJr+rgmv0kKGiGmqyyyr L6M2BuMpkX8f0mweKz2Xgkf0GCW6ovX4yeF9bZC9VhjnwB1aov3Vbyo9tNgTGFEZTlpn PFau6OyWaDUbHi1RUDFKRo6s3n/qIE7YDXlscTscoKY8uEYiod5jIcG4aTjleM2sHnlx XFPnIuU3sek8SyIYqhhO5bL63kPvhVttnFL+49NWXf0cP337y2MPHX0eDfh4FFNryDT5 9niw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=DMctyA7z97eyN2JvDNB9L3xQr/YlQGTFBf3sd5NKHWU=; b=CH1yjT9b4vBeK8teMQ89FgfST1uAE3ggoO40suQXplCAbROTyO5Preq1WQpT366Jwb 29kdQp9/FMeOklFl0asD4R0+2XQcUylrMFDbi1gSxFoALL15aOlrd8Baiw9BZkRr9Ghe 3zfc77Bqi094ZYKLGIo8K7lZKnhXoceZateG90mmTI7ze6wDs8O6WquoIAqadom1Juob RaEtOz46lmRwwA6N6Uag6ui8n/oEd9m28w+OR/vl7RSPsnEOn/1wAkvbmiEIjSSfuKc2 pZruTdCV6ZwSCAGPTQ9ALNDwy1aJ/RYKYq7pq2QHVIm3Dep8cYHEszkcMBLV9pdgAhm9 kDig== X-Gm-Message-State: APjAAAXBDYUj2TzJOglwQB23ldoNaxb4fk0Rs6TIwPbeMDIMVLBwaPbb Oo1dL3MT9c6aHF2PQnrTBEmxXij7lKe79B1rn53W2g== X-Google-Smtp-Source: APXvYqxdU0nvSDit1sUOCjKYz8qzZqWLce5r/cRVMP78amOH13agUjF/G5uzdRyMv/pxEr2rQ+w8GfXKV+Y/DRwCxOw= X-Received: by 2002:a9d:6c9:: with SMTP id 67mr43495otx.363.1582826115418; Thu, 27 Feb 2020 09:55:15 -0800 (PST) MIME-Version: 1.0 References: <20200221182503.28317-1-logang@deltatee.com> <20200227171704.GK31668@ziepe.ca> <20200227174311.GL31668@ziepe.ca> In-Reply-To: <20200227174311.GL31668@ziepe.ca> From: Dan Williams Date: Thu, 27 Feb 2020 09:55:04 -0800 Message-ID: Subject: Re: [PATCH v3 0/7] Allow setting caching mode in arch_add_memory() for P2PDMA To: Jason Gunthorpe Cc: Logan Gunthorpe , Linux Kernel Mailing List , Linux ARM , linux-ia64@vger.kernel.org, linuxppc-dev , linux-s390 , Linux-sh , platform-driver-x86@vger.kernel.org, Linux MM , Michal Hocko , David Hildenbrand , Andrew Morton , Christoph Hellwig , Catalin Marinas , Will Deacon , Benjamin Herrenschmidt , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Peter Zijlstra , Eric Badger Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Feb 27, 2020 at 9:43 AM Jason Gunthorpe wrote: > > On Thu, Feb 27, 2020 at 10:21:50AM -0700, Logan Gunthorpe wrote: > > > > > > On 2020-02-27 10:17 a.m., Jason Gunthorpe wrote: > > >> Instead of this, this series proposes a change to arch_add_memory() > > >> to take the pgprot required by the mapping which allows us to > > >> explicitly set pagetable entries for P2PDMA memory to WC. > > > > > > Is there a particular reason why WC was selected here? I thought for > > > the p2pdma cases there was no kernel user that touched the memory? > > > > Yes, that's correct. I choose WC here because the existing users are > > registering memory blocks without side effects which fit the WC > > semantics well. > > Hm, AFAIK WC memory is not compatible with the spinlocks/mutexs/etc in > Linux, so while it is true the memory has no side effects, there would > be surprising concurrency risks if anything in the kernel tried to > write to it. > > Not compatible means the locks don't contain stores to WC memory the > way you would expect. AFAIK on many CPUs extra barriers are required > to keep WC stores ordered, the same way ARM already has extra barriers > to keep UC stores ordered with locking.. > > The spinlocks are defined to contain UC stores though. How are spinlocks and mutexes getting into p2pdma ranges in the first instance? Even with UC, the system has bigger problems if it's trying to send bus locks targeting PCI, see the flurry of activity of trying to trigger faults on split locks [1]. This does raise a question about separating the cacheability of the 'struct page' memmap from the BAR range. You get this for free if the memmap is dynamically allocated from "System RAM", but perhaps memremap_pages() should explicitly prevent altmap configurations that try to place the map in PCI space? > If there is no actual need today for WC I would suggest using UC as > the default. That's reasonable, but it still seems to be making a broken configuration marginally less broken. I'd be more interested in safeguards that prevent p2pdma mappings from being used for any cpu atomic cycles. [1]: https://lwn.net/Articles/784864/